MIREX Wiki - User contributions [en]

2017:Set List Identification Results

2017-10-25T15:42:49Z

Ming-Chi Yen: /* General Legend */

2017:Set List Identification Results

2017-10-25T15:40:12Z

Ming-Chi Yen: /* General Legend */

2017:Set List Identification Results

2017-10-25T15:39:53Z

Ming-Chi Yen: /* General Legend */

2017:Set List Identification Results

2017-10-25T15:39:18Z

Ming-Chi Yen: /* General Legend */

2017:Set List Identification Results

2017-10-25T15:38:38Z

Ming-Chi Yen: /* General Legend */

2017:Set List Identification Results

2017-10-25T15:38:09Z

Ming-Chi Yen: /* General Legend */

2017:Set List Identification Results

2017-10-25T15:37:53Z

Ming-Chi Yen: /* General Legend */

2017:Set List Identification Results

2017-10-25T15:37:32Z

Ming-Chi Yen: /* General Legend */

2017:Set List Identification Results

2017-10-25T08:19:37Z

Ming-Chi Yen: /* Summary */

2017:Set List Identification Results

2017-10-09T16:24:02Z

Ming-Chi Yen: /* Testing Set */

2017:Set List Identification Results

2017-10-09T16:14:20Z

Ming-Chi Yen: /* Developing Set */

2017:Set List Identification Results

2017-10-09T15:23:25Z

Ming-Chi Yen: /* General Legend */

2017:MIREX2017 Results

2017-10-09T14:58:59Z

Ming-Chi Yen: /* Results by Task (More results are coming) */

==Overall Results Poster==
Coming soon

==Results by Task (More results are coming) ==
* [[2017:Audio Fingerprinting Results|Audio Fingerprinting Results]]
* [[2017:Set List Identification Results | Set List Identification Results]]

2017:Set List Identification Results

2017-10-09T14:58:49Z

Ming-Chi Yen: Created page with "== Introduction == Set List Identification was evaluated against on two data sets: ''Developing set'' and ''Testing set'' ===Developing set Information=== This set contain 3 ar..."

2016:Set List Identification Results

2016-08-03T02:27:53Z

Ming-Chi Yen: /* General Legend */

== Introduction ==

Set List Identification was evaluated against on two data sets: ''Developing set'' and ''Testing set''

===Developing set Information===
This set contain 3 artists and 7 live concerts, the following information would be release ([https://www.dropbox.com/sh/t83ogdrxi0f050n/AABb11MCcQUokqSjOsqhArOFa?dl=0 Dropbox])
* artist
* live concert name and links
* studio collection list
* start/end time tags

We extract features for the convenience of participants, the links is the tool we used. ([https://www.dropbox.com/s/bote36k8pkmt2f8/MIREX_2015_Setlist_ID_Developing_set_chroma_fea.rar?dl=0 Dropbox])
*chroma (CRP features [http://resources.mpi-inf.mpg.de/MIR/chromatoolbox/ Chroma Toolbox])

Collection statistics:
* 3 artists
* 7 live concerts
* 279 tracks

=== Testing set Information===
This set contain 7 artists and 13 live concerts, no information would be release.

Collection statistics:
* 7 artists
* 13 live concerts
* 873 tracks

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub Code
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! YWYW1
| style="text-align: center;" | -- || Ming-Chi Yen, Ju-Chiang Wang, Yi-Hsuan Yang, Hsin-Min Wang
|-
! YWJ1
| style="text-align: center;" | -- || Ming-Chi Yen, Hsin-Min Wang, Jyh-Shing Roger Jang
|-
|}

== Results ==

===Developing Set ===
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" |
! colspan=3 align="center" | YWYW1
! colspan=3 align="center" | YWJ1
|-
! width="80" align="center" | Concert ID
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
|-align="right"
! 001 !! 15 !!179.252 !! 176.876 !! 19 !! 179.252 !! 176.876
|-align="right"
! 002 !! 12 !! 11.025 !! 21.615 !! 8 !! 11.025 !! 21.615
|-align="right"
! 024 !! 4 !! 19.124 !! 22.848 !! 5 !! 19.124 !! 22.848
|-align="right"
! 036 !! 18 !! 40.547 !! 49.772 !! 13 !! 40.547 !! 49.772
|-align="right"
! 043 !! 13 !! 341.386 !! 427.867 !! 12 !! 341.386 !! 427.867
|-align="right"
! 069 !! 16 !! 129.983 !! 162.558 !! 12 !! 129.983 !! 162.558
|-align="right"
! 073 !! 13 !! 239.447 !! 309.902 !! 12 !! 239.447 !! 309.902
|-align="right"
! Avg !! 13 !! 137.252 !! 167.349 !! 11.57 !! 137.252 !! 167.349
|}

=== Testing Set ===

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" |
! colspan=3 align="center" | YWYW1
! colspan=3 align="center" | YWJ1
|-
! width="80" align="center" | Concert ID
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
|-align="right"
! 004 !! 6 !! 85.644 !! 33.271 !! 7 !! 85.644 !! 33.271
|-align="right"
! 005 !! 20 !! 304.753 !! 413.590 !! 18 !! 304.753 !! 413.590
|-align="right"
! 009 !! 13 !! 21.407 !! 66.230 !! 8 !! 21.407 !! 66.230
|-align="right"
! 031 !! 10 !! 51.801 !! 71.234 !! 6 !! 51.801 !! 71.234
|-align="right"
! 046 !! 5 !! 11.912 !! 24.883 !! 10 !! 11.912 !! 24.883
|-align="right"
! 047 !! 9 !! 59.835 !! 36.959 !! 11 !! 59.835 !! 36.959
|-align="right"
! 048 !! 14 !! 117.165 !! 125.612 !! 14 !! 117.165 !! 125.612
|-align="right"
! 065 !! 7 !! 178.820 !! 255.402 !! 7 !! 178.820 !! 255.402
|-align="right"
! 066 !! 9 !! 23.994 !! 23.699 !! 7 !! 23.994 !! 23.699
|-align="right"
! 067 !! 11 !! 241.434 !! 301.096 !! 9 !! 241.434 !! 301.096
|-align="right"
! 070 !! 12 !! 69.243 !! 96.336 !! 12 !! 69.243 !! 96.336
|-align="right"
! 076 !! 7 !! 206.096 !! 315.669 !! 5 !! 206.096 !! 315.669
|-align="right"
! 077 !! 13 !! 290.077 !! 278.479 !! 12 !! 290.077 !! 278.479
|-align="right"
! Avg !! 10.46 !! 127.860 !! 157.112 !! 9.69 !! 127.860 !! 157.112

|}

===Summary===
{| border="1" cellspacing="0" style="text-align: left; width: 320px;"
|- style="background: yellow;"
! width="80" | Sub Code
! width="80" | Avg ED
! width="80" | Avg sBD
! width="80" | Avg eBD
|-align="right"
! YWYW1 !! 11.35 !! 131.147 !! 160.695
|-align="right"
! YWJ1 !! 10.35 !! 131.147 !! 160.695

[[Category: Results]]

2016:Set List Identification Results

2016-08-03T02:27:38Z

Ming-Chi Yen: /* General Legend */

== Introduction ==

Set List Identification was evaluated against on two data sets: ''Developing set'' and ''Testing set''

===Developing set Information===
This set contain 3 artists and 7 live concerts, the following information would be release ([https://www.dropbox.com/sh/t83ogdrxi0f050n/AABb11MCcQUokqSjOsqhArOFa?dl=0 Dropbox])
* artist
* live concert name and links
* studio collection list
* start/end time tags

We extract features for the convenience of participants, the links is the tool we used. ([https://www.dropbox.com/s/bote36k8pkmt2f8/MIREX_2015_Setlist_ID_Developing_set_chroma_fea.rar?dl=0 Dropbox])
*chroma (CRP features [http://resources.mpi-inf.mpg.de/MIR/chromatoolbox/ Chroma Toolbox])

Collection statistics:
* 3 artists
* 7 live concerts
* 279 tracks

=== Testing set Information===
This set contain 7 artists and 13 live concerts, no information would be release.

Collection statistics:
* 7 artists
* 13 live concerts
* 873 tracks

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub Code
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! YWYW1
| style="text-align: center;" | -- || Ming-Chi Yen, Ju-Chiang Wang, Yi-Hsuan Yang, Hsin-Min Wang
|-
! YWJ
| style="text-align: center;" | -- || Ming-Chi Yen, Hsin-Min Wang, Jyh-Shing Roger Jang
|-
|}

== Results ==

===Developing Set ===
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" |
! colspan=3 align="center" | YWYW1
! colspan=3 align="center" | YWJ1
|-
! width="80" align="center" | Concert ID
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
|-align="right"
! 001 !! 15 !!179.252 !! 176.876 !! 19 !! 179.252 !! 176.876
|-align="right"
! 002 !! 12 !! 11.025 !! 21.615 !! 8 !! 11.025 !! 21.615
|-align="right"
! 024 !! 4 !! 19.124 !! 22.848 !! 5 !! 19.124 !! 22.848
|-align="right"
! 036 !! 18 !! 40.547 !! 49.772 !! 13 !! 40.547 !! 49.772
|-align="right"
! 043 !! 13 !! 341.386 !! 427.867 !! 12 !! 341.386 !! 427.867
|-align="right"
! 069 !! 16 !! 129.983 !! 162.558 !! 12 !! 129.983 !! 162.558
|-align="right"
! 073 !! 13 !! 239.447 !! 309.902 !! 12 !! 239.447 !! 309.902
|-align="right"
! Avg !! 13 !! 137.252 !! 167.349 !! 11.57 !! 137.252 !! 167.349
|}

=== Testing Set ===

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" |
! colspan=3 align="center" | YWYW1
! colspan=3 align="center" | YWJ1
|-
! width="80" align="center" | Concert ID
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
|-align="right"
! 004 !! 6 !! 85.644 !! 33.271 !! 7 !! 85.644 !! 33.271
|-align="right"
! 005 !! 20 !! 304.753 !! 413.590 !! 18 !! 304.753 !! 413.590
|-align="right"
! 009 !! 13 !! 21.407 !! 66.230 !! 8 !! 21.407 !! 66.230
|-align="right"
! 031 !! 10 !! 51.801 !! 71.234 !! 6 !! 51.801 !! 71.234
|-align="right"
! 046 !! 5 !! 11.912 !! 24.883 !! 10 !! 11.912 !! 24.883
|-align="right"
! 047 !! 9 !! 59.835 !! 36.959 !! 11 !! 59.835 !! 36.959
|-align="right"
! 048 !! 14 !! 117.165 !! 125.612 !! 14 !! 117.165 !! 125.612
|-align="right"
! 065 !! 7 !! 178.820 !! 255.402 !! 7 !! 178.820 !! 255.402
|-align="right"
! 066 !! 9 !! 23.994 !! 23.699 !! 7 !! 23.994 !! 23.699
|-align="right"
! 067 !! 11 !! 241.434 !! 301.096 !! 9 !! 241.434 !! 301.096
|-align="right"
! 070 !! 12 !! 69.243 !! 96.336 !! 12 !! 69.243 !! 96.336
|-align="right"
! 076 !! 7 !! 206.096 !! 315.669 !! 5 !! 206.096 !! 315.669
|-align="right"
! 077 !! 13 !! 290.077 !! 278.479 !! 12 !! 290.077 !! 278.479
|-align="right"
! Avg !! 10.46 !! 127.860 !! 157.112 !! 9.69 !! 127.860 !! 157.112

|}

===Summary===
{| border="1" cellspacing="0" style="text-align: left; width: 320px;"
|- style="background: yellow;"
! width="80" | Sub Code
! width="80" | Avg ED
! width="80" | Avg sBD
! width="80" | Avg eBD
|-align="right"
! YWYW1 !! 11.35 !! 131.147 !! 160.695
|-align="right"
! YWJ1 !! 10.35 !! 131.147 !! 160.695

[[Category: Results]]

2016:Set List Identification Results

2016-07-31T16:26:57Z

Ming-Chi Yen: /* General Legend */

== Introduction ==

Set List Identification was evaluated against on two data sets: ''Developing set'' and ''Testing set''

===Developing set Information===
This set contain 3 artists and 7 live concerts, the following information would be release ([https://www.dropbox.com/sh/t83ogdrxi0f050n/AABb11MCcQUokqSjOsqhArOFa?dl=0 Dropbox])
* artist
* live concert name and links
* studio collection list
* start/end time tags

We extract features for the convenience of participants, the links is the tool we used. ([https://www.dropbox.com/s/bote36k8pkmt2f8/MIREX_2015_Setlist_ID_Developing_set_chroma_fea.rar?dl=0 Dropbox])
*chroma (CRP features [http://resources.mpi-inf.mpg.de/MIR/chromatoolbox/ Chroma Toolbox])

Collection statistics:
* 3 artists
* 7 live concerts
* 279 tracks

=== Testing set Information===
This set contain 7 artists and 13 live concerts, no information would be release.

Collection statistics:
* 7 artists
* 13 live concerts
* 873 tracks

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub Code
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! YWYW1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YWYW1.pdf PDF] || Ming-Chi Yen, Ju-Chiang Wang, Yi-Hsuan Yang, Hsin-Min Wang
|-
! YWJ
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YWYW2.pdf PDF] || Ming-Chi Yen, Hsin-Min Wang, Jyh-Shing Roger Jang
|-
|}

== Results ==

===Developing Set ===
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" |
! colspan=3 align="center" | YWYW1
! colspan=3 align="center" | YWJ1
|-
! width="80" align="center" | Concert ID
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
|-align="right"
! 001 !! 15 !!179.252 !! 176.876 !! 19 !! 179.252 !! 176.876
|-align="right"
! 002 !! 12 !! 11.025 !! 21.615 !! 8 !! 11.025 !! 21.615
|-align="right"
! 024 !! 4 !! 19.124 !! 22.848 !! 5 !! 19.124 !! 22.848
|-align="right"
! 036 !! 18 !! 40.547 !! 49.772 !! 13 !! 40.547 !! 49.772
|-align="right"
! 043 !! 13 !! 341.386 !! 427.867 !! 12 !! 341.386 !! 427.867
|-align="right"
! 069 !! 16 !! 129.983 !! 162.558 !! 12 !! 129.983 !! 162.558
|-align="right"
! 073 !! 13 !! 239.447 !! 309.902 !! 12 !! 239.447 !! 309.902
|-align="right"
! Avg !! 13 !! 137.252 !! 167.349 !! 11.57 !! 137.252 !! 167.349
|}

=== Testing Set ===

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" |
! colspan=3 align="center" | YWYW1
! colspan=3 align="center" | YWJ1
|-
! width="80" align="center" | Concert ID
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
|-align="right"
! 004 !! 6 !! 85.644 !! 33.271 !! 7 !! 85.644 !! 33.271
|-align="right"
! 005 !! 20 !! 304.753 !! 413.590 !! 18 !! 304.753 !! 413.590
|-align="right"
! 009 !! 13 !! 21.407 !! 66.230 !! 8 !! 21.407 !! 66.230
|-align="right"
! 031 !! 10 !! 51.801 !! 71.234 !! 6 !! 51.801 !! 71.234
|-align="right"
! 046 !! 5 !! 11.912 !! 24.883 !! 10 !! 11.912 !! 24.883
|-align="right"
! 047 !! 9 !! 59.835 !! 36.959 !! 11 !! 59.835 !! 36.959
|-align="right"
! 048 !! 14 !! 117.165 !! 125.612 !! 14 !! 117.165 !! 125.612
|-align="right"
! 065 !! 7 !! 178.820 !! 255.402 !! 7 !! 178.820 !! 255.402
|-align="right"
! 066 !! 9 !! 23.994 !! 23.699 !! 7 !! 23.994 !! 23.699
|-align="right"
! 067 !! 11 !! 241.434 !! 301.096 !! 9 !! 241.434 !! 301.096
|-align="right"
! 070 !! 12 !! 69.243 !! 96.336 !! 12 !! 69.243 !! 96.336
|-align="right"
! 076 !! 7 !! 206.096 !! 315.669 !! 5 !! 206.096 !! 315.669
|-align="right"
! 077 !! 13 !! 290.077 !! 278.479 !! 12 !! 290.077 !! 278.479
|-align="right"
! Avg !! 10.46 !! 127.860 !! 157.112 !! 9.69 !! 127.860 !! 157.112

|}

===Summary===
{| border="1" cellspacing="0" style="text-align: left; width: 320px;"
|- style="background: yellow;"
! width="80" | Sub Code
! width="80" | Avg ED
! width="80" | Avg sBD
! width="80" | Avg eBD
|-align="right"
! YWYW1 !! 11.35 !! 131.147 !! 160.695
|-align="right"
! YWJ1 !! 10.35 !! 131.147 !! 160.695

[[Category: Results]]

2016:Set List Identification Results

2016-07-31T16:25:45Z

== Introduction ==

Set List Identification was evaluated against on two data sets: ''Developing set'' and ''Testing set''

===Developing set Information===
This set contain 3 artists and 7 live concerts, the following information would be release ([https://www.dropbox.com/sh/t83ogdrxi0f050n/AABb11MCcQUokqSjOsqhArOFa?dl=0 Dropbox])
* artist
* live concert name and links
* studio collection list
* start/end time tags

We extract features for the convenience of participants, the links is the tool we used. ([https://www.dropbox.com/s/bote36k8pkmt2f8/MIREX_2015_Setlist_ID_Developing_set_chroma_fea.rar?dl=0 Dropbox])
*chroma (CRP features [http://resources.mpi-inf.mpg.de/MIR/chromatoolbox/ Chroma Toolbox])

Collection statistics:
* 3 artists
* 7 live concerts
* 279 tracks

=== Testing set Information===
This set contain 7 artists and 13 live concerts, no information would be release.

Collection statistics:
* 7 artists
* 13 live concerts
* 873 tracks

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub Code
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! YWYW1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YWYW1.pdf PDF] || [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yi-Hsuan Yang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]
|-
! YWJ
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YWYW2.pdf PDF] || [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://www.iis.sinica.edu.tw Hsin-Min Wang], Jyh-Shing Roger Jang
|-
|}

== Results ==

===Developing Set ===
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" |
! colspan=3 align="center" | YWYW1
! colspan=3 align="center" | YWJ1
|-
! width="80" align="center" | Concert ID
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
|-align="right"
! 001 !! 15 !!179.252 !! 176.876 !! 19 !! 179.252 !! 176.876
|-align="right"
! 002 !! 12 !! 11.025 !! 21.615 !! 8 !! 11.025 !! 21.615
|-align="right"
! 024 !! 4 !! 19.124 !! 22.848 !! 5 !! 19.124 !! 22.848
|-align="right"
! 036 !! 18 !! 40.547 !! 49.772 !! 13 !! 40.547 !! 49.772
|-align="right"
! 043 !! 13 !! 341.386 !! 427.867 !! 12 !! 341.386 !! 427.867
|-align="right"
! 069 !! 16 !! 129.983 !! 162.558 !! 12 !! 129.983 !! 162.558
|-align="right"
! 073 !! 13 !! 239.447 !! 309.902 !! 12 !! 239.447 !! 309.902
|-align="right"
! Avg !! 13 !! 137.252 !! 167.349 !! 11.57 !! 137.252 !! 167.349
|}

=== Testing Set ===

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" |
! colspan=3 align="center" | YWYW1
! colspan=3 align="center" | YWJ1
|-
! width="80" align="center" | Concert ID
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
|-align="right"
! 004 !! 6 !! 85.644 !! 33.271 !! 7 !! 85.644 !! 33.271
|-align="right"
! 005 !! 20 !! 304.753 !! 413.590 !! 18 !! 304.753 !! 413.590
|-align="right"
! 009 !! 13 !! 21.407 !! 66.230 !! 8 !! 21.407 !! 66.230
|-align="right"
! 031 !! 10 !! 51.801 !! 71.234 !! 6 !! 51.801 !! 71.234
|-align="right"
! 046 !! 5 !! 11.912 !! 24.883 !! 10 !! 11.912 !! 24.883
|-align="right"
! 047 !! 9 !! 59.835 !! 36.959 !! 11 !! 59.835 !! 36.959
|-align="right"
! 048 !! 14 !! 117.165 !! 125.612 !! 14 !! 117.165 !! 125.612
|-align="right"
! 065 !! 7 !! 178.820 !! 255.402 !! 7 !! 178.820 !! 255.402
|-align="right"
! 066 !! 9 !! 23.994 !! 23.699 !! 7 !! 23.994 !! 23.699
|-align="right"
! 067 !! 11 !! 241.434 !! 301.096 !! 9 !! 241.434 !! 301.096
|-align="right"
! 070 !! 12 !! 69.243 !! 96.336 !! 12 !! 69.243 !! 96.336
|-align="right"
! 076 !! 7 !! 206.096 !! 315.669 !! 5 !! 206.096 !! 315.669
|-align="right"
! 077 !! 13 !! 290.077 !! 278.479 !! 12 !! 290.077 !! 278.479
|-align="right"
! Avg !! 10.46 !! 127.860 !! 157.112 !! 9.69 !! 127.860 !! 157.112

|}

===Summary===
{| border="1" cellspacing="0" style="text-align: left; width: 320px;"
|- style="background: yellow;"
! width="80" | Sub Code
! width="80" | Avg ED
! width="80" | Avg sBD
! width="80" | Avg eBD
|-align="right"
! YWYW1 !! 11.35 !! 131.147 !! 160.695
|-align="right"
! YWJ1 !! 10.35 !! 131.147 !! 160.695

[[Category: Results]]

2015:Set List Identification Results

2015-10-21T07:35:29Z

Ming-Chi Yen: /* General Legend */

== Introduction ==

Set List Identification was evaluated against on two data sets: ''Developing set'' and ''Testing set''

===Developing set Information===
This set contain 3 artists and 7 live concerts, the following information would be release ([https://www.dropbox.com/sh/t83ogdrxi0f050n/AABb11MCcQUokqSjOsqhArOFa?dl=0 Dropbox])
* artist
* live concert name and links
* studio collection list
* start/end time tags

We extract features for the convenience of participants, the links is the tool we used. ([https://www.dropbox.com/s/bote36k8pkmt2f8/MIREX_2015_Setlist_ID_Developing_set_chroma_fea.rar?dl=0 Dropbox])
*chroma (CRP features [http://resources.mpi-inf.mpg.de/MIR/chromatoolbox/ Chroma Toolbox])

Collection statistics:
* 3 artists
* 7 live concerts
* 279 tracks

=== Testing set Information===
This set contain 7 artists and 13 live concerts, no information would be release.

Collection statistics:
* 7 artists
* 13 live concerts
* 873 tracks

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub Code
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! YWYW1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YWYW1.pdf PDF] || [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yi-Hsuan Yang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]
|-
! YWYW2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YWYW2.pdf PDF] || [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yi-Hsuan Yang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]
|-
! YWYW3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YWYW3.pdf PDF] || [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yi-Hsuan Yang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]

|}

== Results ==

===Developing Set ===
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" |
! colspan=3 align="center" | YWYW1
! colspan=3 align="center" | YWYW2
! colspan=3 align="center" | YWYW3
|-
! width="80" align="center" | Concert ID
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
|-align="right"
! 001 !! 15 !!179.252 !! 176.876 !! 19 !! 179.252 !! 176.876 !! 20 !! 1718.929 !! 1636.810
|-align="right"
! 002 !! 12 !! 11.025 !! 21.615 !! 8 !! 11.025 !! 21.615 !! 8 !! 12.157 !! 15.076
|-align="right"
! 024 !! 4 !! 19.124 !! 22.848 !! 5 !! 19.124 !! 22.848 !! 3 !! 108.565 !! 52.557
|-align="right"
! 036 !! 18 !! 40.547 !! 49.772 !! 13 !! 40.547 !! 49.772 !! 12 !! 67.756 !! 59.672
|-align="right"
! 043 !! 13 !! 341.386 !! 427.867 !! 12 !! 341.386 !! 427.867 !! 17 !! 284.778 !! 346.389
|-align="right"
! 069 !! 16 !! 129.983 !! 162.558 !! 12 !! 129.983 !! 162.558 !! 12 !! 83.048 !! 42.002
|-align="right"
! 073 !! 13 !! 239.447 !! 309.902 !! 12 !! 239.447 !! 309.902 !! 16 !! 63.693 !! 95.713
|-align="right"
! Avg !! 13 !! 137.252 !! 167.349 !! 11.57 !! 137.252 !! 167.349 !! 12.57 !! 334.132 !! 321.174
|}

=== Testing Set ===

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" |
! colspan=3 align="center" | YWYW1
! colspan=3 align="center" | YWYW2
! colspan=3 align="center" | YWYW3
|-
! width="80" align="center" | Concert ID
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
|-align="right"
! 004 !! 6 !! 85.644 !! 33.271 !! 7 !! 85.644 !! 33.271 !! 8 !! 490.658 !! 445.986
|-align="right"
! 005 !! 20 !! 304.753 !! 413.590 !! 18 !! 304.753 !! 413.590 !! 16 !! 144.300 !! 185.604
|-align="right"
! 009 !! 13 !! 21.407 !! 66.230 !! 8 !! 21.407 !! 66.230 !! 11 !! 38.930 !! 51.577
|-align="right"
! 031 !! 10 !! 51.801 !! 71.234 !! 6 !! 51.801 !! 71.234 !! 8 !! 117.670 !! 63.418
|-align="right"
! 046 !! 5 !! 11.912 !! 24.883 !! 10 !! 11.912 !! 24.883 !! 13 !! 840.530 !! 818.607
|-align="right"
! 047 !! 9 !! 59.835 !! 36.959 !! 11 !! 59.835 !! 36.959 !! 18 !! 195.221 !! 131.040
|-align="right"
! 048 !! 14 !! 117.165 !! 125.612 !! 14 !! 117.165 !! 125.612 !! 14 !! 132.044 !! 157.874
|-align="right"
! 065 !! 7 !! 178.820 !! 255.402 !! 7 !! 178.820 !! 255.402 !! 6 !! 155.572 !! 227.516
|-align="right"
! 066 !! 9 !! 23.994 !! 23.699 !! 7 !! 23.994 !! 23.699 !! 8 !! 1371.639 !! 1372.828
|-align="right"
! 067 !! 11 !! 241.434 !! 301.096 !! 9 !! 241.434 !! 301.096 !! 9 !! 107.313 !! 93.293
|-align="right"
! 070 !! 12 !! 69.243 !! 96.336 !! 12 !! 69.243 !! 96.336 !! 11 !! 80.615 !! 98.123
|-align="right"
! 076 !! 7 !! 206.096 !! 315.669 !! 5 !! 206.096 !! 315.669 !! 7 !! 128.045 !! 78.691
|-align="right"
! 077 !! 13 !! 290.077 !! 278.479 !! 12 !! 290.077 !! 278.479 !! 13 !! 315.441 !! 290.992
|-align="right"
! Avg !! 10.46 !! 127.860 !! 157.112 !! 9.69 !! 127.860 !! 157.112 !! 10.92 !! 316.768 !! 308.888

|}

===Summary===
{| border="1" cellspacing="0" style="text-align: left; width: 320px;"
|- style="background: yellow;"
! width="80" | Sub Code
! width="80" | Avg ED
! width="80" | Avg sBD
! width="80" | Avg eBD
|-align="right"
! YWYW1 !! 11.35 !! 131.147 !! 160.695
|-align="right"
! YWYW2 !! 10.35 !! 131.147 !! 160.695
|-align="right"
! YWYW3 !! 11.50 !! 322.845 !! 313.188

[[Category: Results]]

2015:Set List Identification Results

2015-10-21T07:35:06Z

Ming-Chi Yen: /* General Legend */

== Introduction ==

Set List Identification was evaluated against on two data sets: ''Developing set'' and ''Testing set''

===Developing set Information===
This set contain 3 artists and 7 live concerts, the following information would be release ([https://www.dropbox.com/sh/t83ogdrxi0f050n/AABb11MCcQUokqSjOsqhArOFa?dl=0 Dropbox])
* artist
* live concert name and links
* studio collection list
* start/end time tags

We extract features for the convenience of participants, the links is the tool we used. ([https://www.dropbox.com/s/bote36k8pkmt2f8/MIREX_2015_Setlist_ID_Developing_set_chroma_fea.rar?dl=0 Dropbox])
*chroma (CRP features [http://resources.mpi-inf.mpg.de/MIR/chromatoolbox/ Chroma Toolbox])

Collection statistics:
* 3 artists
* 7 live concerts
* 279 tracks

=== Testing set Information===
This set contain 7 artists and 13 live concerts, no information would be release.

Collection statistics:
* 7 artists
* 13 live concerts
* 873 tracks

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! YWYW1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YWYW1.pdf PDF] || [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yi-Hsuan Yang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]
|-
! YWYW2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YWYW2.pdf PDF] || [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yi-Hsuan Yang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]
|-
! YWYW3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YWYW3.pdf PDF] || [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yi-Hsuan Yang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]

|}

== Results ==

===Developing Set ===
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" |
! colspan=3 align="center" | YWYW1
! colspan=3 align="center" | YWYW2
! colspan=3 align="center" | YWYW3
|-
! width="80" align="center" | Concert ID
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
|-align="right"
! 001 !! 15 !!179.252 !! 176.876 !! 19 !! 179.252 !! 176.876 !! 20 !! 1718.929 !! 1636.810
|-align="right"
! 002 !! 12 !! 11.025 !! 21.615 !! 8 !! 11.025 !! 21.615 !! 8 !! 12.157 !! 15.076
|-align="right"
! 024 !! 4 !! 19.124 !! 22.848 !! 5 !! 19.124 !! 22.848 !! 3 !! 108.565 !! 52.557
|-align="right"
! 036 !! 18 !! 40.547 !! 49.772 !! 13 !! 40.547 !! 49.772 !! 12 !! 67.756 !! 59.672
|-align="right"
! 043 !! 13 !! 341.386 !! 427.867 !! 12 !! 341.386 !! 427.867 !! 17 !! 284.778 !! 346.389
|-align="right"
! 069 !! 16 !! 129.983 !! 162.558 !! 12 !! 129.983 !! 162.558 !! 12 !! 83.048 !! 42.002
|-align="right"
! 073 !! 13 !! 239.447 !! 309.902 !! 12 !! 239.447 !! 309.902 !! 16 !! 63.693 !! 95.713
|-align="right"
! Avg !! 13 !! 137.252 !! 167.349 !! 11.57 !! 137.252 !! 167.349 !! 12.57 !! 334.132 !! 321.174
|}

=== Testing Set ===

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" |
! colspan=3 align="center" | YWYW1
! colspan=3 align="center" | YWYW2
! colspan=3 align="center" | YWYW3
|-
! width="80" align="center" | Concert ID
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
! width="80" align="center" | ED
! width="80" align="center" | sBD
! width="80" align="center" | eBD
|-align="right"
! 004 !! 6 !! 85.644 !! 33.271 !! 7 !! 85.644 !! 33.271 !! 8 !! 490.658 !! 445.986
|-align="right"
! 005 !! 20 !! 304.753 !! 413.590 !! 18 !! 304.753 !! 413.590 !! 16 !! 144.300 !! 185.604
|-align="right"
! 009 !! 13 !! 21.407 !! 66.230 !! 8 !! 21.407 !! 66.230 !! 11 !! 38.930 !! 51.577
|-align="right"
! 031 !! 10 !! 51.801 !! 71.234 !! 6 !! 51.801 !! 71.234 !! 8 !! 117.670 !! 63.418
|-align="right"
! 046 !! 5 !! 11.912 !! 24.883 !! 10 !! 11.912 !! 24.883 !! 13 !! 840.530 !! 818.607
|-align="right"
! 047 !! 9 !! 59.835 !! 36.959 !! 11 !! 59.835 !! 36.959 !! 18 !! 195.221 !! 131.040
|-align="right"
! 048 !! 14 !! 117.165 !! 125.612 !! 14 !! 117.165 !! 125.612 !! 14 !! 132.044 !! 157.874
|-align="right"
! 065 !! 7 !! 178.820 !! 255.402 !! 7 !! 178.820 !! 255.402 !! 6 !! 155.572 !! 227.516
|-align="right"
! 066 !! 9 !! 23.994 !! 23.699 !! 7 !! 23.994 !! 23.699 !! 8 !! 1371.639 !! 1372.828
|-align="right"
! 067 !! 11 !! 241.434 !! 301.096 !! 9 !! 241.434 !! 301.096 !! 9 !! 107.313 !! 93.293
|-align="right"
! 070 !! 12 !! 69.243 !! 96.336 !! 12 !! 69.243 !! 96.336 !! 11 !! 80.615 !! 98.123
|-align="right"
! 076 !! 7 !! 206.096 !! 315.669 !! 5 !! 206.096 !! 315.669 !! 7 !! 128.045 !! 78.691
|-align="right"
! 077 !! 13 !! 290.077 !! 278.479 !! 12 !! 290.077 !! 278.479 !! 13 !! 315.441 !! 290.992
|-align="right"
! Avg !! 10.46 !! 127.860 !! 157.112 !! 9.69 !! 127.860 !! 157.112 !! 10.92 !! 316.768 !! 308.888

|}

===Summary===
{| border="1" cellspacing="0" style="text-align: left; width: 320px;"
|- style="background: yellow;"
! width="80" | Sub Code
! width="80" | Avg ED
! width="80" | Avg sBD
! width="80" | Avg eBD
|-align="right"
! YWYW1 !! 11.35 !! 131.147 !! 160.695
|-align="right"
! YWYW2 !! 10.35 !! 131.147 !! 160.695
|-align="right"
! YWYW3 !! 11.50 !! 322.845 !! 313.188

[[Category: Results]]

2015:Set List Identification Results

2015-10-21T07:29:42Z

Ming-Chi Yen:

2015-06-18T08:47:24Z

Ming-Chi Yen: /* Developing set */

__TOC__

==Description==

'''This task is new for 2015!'''

This task requires that algorithm identify the '''set list''' (See [http://en.wikipedia.org/wiki/Set_list Set list]). Set list is the song sequence in a live concert. It shows the order of songs will be performed in a live concert.

Recently, more and more full-length live concert videos have become available on website (e.g. [https://www.youtube.com/ Youtube]). Most of them are lacking sufficient information to describe itself, such as the set list, and start/end time of each song. In this task, we collect the audio of live concerts and studio songs, applying music information retrieval techniques to answer this question -- what songs had been sung in this concert and when are the songs start and end.

For the first step of this task, we assume that '''artist is known'''. In the live concert, '''the performers play their studio songs only''', however the ultimate goal is granted a full-length live concert audio and studio song database, we still can find out the set list and the start/end time of each song.

here are two sub tasks in this task:

===Sub task 1: Song sequence identification===
*To identify the order of songs which be performed in a live concert.

In this sub task, the participants known the the artist and artist's studio song collection. Assigning a live concert audio and studio songs collection of a specific artist, all songs in live concert are included in studio songs collection, to identify the order of songs in this live concert.

===Sub task 2: Time boundary identification===
*To identify the start/end time of each song in song sequence

In this sub task, the participants known the artist, artist's studio song collection and the '''song sequence'''. Assigning a live concert audio, song sequence and studio songs collection of a specific artist, all songs in live concert are included in studio songs collection, to identify start time and end time of each song in the live concert.

== Data ==
To satisfy our assessment, we pre-process all audio -- '''remove the "out of artist song" form live concert audio''' for following our assumption. (See the [http://140.109.22.101/MediaWiki/index.php?title=Set_List_Identification#Description description])

We provide two set for this task,participating algorithms will have to read audio in the following format.

* Sample rate: 22050 Hz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

===Developing set===
This set contain 3 artists and 7 live concerts, the following information would be release (Dropbox [https://www.dropbox.com/sh/t83ogdrxi0f050n/AABb11MCcQUokqSjOsqhArOFa?dl=0])
* artist
* live concert name and links
* studio collection list
* start/end time tags

We also prepare feature for participants, the links is the tool we used.
*chroma (CRP features [http://resources.mpi-inf.mpg.de/MIR/chromatoolbox/ Chroma Toolbox])

Collection statistics:
* 3 artists
* 7 live concerts
* 279 tracks

=== Testing set ===
This set contain 7 artists and 13 live concerts, no information would be release.

Collection statistics:
* 7 artists
* 13 live concerts
* 873 tracks

== Evaluation ==

For two tasks, the evaluation metrics were different.

=== Sub task 1===

* Edit distance (see [http://en.wikipedia.org/wiki/Edit_distance Edit distance])

We evaluated the two sequence (ground truth and your result) by edit distance, there three errors included
* insertion error <math>I</math>
* substitution error <math>S</math>
* deletion error <math>D</math>

Edit Distance: <big><math>ED = I+S+D </math> </big>

Percent Correct: <big><math>Corr = \frac{N-D-S}{N}</math></big>

Percent Accuracy: <big><math> Acc = \frac{N-D-S-I}{N}</math></big>

=== Sub task 2===

* average time boundary

We will evaluate two time time boundaries as follow: average start time boundary and average start time boundary. The evaluation function is described below:

Set list contains <big><math>N</math></big> songs

Start time of ground truth：<big><math>sBD_{GT}</math></big>

End time of ground truth：<big><math>eBD_{GT}</math></big>

Start time of identification result：<big><math>sBD_{ID}</math></big>

End time of identification result：<big><math>eBD_{ID}</math></big>

<big><math> AVGsBD = \frac{|sBD_{GT} - sBD_{ID}|}{N} </math></big>,

<big><math> AVGeBD = \frac{|eBD_{GT} - eBD_{ID}|}{N} </math></big>,

=== Runtime performance ===
In addition computation times for feature extraction and training/classification will be measured.

== Submission Format ==
* '''\n''' is end of line

Submission to this task will have to conform to a specified format detailed below.
=== Implementation details ===
we recommend your submission folder construction as follow:
/root_folder/... all the code you submitted
/root_folder/extract_feature/... all feature your extracted
/root_folder/output/... the folder to save results

=== Sub task 1 ===

==== Input file ====
The input for studio songs list file format will be of the form:

/path/to/artist_1/studio/song/001.wav\n 1st
/path/to/artist_1/studio/song/002.wav\n 2nd
/path/to/artist_1/studio/song/003.wav\n 3rd
...

The input for live concert list file format will be of the form:

/path/to/artist_1/live/concert/001.wav\n

==== Output file ====
The output is a list file (song ID sequence), '''the song ID is the order of input list file''', not the file name of *.wav file.

3\n <-- 003.wav is the first song of set list for your identification result
17\n
59\n
...

=== Sub task 2 ===
==== Input file ====

The input is a list of song ID (song ID sequence), '''the song ID is the order of studio songs list file'''.

Your system should read the *.wav file according that order and find the time boundary of the song.

3\n
17\n
59\n
...

==== Output file ====

The output for studio songs time boundary list file format will be of the form:
* please round the time boundary to millisecond
* '''\t''' is tab space
Start time end time
hours.minutes.seconds.milliseconds \t hours.minutes.seconds.milliseconds\n ...for input input sond ID:3
hours.minutes.seconds.milliseconds \t hours.minutes.seconds.milliseconds\n ...for input input sond ID:17
hours.minutes.seconds.milliseconds \t hours.minutes.seconds.milliseconds\n ...for input input sond ID:59
...

Examples:
0.7.23.521 0.13.24.512
0.14.3.021 0.19.53.38
0.20.9.893 0.27.15.987
...
...
0.56.22.433 1.1.46.593
1.3.51.146 1.9.21.138
...

=== Packaging submissions ===

All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guarenteed).

All submissions should include a README file including the following the information:

* Which task you want to participate (sub task1, sub task2 or all)
* Command line calling format for all executables and an example formatted set of commands
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Any required environments (and versions), e.g. python, java, bash, matlab.

== Time and hardware limits ==
Due to the potentially high number of particpants in this and other audio tasks, hard limits on the runtime of submissions are specified.

A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.

== Potential Participants ==
name / email

2015:Set List Identification

2015-06-16T11:56:42Z

Ming-Chi Yen: /* Data */

__TOC__

==Description==

'''This task is new for 2015!'''

This task requires that algorithm identify the '''set list''' (See [http://en.wikipedia.org/wiki/Set_list Set list]). Set list is the song sequence in a live concert. It shows the order of songs will be performed in a live concert.

Recently, more and more full-length live concert videos have become available on website (e.g. [https://www.youtube.com/ Youtube]). Most of them are lacking sufficient information to describe itself, such as the set list, and start/end time of each song. In this task, we collect the audio of live concerts and studio songs, applying music information retrieval techniques to answer this question -- what songs had been sung in this concert and when are the songs start and end.

For the first step of this task, we assume that '''artist is known'''. In the live concert, '''the performers play their studio songs only''', however the ultimate goal is granted a full-length live concert audio and studio song database, we still can find out the set list and the start/end time of each song.

here are two sub tasks in this task:

===Sub task 1: Song sequence identification===
*To identify the order of songs which be performed in a live concert.

In this sub task, the participants known the the artist and artist's studio song collection. Assigning a live concert audio and studio songs collection of a specific artist, all songs in live concert are included in studio songs collection, to identify the order of songs in this live concert.

===Sub task 2: Time boundary identification===
*To identify the start/end time of each song in song sequence

In this sub task, the participants known the artist, artist's studio song collection and the '''song sequence'''. Assigning a live concert audio, song sequence and studio songs collection of a specific artist, all songs in live concert are included in studio songs collection, to identify start time and end time of each song in the live concert.

== Data ==
To satisfy our assessment, we pre-process all audio -- '''remove the "out of artist song" form live concert audio''' for following our assumption. (See the [http://140.109.22.101/MediaWiki/index.php?title=Set_List_Identification#Description description])

We provide two set for this task,participating algorithms will have to read audio in the following format.

* Sample rate: 22050 Hz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

===Developing set===
This set contain 3 artists and 7 live concerts, the following information would be release:
* artist
* live concert name and links
* studio collection list
* start/end time tags

We also prepare feature for participants, the links is the tool we used.
*chroma (CRP features [http://resources.mpi-inf.mpg.de/MIR/chromatoolbox/ Chroma Toolbox])

Collection statistics:
* 3 artists
* 7 live concerts
* 279 tracks

=== Testing set ===
This set contain 7 artists and 13 live concerts, no information would be release.

Collection statistics:
* 7 artists
* 13 live concerts
* 873 tracks

== Evaluation ==

For two tasks, the evaluation metrics were different.

=== Sub task 1===

* Edit distance (see [http://en.wikipedia.org/wiki/Edit_distance Edit distance])

We evaluated the two sequence (ground truth and your result) by edit distance, there three errors included
* insertion error <math>I</math>
* substitution error <math>S</math>
* deletion error <math>D</math>

Edit Distance: <big><math>ED = I+S+D </math> </big>

Percent Correct: <big><math>Corr = \frac{N-D-S}{N}</math></big>

Percent Accuracy: <big><math> Acc = \frac{N-D-S-I}{N}</math></big>

=== Sub task 2===

* average time boundary

We will evaluate two time time boundaries as follow: average start time boundary and average start time boundary. The evaluation function is described below:

Set list contains <big><math>N</math></big> songs

Start time of ground truth：<big><math>sBD_{GT}</math></big>

End time of ground truth：<big><math>eBD_{GT}</math></big>

Start time of identification result：<big><math>sBD_{ID}</math></big>

End time of identification result：<big><math>eBD_{ID}</math></big>

<big><math> AVGsBD = \frac{|sBD_{GT} - sBD_{ID}|}{N} </math></big>,

<big><math> AVGeBD = \frac{|eBD_{GT} - eBD_{ID}|}{N} </math></big>,

=== Runtime performance ===
In addition computation times for feature extraction and training/classification will be measured.

== Submission Format ==
* '''\n''' is end of line

Submission to this task will have to conform to a specified format detailed below.
=== Implementation details ===
we recommend your submission folder construction as follow:
/root_folder/... all the code you submitted
/root_folder/extract_feature/... all feature your extracted
/root_folder/output/... the folder to save results

=== Sub task 1 ===

==== Input file ====
The input for studio songs list file format will be of the form:

/path/to/artist_1/studio/song/001.wav\n 1st
/path/to/artist_1/studio/song/002.wav\n 2nd
/path/to/artist_1/studio/song/003.wav\n 3rd
...

The input for live concert list file format will be of the form:

/path/to/artist_1/live/concert/001.wav\n

==== Output file ====
The output is a list file (song ID sequence), '''the song ID is the order of input list file''', not the file name of *.wav file.

3\n <-- 003.wav is the first song of set list for your identification result
17\n
59\n
...

=== Sub task 2 ===
==== Input file ====

The input is a list of song ID (song ID sequence), '''the song ID is the order of studio songs list file'''.

Your system should read the *.wav file according that order and find the time boundary of the song.

3\n
17\n
59\n
...

==== Output file ====

The output for studio songs time boundary list file format will be of the form:
* please round the time boundary to millisecond
* '''\t''' is tab space
Start time end time
hours.minutes.seconds.milliseconds \t hours.minutes.seconds.milliseconds\n ...for input input sond ID:3
hours.minutes.seconds.milliseconds \t hours.minutes.seconds.milliseconds\n ...for input input sond ID:17
hours.minutes.seconds.milliseconds \t hours.minutes.seconds.milliseconds\n ...for input input sond ID:59
...

Examples:
0.7.23.521 0.13.24.512
0.14.3.021 0.19.53.38
0.20.9.893 0.27.15.987
...
...
0.56.22.433 1.1.46.593
1.3.51.146 1.9.21.138
...

=== Packaging submissions ===

All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guarenteed).

All submissions should include a README file including the following the information:

* Which task you want to participate (sub task1, sub task2 or all)
* Command line calling format for all executables and an example formatted set of commands
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Any required environments (and versions), e.g. python, java, bash, matlab.

== Time and hardware limits ==
Due to the potentially high number of particpants in this and other audio tasks, hard limits on the runtime of submissions are specified.

A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.

== Potential Participants ==
name / email