2010:Audio Cover Song Identification - Revision history

Kriswest: /* Submission opening date */

2010-06-05T09:12:27Z

‎Submission opening date

Jdownie: /* Task specific mailing list */

2010-06-04T01:26:15Z

‎Task specific mailing list

Kriswest at 20:16, 24 May 2010

2010-05-24T20:16:06Z

Kriswest: tweaking data section and a few other bits

2010-05-24T15:39:23Z

tweaking data section and a few other bits

Show changes

Kriswest: /* Output File */

2010-05-24T14:32:17Z

‎Output File

Kriswest: better dist mat definition

2010-05-24T13:11:02Z

better dist mat definition

Kriswest: mailing list and minor tweaks

2010-05-24T13:03:45Z

mailing list and minor tweaks

CameronJones: /* 2010 AUDIO COVER SONG IDENTIFICATION TASK OVERVIEW */

2010-05-10T10:32:02Z

‎2010 AUDIO COVER SONG IDENTIFICATION TASK OVERVIEW

MertBay at 17:50, 22 March 2010

2010-03-22T17:50:33Z

MertBay: /* 2010 AUDIO COVER SONG IDENTIFICATION TASK OVERVIEW */

2010-03-22T17:49:08Z

‎2010 AUDIO COVER SONG IDENTIFICATION TASK OVERVIEW

@@ Line 174: / Line 174: @@
 == Submission opening date ==
 == Submission closing date ==
 TBA

@@ Line 11: / Line 11: @@
 === Task specific mailing list ===
-A specific mailing list is provided for the discussion of this task and related tasks ( [[2010:Audio Classification (Test/Train) tasks]], [[2010:Audio_Cover_Song_Identification]], [[2010:Audio_Tag_Classification]], [[2010:Audio_Music_Similarity_and_Retrieval]]) at: [https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com00 https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com00]. If you wish to participate in any of these tasks please sign up to this mailing listas discussion of the task format and evaluation should be conducted there.
+In the past we have use a specific mailing list for the discussion of this task and related tasks (e.g., [[2010:Audio Classification (Train/Test) Tasks]], [[2010:Audio Cover Song Identification]], [[2010:Audio Tag Classification]], [[2010:Audio Music Similarity and Retrieval]]). This year, however, we are asking that all discussions take place on the MIREX  [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.
 == Data ==

@@ Line 1: / Line 1: @@
-=2010 AUDIO COVER SONG IDENTIFICATION TASK OVERVIEW=
+__TOC__
 ==Description==
 Within the a collection of pieces in the cover song datasets, there are embedded a number of different "original songs" or compositions each represented by a number of different "versions". The "cover songs" or "versions" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.
-Using each of these version files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other versions of the "seed/query" file.
+Using each of these version files in turn as as the "seed/query" file, we examine the returned ranked lists of items from each algorithm for the presence of the other versions of the "seed/query" file.
 === Task specific mailing list ===
@@ Line 22: / Line 17: @@
 Two datasets will be used to evaluate cover song identification:
-===US Pop Music Collection (aka Mixed Collection)===
+===US Pop Music Collection Cover Song (aka Mixed Collection)===
 This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files.

@@ Line 65: / Line 65: @@
 <pre>
 Distance matrix header text with system name
-\t</path/to/audio/file/1.wav>
+\t</path/to/audio/file/track1.wav>
-\t</path/to/audio/file/2.wav>
+\t</path/to/audio/file/track2.wav>
-\t</path/to/audio/file/3.wav>
+\t</path/to/audio/file/track3.wav>
 ...
-N\t</path/to/audio/file/N.wav>
+N\t</path/to/audio/file/trackN.wav>
-Q/R\t1\t2\t3\t...\tM
+Q/R\t1\t2\t3\t4\t...\tN
-\t<dist query 1 to 1>\t<dist query 1 to 2>\t<dist query 1 to 3>\t...\t<dist query 1 to M>
+\t<dist 1 to 1>\t<dist 1 to 2>\t<dist 1 to 3>\t<dist 1 to 4>\t...\t<dist 1 to N>
-\t<dist query 2 to 1>\t<dist query 2 to 2>\t<dist query 2 to 3>\t...\t<dist query 2 to M>
+\t<dist 3 to 2>\t<dist 3 to 2>\t<dist 3 to 3>\t<dist 3 to 4>\t...\t<dist 3 to N>
 </pre>
-where N is 330 (number of queries) and M is 1000, candidate tracks in dataset.
+where N is 1000 (number of candidate tracks in dataset) and the queries are drawn from this set (and bear the same track indexes if possible).
 which might look like:
@@ Line 84: / Line 82: @@
 <pre>
 Example distance matrix 0.1
-    /path/to/audio/file/1.wav
+    /path/to/audio/file/track1.wav
-    /path/to/audio/file/2.wav
+    /path/to/audio/file/track2.wav
-    /path/to/audio/file/3.wav
+    /path/to/audio/file/track3.wav
-    /path/to/audio/file/4.wav
+    /path/to/audio/file/track4.wav
 Q/R   1        2        3        4        5
      0.00000  1.24100  0.2e-4   0.42559  0.21313
      50.2e-4  0.62640  0.00000  0.38000  0.15152
 </pre>
 All distances should be zero or positive (0.0+) and should not be infinite or NaN. Values should be separated by a TAB.
-As N (collection searched for covers) is 1000 and there are 330 original tracks, the distance matrix should be preceded by 1000 rows of file paths and should be composed of 1000 columns of distance (separated by tab characters) and 330 rows (one for each original track). Each row corresponds to a particular query song (the track to find covers of). Please ensure that the query songs are listed in exactly the same order as they appear in the list file you are passed.
+As N (collection searched for covers) is 1000 and there are 330 original tracks, the distance matrix should be preceded by a system name, 1000 rows of file paths and should be composed of 1000 columns of distance (separated by tab characters) and 330 rows (one for each original track query). Each row corresponds to a particular query song (the track to find covers of).
 ==Evaluation==
 We will employ the same measures used in [[2007:Audio Cover Song]].

@@ Line 4: / Line 4: @@
 The Audio Cover Song task was a new task for MIREX 2006 and was last run in 2008. It was closely related to the [[2010:Audio Music Similarity and Retrieval]] (AMS) task as the cover songs were embedded in the Audio Music Similarity and Retrieval test collection.
 ==Task Description==
@@ Line 12: / Line 13: @@
 On top of the previous Audio Cover Song dataset, we are going to use the  [http://www.mazurka.org.uk/ Mazurka dataset]. We are going to randomly choose 11 versions from 49 mazurkas and run it as a separate subtask. The I/O format will be the same as previous years. Systems will return a distance matrix of 539x539.
 === Task specific mailing list ===
@@ Line 58: / Line 60: @@
 === Output File ===
 The only output will be a '''distance''' matrix file that is 330 rows by 1000 columns in the following format:
 <pre>
-Example distance matrix 0.1 (replace this line with your system name)
+Distance matrix header text with system name
-   path/to/audio/file/1.wav
+\t</path/to/audio/file/1.wav>
-   path/to/audio/file/2.wav
+\t</path/to/audio/file/2.wav>
-   path/to/audio/file/3.wav
+\t</path/to/audio/file/3.wav>
 ...
-N    path/to/audio/file/N.wav
+N\t</path/to/audio/file/N.wav>
-Q/R    1        2        3        ...        N
+Q/R\t1\t2\t3\t...\tM
-   0.0      1.241    0.2e-4     ...    0.4255934
+\t<dist query 1 to 1>\t<dist query 1 to 2>\t<dist query 1 to 3>\t...\t<dist query 1 to M>
-   1.241    0.000    0.6264     ...    0.2356447
+\t<dist query 2 to 1>\t<dist query 2 to 2>\t<dist query 2 to 3>\t...\t<dist query 2 to M>
-   50.2e-4  0.6264   0.0000     ...    0.3800000
+\t<dist query 3 to 2>\t<dist query 3 to 2>\t<dist query 3 to 3>\t...\t<dist query 3 to M>
-...    ...    ...      ...        ...    0.7172300
+...\t...\t...\t...\t...\t...
-    0.42559  0.23567  0.38       ...    0.000
+N\t<dist N to 1>\t<dist N to 2>\t<dist N to 3>\t...\t<dist N to M>
 </pre>
 All distances should be zero or positive (0.0+) and should not be infinite or NaN. Values should be separated by a TAB.

@@ Line 3: / Line 3: @@
 The text of this section is copied from the 2009 page. Please add your comments and discussions for 2010.
-The Audio Cover Song task was a new task for MIREX 2006 and was last run in 2008. It was closely related to the [[Audio Music Similarity and Retrieval]] (AMS) task as the cover songs were embedded in the Audio Music Similarity and Retrieval test collection.
+The Audio Cover Song task was a new task for MIREX 2006 and was last run in 2008. It was closely related to the [[2010:Audio Music Similarity and Retrieval]] (AMS) task as the cover songs were embedded in the Audio Music Similarity and Retrieval test collection.
 ==Task Description==

@@ Line 15: / Line 15: @@
 == Discussions  for 2010 ==
-Is there any interest in running a general identification sub-task? That is, the input would be a reference song and a test song, and the algorithm would classify the reference/test pair as either a reference/cover or a reference/non-cover. Ultimately, this is what we want any cover song detection algorithm to do and given the rather large strides the MIR community has made on cover song detection over the past couple years, I think this new sub-task would be both challenging and meaningful. - Suman Ravuri
 === Command Line Calling Format ===