Difference between revisions of "2007:Audio Music Similarity and Retrieval"

From MIREX Wiki
(→‎Participants)
m (Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2007:\1))
 
(45 intermediate revisions by 16 users not shown)
Line 1: Line 1:
 
== Status ==
 
== Status ==
−
This is only a very basic draft version of a task proposal. Once more people show interest we can fill in the details.
+
A provisional specification of the Audio Music Similarity task is detailed below. This proposal may be refined based on feedback from the particpants.
  
−
Note that audio music similarity and retrieval algorithms have been evaluated at MIREX 2006.
+
Note that audio music similarity and retrieval algorithms have been evaluated at MIREX 2006 ( [[2006:Audio_Music_Similarity_and_Retrieval]] ).
  
 
Related MIREX 2007 task proposals:  
 
Related MIREX 2007 task proposals:  
−
* [[Audio Music Mood Classification]]
+
* [[2007:Audio Music Mood Classification]]
−
* [[Audio Artist Identification]]
+
* [[2007:Audio Artist Identification]]
−
* [[Audio Artist Similarity]]
+
* [[2007:Audio Genre Classification]]
−
* [[Audio Genre Classification]]
+
* [[2007:Audio Cover song identification]]
  
 
Please feel free to edit this page.
 
Please feel free to edit this page.
  
 
== Data ==
 
== Data ==
−
The data used for last year's audio similarity retrieval task (USPOP + USCRAP) could be used. In addition, the Magnatune data used for the ISMIR 2004 genre classification contest could be used.  
+
Collection statistics: 7000 30-second audio clips in 22.05kHz mono wav format drawn from 10 genres (700 clips from each genre).
 +
Genres:
 +
*Blues
 +
*Jazz
 +
*Country/Western
 +
*Baroque
 +
*Classical
 +
*Romantic
 +
*Electronica
 +
*Hip-Hop
 +
*Rock
 +
*HardRock/Metal
  
−
Please edit this if you have suggestions to add or if you disagree.
+
== Evaluation ==
 +
 
 +
Two distinct evaluations will be performed
 +
* Human Evaluation
 +
* Objective statistics derived from the results lists
 +
 
 +
Note that at MIREX 2006 particpating algorithms were required to return full distance matrices showing the distance between all tracks. This year we are also going to allow a sparse distance matrix format (detailed below) where only the
 +
ranks of the top 100 results for each query in the collection are returned.
  
−
== Evaluation ==
+
=== Human Evaluation ===
−
Similar procedures to the one used last year will be used.
+
 
 +
The primary evaluation will involve subjective judgments by human evaluators of the retrieved sets using IMIRSEL's Evalutron 6000 system. This year algorithms will be presented with the same 30 second preview clip that will be reviewed by the human evaluators.
 +
 
 +
* Evaluator question: Given a search based on track A, the following set of results was returned by all systems. Please place each returned track into one of three classes (not similar, somewhat similar, very similar) and provide an inidcation on a continuous scale of 0 - 10 of high similar the track is to the query.
 +
* ~120 randomly selected queries,  5 results per query, 1 set of eyes, ~10 participating labs
 +
* Higher number of queries preferred as IR research indicates variance is in queries
 +
* The songs by the same artist as the query will be filtered out of each result list (artist-filtering) to avoid colouring an evaluators judgement (a cover song or song by the same artist in a result list is likely to reduce the relative ranking of other similar but independent songs - use of songs by the same artist may allow over-fitting to affect the results)
 +
* It will be possible for researchers to use this data for other types of system comparisons after MIREX 2007 results have been finalized.
 +
* Human evaluation to be designed and led by IMIRSEL following a similar format to that used at MIREX 2006
 +
* Human evaluators will be drawn from the participating labs (and any volunteers from IMIRSEL or on the MIREX lists)
 +
 
 +
=== Objective Statistics derived from the distance matrix ===
 +
 
 +
Statistics of each distance matrix will be calculated including:
 +
 
 +
* Average % of Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Precision at 5, 10, 20 & 50
 +
* Average % of Genre matches in the top 5, 10, 20 & 50 results after artist filtering of results
 +
* Average % of available Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Recall at 5, 10, 20 & 50 (just normalising scores when less than 20 matches for an artist, album or genre are available in the database)
 +
* Always similar - Maximum # times a file was in the top 5, 10, 20 & 50 results
 +
* % File never similar (never in a top 5, 10, 20 & 50 result list)
 +
* % of 'test-able' song triplets where triangular inequality holds
 +
** Note that as we are not requiring full distance matrices this year we will only be testing triangles that are found in the sparse distance matrix.
 +
* Plot of the  "number of times similar curve" -  plot  of song number vs. number of times it appeared in a top 20 list with songs sorted according to number times it appeared in a top 20 list (to produce the curve). Systems with a sharp rise at the end of this plot have "hubs", while a long 'zero' tail shows many never similar results.
 +
 
 +
=== Additional Data Reported ===
 +
 
 +
In addition computation times for feature extraction/Index-building and querying
 +
will be measured.
 +
 
 +
== Submission format ==
 +
Submission to this task will have to conform to a specified format detailed
 +
below.
 +
 
 +
=== Audio formats ===
 +
Participating algorithms will have to read audio in the following format:
 +
 
 +
* Sample rate: 22 KHz
 +
* Sample size: 16 bit
 +
* Number of channels: 1 (mono)
 +
* Encoding: WAV
 +
* clip length: 30 secs from the middle of each file
 +
 
 +
=== Implementation details ===
 +
Scratch folders will be provided for all submissions for the storage of feature files and any model or index files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique filenames will be assigned to each audio track.
 +
 
 +
The audio files to be used in the task will be specified in a simple ASCII list file. This file will contain one path per line with no header line. Executables will have to accept the path to these list files as a command line parameter. The formats for the list files are specified below.
 +
 
 +
Multi-processor compute nodes (2, 4 or 8 cores) will be used to run this task. Hence, participants could attempt to use parrallelism. Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 2, 4 or 8 thread configurations. Single threaded submissions will, of course, be accepted but may be disadvantaged by time constraints.
 +
 
 +
Submissions will have to output either a full distance matrix or a search results file with the top 100 search results for each track in the collection. This list of results will be used to extract the artist-filtered results to present to the human evaluators and will facilitate the computation of the objective statistics.
 +
 
 +
=== I/O formats ===
 +
In this section the input and output files used in this task are described as
 +
are the command line calling format requirements for submissions.
 +
 
 +
==== Audio collection list file (input)====
 +
The list file passed for feature extraction and indexing will be a simple ASCII list file. This file will contain one path per line with no header line, all paths will be absolute (full paths).
 +
 
 +
e.g.
 +
 
 +
  /aDirectory/collectionFolder/b002342.wav
 +
  /aDirectory/collectionFolder/a005921.wav
 +
  ...
 +
 
 +
==== Search results output file ====
 +
 
 +
It is encouraged for participants to generate full distance matrices in the MIREX 2006 format. Please see [2006:Audio_Music_Similarity_and_Retrieval#Output_File]] for details of this format.
 +
 
 +
Alternatively, if computation is a concern, we will accept the sparse format as detailed below.
 +
 
 +
SPARSE FORMAT:
 +
Participating algorithms should perform feature extraction, indexing and output a simple ASCII file listing a name for the algorithm and the top 100 search results for every track in the collection.
 +
 
 +
This file should start with a header line with a name for the algorithm and should be followed by the results for one query per line, prefixed by the  filename portion of the query path. This should be followed by a tab character and a tab separated, ordered list of the top 100 search results. Each result should include the result filename (e.g. a034728.wav) and the distance (e.g. 17.1 or 0.23) separated by a a comma.
 +
 
 +
E.g.
 +
 +
  MyAlgorithm (my.email@address.com)
 +
  <example 1 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance>
 +
  <example 1 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance>
 +
  ...
 +
 
 +
 
 +
which might look like:
 +
 
 +
  MyAlgorithm (my.email@address.com)
 +
  a009342.wav b229311.wav,0.16 a023821.wav,0.19 a001329,0.24  ... etc.
 +
  a009343.wav a661931.wav,0.12 a043322.wav,0.17 c002346,0.21  ... etc.
 +
  a009347.wav a671239.wav,0.13 c112393.wav,0.20 b083293,0.25  ... etc.
 +
  ...
 +
 
 +
The path to which this list file should be written must be accepted as a parameter on the command line.
 +
 
 +
==== Example submission calling formats ====
 +
 
 +
  extractFeatures.sh /path/to/scratch/folder /path/to/collectionListFile.txt
 +
  Query.sh /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt
 +
 
 +
  doAudioSim.sh -numThreads 8 /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt
 +
 
 +
 
 +
=== Packaging submissions ===
 +
All submissions should be statically linked to all libraries (the presence of
 +
dynamically linked libraries cannot be guarenteed).
 +
 
 +
All submissions should include a README file including the following the
 +
information:
 +
 
 +
* Command line calling format for all executables
 +
* Number of threads/cores used or whether this should be specified on the command line
 +
* Expected memory footprint
 +
* Expected runtime
 +
* Any required environments (and versions), e.g. python, java, bash, matlab.
 +
 
 +
== Time and hardware limits ==
 +
Due to the potentially high number of particpants in this and other audio tasks,
 +
hard limits on the runtime of submissions will be specified.
 +
 
 +
A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times).
 +
 
 +
== Submission opening date ==
 +
7th August 2007 - provisional
 +
 
 +
== Submission closing date ==
 +
21st August 2007 - provisional
 +
 
 +
== Past polls ==
 +
 
 +
Polling is more or less closed. The audio format, as above is 22.05kHz mono wavs. The output format is encouraged to be the full distance matrix as per 2006, but the new sparse format will be accepted if need be.
 +
 
 +
=== Audio format poll ===
 +
 
 +
<poll>
 +
Use the same 30 sec clips for analysis by participating algorithms as presented to human evaluators (necessary due to copyright restrictions)?
 +
Yes
 +
No, my algorithm needs longer clips
 +
No, I don't think this necessary
 +
No, for some other reason
 +
</poll>
 +
 
 +
<poll>
 +
What is your preferred audio format? Remember that the less audio data we have to process the larger the dataset can be...
 +
22 khz mono WAV
 +
22 khz stereo WAV
 +
44 khz mono WAV
 +
44 khz stereo WAV
 +
22 khz mono MP3 128kb
 +
22 khz stereo MP3 128kb
 +
44 khz mono MP3 128kb
 +
44 khz stereo MP3 128kb
 +
</poll>
 +
 
 +
=== Return format poll ===
 +
<poll>
 +
Do you prefer to return a full distance matrix (NxN) or a sparse matrix (Nx100)?
 +
Full distance matrix
 +
Sparse Matrix
 +
</poll>
  
−
As Magnatune and USPOP are freely available overfitting is possible. More interesting than the final ranking will be the accompanying papers in which the participants describe their work.
+
Note: If there is no significant demand for a sparse format it will be dropped.
  
−
Please edit this if you have suggestions to add or if you disagree.
+
* I just wanted to say I am mostly ambivalent about this, but lean ever so slightly to the sparse matrix, as it will result in a few less disc writes and I am going to be pushing that 72  hr cap, so any trimming of run time is useful.  That said I do see the value of preserving a more exhaustive dataset so either way...
 +
[[User:Bfields|Bfields]] 08:24, 3 August 2007 (CDT)
  
 
== Participants ==
 
== Participants ==
Line 32: Line 208:
 
* Tim Pohle (''firstname.lastname''@jku.at)
 
* Tim Pohle (''firstname.lastname''@jku.at)
 
* Kris West (kw at cmp dot uea dot ac dot uk)
 
* Kris West (kw at cmp dot uea dot ac dot uk)
−
* Julien Ricard
+
* Julien Ricard (''firstname.lastname''@gmail.com)
 
* Abhinav Singh (abhinavs at iitg.ernet.in) and S.R.M.Prasanna (prasanna at iitg.ernet.in)
 
* Abhinav Singh (abhinavs at iitg.ernet.in) and S.R.M.Prasanna (prasanna at iitg.ernet.in)
−
* ...
+
* Ben Fields (map01bf at gold dot ac dot uk)
 +
* Christoph Bastuck (bsk at idmt.fhg.de)
 +
* Aliaksandr Paradzinets (aliaksandr.paradzinets {at} ec-lyon.fr)
 +
* Vitor Soares (''firstname.lastname''{at} clustermedialabs.com)
 +
* Kai Chen('lastnamefirstname dot dr @gmail.com)
 +
* Kurt Jacobson (''firstname.lastname''@elec.qmul.ac.uk)
 +
* Yi-Hsuan Yang (affige at gmail dot com)
 +
* Luke Barrington (lukeinusa at gmail)
 +
* Douglas Turnbull (dturnbul AT cs.ucsd.edu)
 +
* George Tzanetakis (gtzan at cs dot uvic dot ca)

Latest revision as of 15:57, 13 May 2010

Status

A provisional specification of the Audio Music Similarity task is detailed below. This proposal may be refined based on feedback from the particpants.

Note that audio music similarity and retrieval algorithms have been evaluated at MIREX 2006 ( 2006:Audio_Music_Similarity_and_Retrieval ).

Related MIREX 2007 task proposals:

Please feel free to edit this page.

Data

Collection statistics: 7000 30-second audio clips in 22.05kHz mono wav format drawn from 10 genres (700 clips from each genre). Genres:

  • Blues
  • Jazz
  • Country/Western
  • Baroque
  • Classical
  • Romantic
  • Electronica
  • Hip-Hop
  • Rock
  • HardRock/Metal

Evaluation

Two distinct evaluations will be performed

  • Human Evaluation
  • Objective statistics derived from the results lists

Note that at MIREX 2006 particpating algorithms were required to return full distance matrices showing the distance between all tracks. This year we are also going to allow a sparse distance matrix format (detailed below) where only the ranks of the top 100 results for each query in the collection are returned.

Human Evaluation

The primary evaluation will involve subjective judgments by human evaluators of the retrieved sets using IMIRSEL's Evalutron 6000 system. This year algorithms will be presented with the same 30 second preview clip that will be reviewed by the human evaluators.

  • Evaluator question: Given a search based on track A, the following set of results was returned by all systems. Please place each returned track into one of three classes (not similar, somewhat similar, very similar) and provide an inidcation on a continuous scale of 0 - 10 of high similar the track is to the query.
  • ~120 randomly selected queries, 5 results per query, 1 set of eyes, ~10 participating labs
  • Higher number of queries preferred as IR research indicates variance is in queries
  • The songs by the same artist as the query will be filtered out of each result list (artist-filtering) to avoid colouring an evaluators judgement (a cover song or song by the same artist in a result list is likely to reduce the relative ranking of other similar but independent songs - use of songs by the same artist may allow over-fitting to affect the results)
  • It will be possible for researchers to use this data for other types of system comparisons after MIREX 2007 results have been finalized.
  • Human evaluation to be designed and led by IMIRSEL following a similar format to that used at MIREX 2006
  • Human evaluators will be drawn from the participating labs (and any volunteers from IMIRSEL or on the MIREX lists)

Objective Statistics derived from the distance matrix

Statistics of each distance matrix will be calculated including:

  • Average % of Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Precision at 5, 10, 20 & 50
  • Average % of Genre matches in the top 5, 10, 20 & 50 results after artist filtering of results
  • Average % of available Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Recall at 5, 10, 20 & 50 (just normalising scores when less than 20 matches for an artist, album or genre are available in the database)
  • Always similar - Maximum # times a file was in the top 5, 10, 20 & 50 results
  •  % File never similar (never in a top 5, 10, 20 & 50 result list)
  •  % of 'test-able' song triplets where triangular inequality holds
    • Note that as we are not requiring full distance matrices this year we will only be testing triangles that are found in the sparse distance matrix.
  • Plot of the "number of times similar curve" - plot of song number vs. number of times it appeared in a top 20 list with songs sorted according to number times it appeared in a top 20 list (to produce the curve). Systems with a sharp rise at the end of this plot have "hubs", while a long 'zero' tail shows many never similar results.

Additional Data Reported

In addition computation times for feature extraction/Index-building and querying will be measured.

Submission format

Submission to this task will have to conform to a specified format detailed below.

Audio formats

Participating algorithms will have to read audio in the following format:

  • Sample rate: 22 KHz
  • Sample size: 16 bit
  • Number of channels: 1 (mono)
  • Encoding: WAV
  • clip length: 30 secs from the middle of each file

Implementation details

Scratch folders will be provided for all submissions for the storage of feature files and any model or index files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique filenames will be assigned to each audio track.

The audio files to be used in the task will be specified in a simple ASCII list file. This file will contain one path per line with no header line. Executables will have to accept the path to these list files as a command line parameter. The formats for the list files are specified below.

Multi-processor compute nodes (2, 4 or 8 cores) will be used to run this task. Hence, participants could attempt to use parrallelism. Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 2, 4 or 8 thread configurations. Single threaded submissions will, of course, be accepted but may be disadvantaged by time constraints.

Submissions will have to output either a full distance matrix or a search results file with the top 100 search results for each track in the collection. This list of results will be used to extract the artist-filtered results to present to the human evaluators and will facilitate the computation of the objective statistics.

I/O formats

In this section the input and output files used in this task are described as are the command line calling format requirements for submissions.

Audio collection list file (input)

The list file passed for feature extraction and indexing will be a simple ASCII list file. This file will contain one path per line with no header line, all paths will be absolute (full paths).

e.g. 

  /aDirectory/collectionFolder/b002342.wav
  /aDirectory/collectionFolder/a005921.wav
  ...

Search results output file

It is encouraged for participants to generate full distance matrices in the MIREX 2006 format. Please see [2006:Audio_Music_Similarity_and_Retrieval#Output_File]] for details of this format.

Alternatively, if computation is a concern, we will accept the sparse format as detailed below.

SPARSE FORMAT: Participating algorithms should perform feature extraction, indexing and output a simple ASCII file listing a name for the algorithm and the top 100 search results for every track in the collection.

This file should start with a header line with a name for the algorithm and should be followed by the results for one query per line, prefixed by the filename portion of the query path. This should be followed by a tab character and a tab separated, ordered list of the top 100 search results. Each result should include the result filename (e.g. a034728.wav) and the distance (e.g. 17.1 or 0.23) separated by a a comma.

E.g. 

  MyAlgorithm (my.email@address.com)
  <example 1 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance>
  <example 1 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance>
  ...


which might look like: 

  MyAlgorithm (my.email@address.com)
  a009342.wav	b229311.wav,0.16	a023821.wav,0.19	a001329,0.24  ... etc.
  a009343.wav	a661931.wav,0.12	a043322.wav,0.17	c002346,0.21  ... etc.
  a009347.wav	a671239.wav,0.13	c112393.wav,0.20	b083293,0.25  ... etc.
  ...

The path to which this list file should be written must be accepted as a parameter on the command line.

Example submission calling formats

  extractFeatures.sh /path/to/scratch/folder /path/to/collectionListFile.txt
  Query.sh /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt

  doAudioSim.sh -numThreads 8 /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt


Packaging submissions

All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guarenteed).

All submissions should include a README file including the following the information:

  • Command line calling format for all executables
  • Number of threads/cores used or whether this should be specified on the command line
  • Expected memory footprint
  • Expected runtime
  • Any required environments (and versions), e.g. python, java, bash, matlab.

Time and hardware limits

Due to the potentially high number of particpants in this and other audio tasks, hard limits on the runtime of submissions will be specified.

A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times).

Submission opening date

7th August 2007 - provisional

Submission closing date

21st August 2007 - provisional

Past polls

Polling is more or less closed. The audio format, as above is 22.05kHz mono wavs. The output format is encouraged to be the full distance matrix as per 2006, but the new sparse format will be accepted if need be.

Audio format poll

<poll> Use the same 30 sec clips for analysis by participating algorithms as presented to human evaluators (necessary due to copyright restrictions)? Yes No, my algorithm needs longer clips No, I don't think this necessary No, for some other reason </poll>

<poll> What is your preferred audio format? Remember that the less audio data we have to process the larger the dataset can be... 22 khz mono WAV 22 khz stereo WAV 44 khz mono WAV 44 khz stereo WAV 22 khz mono MP3 128kb 22 khz stereo MP3 128kb 44 khz mono MP3 128kb 44 khz stereo MP3 128kb </poll>

Return format poll

<poll> Do you prefer to return a full distance matrix (NxN) or a sparse matrix (Nx100)? Full distance matrix Sparse Matrix </poll>

Note: If there is no significant demand for a sparse format it will be dropped.

  • I just wanted to say I am mostly ambivalent about this, but lean ever so slightly to the sparse matrix, as it will result in a few less disc writes and I am going to be pushing that 72 hr cap, so any trimming of run time is useful. That said I do see the value of preserving a more exhaustive dataset so either way...

Bfields 08:24, 3 August 2007 (CDT)

Participants

If you think there is a slight chance that you might want to participate please add your name and email address here.

  • Klaas Bosteels (firstname.lastname@gmail.com)
  • Thomas Lidy (lastname@ifs.tuwien.ac.at)
  • Elias Pampalk (firstname.lastname@gmail.com)
  • Tim Pohle (firstname.lastname@jku.at)
  • Kris West (kw at cmp dot uea dot ac dot uk)
  • Julien Ricard (firstname.lastname@gmail.com)
  • Abhinav Singh (abhinavs at iitg.ernet.in) and S.R.M.Prasanna (prasanna at iitg.ernet.in)
  • Ben Fields (map01bf at gold dot ac dot uk)
  • Christoph Bastuck (bsk at idmt.fhg.de)
  • Aliaksandr Paradzinets (aliaksandr.paradzinets {at} ec-lyon.fr)
  • Vitor Soares (firstname.lastname{at} clustermedialabs.com)
  • Kai Chen('lastnamefirstname dot dr @gmail.com)
  • Kurt Jacobson (firstname.lastname@elec.qmul.ac.uk)
  • Yi-Hsuan Yang (affige at gmail dot com)
  • Luke Barrington (lukeinusa at gmail)
  • Douglas Turnbull (dturnbul AT cs.ucsd.edu)
  • George Tzanetakis (gtzan at cs dot uvic dot ca)
Retrieved from "https://music-ir.org/mirex/w/index.php?title=2007:Audio_Music_Similarity_and_Retrieval&oldid=6124"