MIREX Wiki - User contributions [en]

2011:Audio Similarity 2011 Graders

2011-09-28T16:51:22Z

Heywhoah:

=AMS 2011 Graders=

Welcome to the AMS grader sign-up page. Please give us your name and email contact information. If you obscure your email, please make it relatively obvious to us how to parse the address.

Template: Name. Location. <Email>

Sample: J. Stephen Downie. Illinois, USA. <jdownie@illinois.edu>

==Special Comments==
We are under time constraints this year because ISMIR 2011
begins Monday, 24th of October. We need to have all the final
results calculated and posted by the 14th of October target date (fingers crossed).

We hope to open the the Evalutron 6000 (E6K) v.2 grading system by Friday, 30th
September. To meet our goal, we must have all the AMS and SMS
similarity grades entered into the E6K by Wednesday, Oct. 12th. So, if you are kind enough to sign up to be a grader, please understand that we really need you complete your assigned grading
by Wednesday, Oct. 12th.

If you are a SMS or AMS participant, we ask that you do what you can to
encourage adults over 18 years of age to be graders.

We are looking for 50 graders for AMS this year. If we make our quota of 50 graders, each grader will be responsible for two query lists. If we fall short, and get around 34 graders, we will be asking each grader to grade 3 queries. In this worst case scenario, we still expect the grading process to take between 2.5 to 3 hours (or less) for each grader.

This year the query lists seem to be moderate in length so tackling two queries lists should not be too onerous. For safety's sake, we would like to see say, an extra 5 or so names on the sign up sheet below. The "extra names" on the sign up sheet will be considered "back up" graders. We will assign grading tasks in the order of the names as they appear below.

=Sign Up Area=

# Brian McFee. California, USA. <bmcfee@cs.ucsd.edu>
# Steve Tjoa. San Francisco, CA, USA. <steve at imagine-research com>
# Jia-Min Ren. Hsinchu, Taiwan. <jmren at mirlab org>
# Sally Jo Cunningham, Hamilton, New Zealand. <sallyjo@cs.waikato.ac.nz>
# Sungkyun Chang. Suwon, Korea. <rayno1 at snu.ac.kr>
# Yin-Tzu Lin. Taipei, Taiwan. <known at cmlab.csie.ntu.edu.tw>
# Franz de Leon. Southampton, UK. <fadl1d09@ecs.soton.ac.uk>
# Simone Sammartino. Málaga, Spain. <ssammartino@ic.uma.es>
# Arthur Flexer, OFAI, Austria <arthur.flexer at ofai.at>
# Dominik Schnitzer, OFAI, Austria <dominik.schnitzer at ofai.at>
# Jan Schlueter, OFAI, Austria <jan.schlueter at ofai.at>
# Cristina de la Bandera. Málaga, Spain. <cdelabandera@ic.uma.es>
# Bart Stasiak. Lodz, Poland. <basta -@- ics.p.lodz.pl>
# Thierry Bertin-Mahieux. New York, USA. <tb2332@columbia.edu>
# Benjamin Martin. Bordeaux, France. <benjamin.martin@labri.fr>
# Ruofeng Chen. Georgia, USA. <ruofengchen (at) gatech (dot) edu>
# Bo Xie. Atlanta, USA. <bo.xie (at) gatech (dot) edu>
# Ming Li. Beijing, China. <liming.ioa (at) gmail dot com>
# Chung-Che Wang. Hsinchu, Taiwan. <geniusturtle (at) mirlab dot org>
# Peter Knees, cp.jku, Austria. <peter.knees (at) jku.at>
# Markus Schedl, cp.jku, Austria <markus.schedl (at) jku.at>
# Audrey Laplante. Montréal, Canada. <audrey.laplante (at) umontreal.ca>
# Ajay Ramaseshan. Espoo, Finland. <ajayram (at) cis.hut.fi>
# Matt Hoffman. New York, New York, USA. <mdhoffma@cs.princeton.edu>

2010:Audio Similarity 2010 Graders

2010-07-14T01:02:01Z

Heywhoah: /* Sign Up Area */

=AMS 2010 Graders=

Welcome to the AMS grader sign-up page. Please give us your name and email contact information. If you obscure your email, please make it relatively obvious to us how to parse the address.

Template: Name. Location. <Email>

Sample: J. Stephen Downie. Illinois, USA. <jdownie@illinois.edu>

=Special Comments=
We are under extraordinary time constraints this year because ISMIR 2010
is being held starting 9 August and we need to have all the final
results calculated and posted by our 2 August target date (fingers crossed).

We hope to open the the Evalutron 6000 (E6K) v.2 grading system by 20
July. To meet our 2 August goal, we must have all the AMS and SMS
similarity grades entered into the E6K by 27 July. YES, THIS GIVES US
ONLY ONE WEEK! So, if you are kind enough to sign up to be a grader,
please understand that we really need you complete your assigned grading
by 27 July.

If you are a SMS or AMS participant, we ask that you do what you can to
encourage adults over 18 years of age to be graders.

We are looking for 50 graders for AMS this year. Each grader will be responsible for two query lists. This year the query lists seem to be moderate in length so tackling two queries lists should not be too onerous. For safety's sake, we would like to see say, an extra five or so names on the sign up sheet below. The "extra names" on the sign up sheet will be considered "back up" graders. We will assign grading tasks in the order of the names as they appear below.

=Sign Up Area=
Martin Ariel Hartmann. Buenos Aires, Argentina. <martin.hartmann@jyu.fi>

Sally Jo Cunningham. Hamilton, New Zealand. <sallyjo@cs.waikato.ac.nz>

Matt Hoffman. New York, USA. <mdhoffma@cs.princeton.edu>

2010:Audio Music Similarity and Retrieval

2010-06-05T13:22:06Z

Heywhoah: Added link to 2009 page and results.

== Description ==
As the size of digitial music collections grow, music similarity has an increasingly important role as an aid to music discovery. A music similarity system can help a music consumer find new music by finding the music that is most musically similar to specific query songs (or is nearest to songs that the consumer already likes).

This page presents the Audio Music Similarity Evaluation, including the submission rules and formats. Additionally background information can be found here that should help explain some of the reasoning behind the approach taken in the evaluation. The intention of the Music Audio Search track is to evaluate music similarity searches (A music search engine that takes a single song as a query aka Query-by-example), not playlist generation or music recommendation.

The Audio Music Similarity and Retrieval task has been run in MIREX 2009, 2007, and 2006.

[[2009:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2009]] || [[2009:Audio_Music_Similarity_and_Retrieval_Results|Results]]

[[2007:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2007]] || [[2007:Audio_Music_Similarity_and_Retrieval_Results|Results]]

[[2006:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2006]] || [[2006:Audio_Music_Similarity_and_Retrieval_Results|Results]]

=== Task specific mailing list ===
In the past we have use a specific mailing list for the discussion of this task and related tasks (e.g., [[2010:Audio Classification (Train/Test) Tasks]], [[2010:Audio Cover Song Identification]], [[2010:Audio Tag Classification]], [[2010:Audio Music Similarity and Retrieval]]). This year, however, we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.

== Data ==
Collection statistics: 7000 30-second audio clips drawn from 10 genres (700 clips from each genre).

The Genres that data was drawn from are:
*Blues
*Jazz
*Country/Western
*Baroque
*Classical
*Romantic
*Electronica
*Hip-Hop
*Rock
*HardRock/Metal

=== Audio formats ===
Participating algorithms will have to read audio in the following format:

* Sample rate: 22 KHz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV
* clip length: 30 secs from the middle of each file

== Evaluation ==
Two distinct evaluations will be performed
* Human Evaluation
* Objective statistics derived from the results lists

Note that at MIREX 2006 particpating algorithms were required to return full distance matrices showing the distance between all tracks, however, in subsequent years we have also supported sparse distance matrix format (detailed below) where only the distances of the top 100 results for each query in the collection are returned.

=== Human Evaluation ===
The primary evaluation will involve subjective judgments by human evaluators of the retrieved sets using IMIRSEL's Evalutron 6000 system. This year algorithms will be presented with the same 30 second preview clip that will be reviewed by the human evaluators.

* Evaluator question: Given a search based on track A, the following set of results was returned by all systems. Please place each returned track into one of three classes (not similar, somewhat similar, very similar) and provide an inidcation on a continuous scale of 0 - 10 of high similar the track is to the query.
* ~120 randomly selected queries, 5 results per query, 1 set of eyes, ~10 participating labs
* Higher number of queries preferred as IR research indicates variance is in queries
* The songs by the same artist as the query will be filtered out of each result list (artist-filtering) to avoid colouring an evaluators judgement (a cover song or song by the same artist in a result list is likely to reduce the relative ranking of other similar but independent songs - use of songs by the same artist may allow over-fitting to affect the results)
* It will be possible for researchers to use this data for other types of system comparisons after MIREX 2010 results have been finalized.
* Human evaluation to be designed and led by IMIRSEL following a similar format to that used at MIREX 2006 (see: [[2006:Evalutron6000_Issues|Evalutron Issues in MIREX 2006]]).
* Human evaluators will be drawn from the participating labs (and any volunteers from IMIRSEL or on the MIREX lists)

=== Objective Statistics derived from the distance matrix ===
Statistics of each distance matrix will be calculated including:

* Average % of Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Precision at 5, 10, 20 & 50
* Average % of Genre matches in the top 5, 10, 20 & 50 results after artist filtering of results
* Average % of available Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Recall at 5, 10, 20 & 50 (just normalising scores when less than 20 matches for an artist, album or genre are available in the database)
* Always similar - Maximum # times a file was in the top 5, 10, 20 & 50 results
* % File never similar (never in a top 5, 10, 20 & 50 result list)
* % of 'test-able' song triplets where triangular inequality holds
** Note that as we are not requiring full distance matrices this year we will only be testing triangles that are found in the sparse distance matrix.
* Plot of the "number of times similar curve" - plot of song number vs. number of times it appeared in a top 20 list with songs sorted according to number times it appeared in a top 20 list (to produce the curve). Systems with a sharp rise at the end of this plot have "hubs", while a long 'zero' tail shows many never similar results.

=== Runtimes ===
In addition computation times for feature extraction/Index-building and querying
will be measured.

== Submission format ==
Submission to this task will have to conform to a specified format detailed below.

=== Implementation details ===
Scratch folders will be provided for all submissions for the storage of feature files and any model or index files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique filenames will be assigned to each audio track.

The audio files to be used in the task will be specified in a simple ASCII list file. This file will contain one path per line with no header line. Executables will have to accept the path to these list files as a command line parameter. The formats for the list files are specified below.

Multi-processor compute nodes (2, 4 or 8 cores) will be used to run this task. Hence, participants could attempt to use parrallelism. Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 2, 4 or 8 thread configurations. Single threaded submissions will, of course, be accepted but may be disadvantaged by time constraints.

Submissions will have to output either a full distance matrix or a search results file with the top 100 search results for each track in the collection. This list of results will be used to extract the artist-filtered results to present to the human evaluators and will facilitate the computation of the objective statistics.

=== I/O formats ===
In this section the input and output files used in this task are described as
are the command line calling format requirements for submissions.

==== Audio collection list file (input)====
The list file passed for feature extraction and indexing will be a simple ASCII list file. This file will contain one path per line with no header line, all paths will be absolute (full paths).

e.g.

/aDirectory/collectionFolder/b002342.wav
/aDirectory/collectionFolder/a005921.wav
...

==== Distance matrix output files ====
Participants should return one of two available output file formats, a full distance matrix or a sparse distance matrix. The sparse distance matrix format is preferred (as the dense distance matrices can be very large).

===== Sparse Distance Matrix =====
If computation or exhaustive search is a concern or not a normal output of the indexing algorithm employed, the sparse distance matric format detailed below may be used:

A simple ASCII file listing a name for the algorithm and the top 100 search results for every track in the collection.

This file should start with a header line with a name for the algorithm and should be followed by the results for one query per line, prefixed by the filename portion of the query path. This should be followed by a tab character and a tab separated, ordered list of the top 100 search results. Each result should include the result filename (e.g. a034728.wav) and the distance (e.g. 17.1 or 0.23) separated by a a comma.

<pre>
MyAlgorithm (my.email@address.com)
<example 1 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance>
<example 2 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance>
...
</pre>

which might look like:

<pre>
MyAlgorithm (my.email@address.com)
a009342.wav b229311.wav,0.16 a023821.wav,0.19 a001329,0.24 ... etc.
a009343.wav a661931.wav,0.12 a043322.wav,0.17 c002346,0.21 ... etc.
a009347.wav a671239.wav,0.13 c112393.wav,0.20 b083293,0.25 ... etc.
...
</pre>

The path to which this list file should be written must be accepted as a parameter on the command line.

===== Full Distance Matrix =====
Full distance matrix files should be generated in the the following format:

* A simple ASCII file listing a name for the algorithm on the first line,
* Numbered paths for each file appearing in the matrix, these can be in any order (i.e. the files don't have to be i the same order as they appeared in the list file) but should index into the columns/rows of of the distance matrix.
* A line beginning with 'Q/R' followed by a tab and tab separated list of the numbers 1 to N, where N is the files covered by the matrix.
* One line per file in the matrix give the distances of that files to each other file in the matrix. All distances should be zero or positive (0.0+) and should not be infinite or NaN. Values should be separated by a single tab character. Obviously the diagonal of the matrix (distance or a track to itself) should be zero.

<pre>
Distance matrix header text with system name
1\t</path/to/audio/file/1.wav>
2\t</path/to/audio/file/2.wav>
3\t</path/to/audio/file/3.wav>
...
N\t</path/to/audio/file/N.wav>
Q/R\t1\t2\t3\t...\tN
1\t0.0\t<dist 1 to 2>\t<dist 1 to 3>\t...\t<dist 1 to N>
2\t<dist 2 to 1>\t0.0\t<dist 2 to 3>\t...\t<dist 2 to N>
3\t<dist 3 to 2>\t<dist 3 to 2>\t0.0\t...\t<dist 3 to N>
...\t...\t...\t...\t...\t...
N\t<dist N to 1>\t<dist N to 2>\t<dist N to 3>\t...\t0.0
</pre>

which might look like:

<pre>
Example distance matrix 0.1
1 /path/to/audio/file/1.wav
2 /path/to/audio/file/2.wav
3 /path/to/audio/file/3.wav
4 /path/to/audio/file/4.wav
Q/R 1 2 3 4
1 0.00000 1.24100 0.2e-4 0.42559
2 1.24100 0.00000 0.62640 0.23564
3 50.2e-4 0.62640 0.00000 0.38000
4 0.42559 0.23567 0.38000 0.00000
</pre>

==== Example submission calling formats ====
extractFeatures.sh /path/to/scratch/folder /path/to/collectionListFile.txt
Query.sh /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt

or

doAudioSim.sh -numThreads 8 /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt

=== Packaging submissions ===
All submissions should be statically linked to all libraries (the presence of
dynamically linked libraries cannot be guarenteed).

All submissions should include a README file including the following the
information:

* Command line calling format for all executables and an example formatted set of commands
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Any required environments (and versions), e.g. python, java, bash, matlab.

== Time and hardware limits ==
Due to the potentially high number of particpants in this and other audio tasks,
hard limits on the runtime of submissions are specified.

A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.

== Submission opening date ==

Friday 4th June 2010

== Submission closing date ==
TBA