MIREX Wiki - User contributions [en]

MIREX Impact Citations

2011-05-12T08:16:55Z

Bfields: /* Submitted Citations */

=Introduction=

Dear MIREX Participant:

We are gathering evidence of impact for MIREX. We would appreciate it greatly if you would post citations to papers (yours or others) that have used MIREX data and/or results. Any acceptable citation format is OK. DOIs or URL to accessible copies especially appreciated. I have started out with some samples from IMIRSEL.

In a similar vein, we are also gathering citations to MIREX-related comments as further evidence of impact for MIREX on the [[MIREX Impact Statements]] page.

We are collecting these statements and citations as evidence influence and success to submit to funding agencies, research administrators and/or future collaborators.

Thank you very much.

J. Stephen Downie 
Director, IMIRSEL 

=Submitted Citations=

# J.S. Downie, D. Byrd, and T. Crawford. "Ten Years of ISMIR: Reflections on Challenges and Opportunities", in the 10th International Symposium on Music Information Retrieval (ISMIR), Kobe, 2009, pp. 13-18. Available: http://ismir2009.ismir.net/proceedings/keynote1.pdf
# M. Bay, A.F. Ehmann and J.S. Downie, "Evaluation of Multiple-F0 Estimation and Tracking Systems", in the 10th International Symposium on Music Information Retrieval (ISMIR), Kobe, 2009, pp. 315-320. Available: http://ismir2009.ismir.net/proceedings/PS2-21.pdf
# E. Law, K. West, M. Mandel, M. Bay and, J. S. Downie, "Evaluation of Algorithms Using Games: The Case of Music Tagging", in the 10th International Symposium on Music Information Retrieval (ISMIR), Kobe, 2009, pp. 387-392. Available: http://ismir2009.ismir.net/proceedings/OS5-5.pdf
# J. S. Downie, M. Bay, A. F. Ehmann and M. C. Jones, "Audio Cover Song Identification: MIREX 2006-2007 Results and analysis", in the 9th International Conference on Music Information Retrieval (ISMIR 2008), Philadelphia, 2008, pp. 468-473. Available: http://ismir2008.ismir.net/papers/ISMIR2008_265.pdf
# X. Hu, J. S. Downie, C. Laurier, M. Bay and A. F. Ehmann, "The 2007 MIREX Audio Mood Classification Task: Lessons learned", in the 9th International Conference on Music Information Retrieval (ISMIR 2008), Philadelphia, 2008, pp. 462-467. Available: http://ismir2008.ismir.net/papers/ISMIR2008_263.pdf
# J. S. Downie, A. F. Ehmann and J. H. Lee, "The Music Information Retrieval Evaluation eXchange (MIREX): Community-led formal evaluations", in the Digital Humanities 2008 , Oulu Finland, 2008, pp. 239-240.
# M. McVicar and T. De Bie, "Exploiting Online Resources to Improve Chord Recognition Accuracy", late-breaking abstract in the 11th International Conference on Music Information Retrieval (ISMIR 2010), Utrecht, Netherlands, 2010.
# M. McVicar and T. De Bie, "Enhancing chord recognition accuracy using web resources", 3rd ACM Workshop on Machine Learning and Music, October 25, 2010, Firenze, Italy.
# M. McVicar, Y. Ni, R. Santos-Rodriguez, T. De Bie, "Using online chord databases to enhance chord recognition", Journal of New Music Research, Special issue on Music and Machine Learning (in press).
#R. Zhou , M. Mattavelli and G. Zoia, "Music Onset Detection Based on Resonator Time-frequency Image", IEEE Transactions On Audio, Speech And Language Processing, vol. 16, num. 8, 2008, p. 1685-1695.
#Ruohua Zhou, Joshua D. Reiss, Marco Mattavelli and Giorgio Zoia, "A Computationally Efficient Method for Polyphonic Pitch Estimation", EURASIP Journal on Advances in Signal Processing, Volume 2009 (2009), Article ID 729494, 11 pages.
# M. Haro and P. Herrera, “From Low-level to Song-level Percussion Descriptors of Polyphonic Music,” in 10th International Society for Music Information Retrieval Conference (ISMIR 2009), Kobe, Japan, 2009. Available: http://ismir2009.ismir.net/proceedings/PS2-9.pdf
# J. Urbano, J. Morato, M. Marrero and D. Martín, "Crowdsourcing Preference Judgments for Evaluation of Music Similarity Tasks", ACM SIGIR Workshop on Crowdsourcing for Search Evaluation, pp. 9-16, 2010. [http://julian-urbano.info/publications/012-crowdsourcing-preference-judgments-evaluation-music-similarity-tasks/012-paper.pdf PDF]
# J. Urbano, M. Marrero, D. Martín and J. Lloréns, "Improving the Generation of Ground Truths based on Partially Ordered Lists", International Society for Music Information Retrieval Conference, pp. 285-290, 2010. [http://julian-urbano.info/publications/010-improving-generation-ground-truths-based-partially-ordered-lists/010-paper.pdf PDF]
# J. Urbano, J. Lloréns, J. Morato and S. Sánchez-Cuadrado, "Using the Shape of Music to Compute the Similarity between Symbolic Musical Pieces", International Symposium on Computer Music Modeling and Retrieval, pp. 385-396, 2010. [http://julian-urbano.info/publications/009-using-shape-music-compute-similarity-between-symbolic-musical-pieces/009-paper.pdf PDF]
# D. Bogdanov, J. Serrà, N. Wack, P. Herrera and X. Serra. Unifying low-level and high-level music similarity measures. IEEE Transactions on Multimedia. In press. [http://mtg.upf.edu/node/1689]
# J. Wu, E. Vincent, S. Raczynski, T. Nishimoto, N. Ono and S. Sagayama. Multipitch estimation by joint modeling of harmonic and transient sounds. In Proc. 2011 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), to appear. [http://hal.inria.fr/inria-00567175/PDF/wu_ICASSP11.pdf]
# E. Vincent, N. Bertin and R. Badeau. Adaptive harmonic spectral decomposition for multiple pitch estimation. IEEE Transactions on Audio, Speech and Language Processing, 18(3), pp. 528-537, 2010. [http://hal.inria.fr/inria-00544094/PDF/vincent_TASLP10.pdf]
# E. Benetos and S. Dixon. Multiple-instrument polyphonic music transcription using a convolutive probabilistic model. In Proc. 8th Sound and Music Computing Conf. (SMC), to appear.
# B. Fields, K.Jacobson, C. Rhodes, M. d'Inverno, M. Sandler, M. Casey, "Analysis and Exploitation of Musician Social Networks for Recommendation and Discovery," IEEE Transactions on Multimedia. in press. [http://dx.doi.org/10.1109/TMM.2011.2111365]
# B. Fields. Contextualize Your Listening: The Playlist as Recommendation Engine, PhD Thesis, Goldsmiths, University of London, April 2011. [http://benfields.net/bfields_thesis.pdf]
# B. Fields, K. Page, T. Crawford, D. De Roure, "The Segment Ontology: bridging music-generic and domain-specific," in Workshop on Advances in Music Information Research (AdMIRe), co-located with the IEEE International Conference on Multimedia and Expo (ICME), Barcelona, Spain, July, 2011. [http://benfields.net/papers/admire2011.pdf]

Audio Music Similarity and Retrieval

2010-11-20T16:59:56Z

Bfields: /* Participation in previous years */

== Description ==

As the size of digital music collections grow,
music similarity has an increasingly important role as an aid to
music discovery. A music similarity system can help a music consumer
find new music by finding the music that is most musically similar to
specific query songs (or is nearest to songs that the consumer
already likes).

This page presents the Audio Music Similarity
Evaluation, including the submission rules and formats. Additionally
background information can be found here that should help explain
some of the reasoning behind the approach taken in the evaluation.
The intention of the Music Audio Search track is to evaluate music
similarity searches (A music search engine that takes a single song
as a query aka Query-by-example), not playlist generation or music
recommendation.

== Participation in previous years ==

{| border="1"
|-
|
'''Year'''
|
'''Participating Algorithms '''
|
'''URL'''
|-
|
2010
|
8
|
https://www.music-ir.org/mirex/wiki/2010:Audio_Music_Similarity_and_Retrieval_Results
|-
|
2009
|
15
|
https://www.music-ir.org/mirex/wiki/2009:Audio_Music_Similarity_and_Retrieval_Results
|-
|
2007
|
12
|
https://www.music-ir.org/mirex/wiki/2007:Audio_Music_Similarity_and_Retrieval_Results
|-
|
2006
|
6
|
https://www.music-ir.org/mirex/wiki/2006:Audio_Music_Similarity_and_Retrieval_Results
|}

2010:Audio Music Similarity and Retrieval Results

2010-09-15T13:47:56Z

Bfields: /* Team ID */

== Introduction ==
These are the results for the 2010 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.

Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 100 songs were randomly selected from the 10 genre groups (10 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical '''BROAD''' score with 3 categories: NS,SS,VS as explained below, and one '''FINE''' score (in the range from 0 to 100). A description and analysis is provided below.

The systems read in 30 second audio clips as their raw data. The same 30 second clips were used in the grading stage.

=== General Legend ===

==== Team ID ====
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BWL1
| MTG-AMS || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/BWL1.pdf PDF] || [http://mtg.upf.edu Dmitry Bogdanov], [http://mtg.upf.edu Joan Serrà], [http://mtg.upf.edu Nicolas Wack], [http://mtg.upf.edu Perfecto Herrera]
|-
! PS1
| PS09 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2009/PS.pdf PDF] || [http://www.cp.jku.at/ Tim Pohle], [http://www.cp.jku.at/ Dominik Schnitzer]
|-
! PSS1
| PSS10 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/PSS1.pdf PDF] || [http://www.cp.jku.at/ Tim Pohle], [http://www.cp.jku.at Klaus Seyerlehner], [http://www.cp.jku.at/ Dominik Schnitzer]
|-
! RZ1
| RND || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RZ1.pdf PDF] || [http://www.cp.jku.at Rainer Zufall]
|-
! SSPK2
| cbmr_sim || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/SSPK2.pdf PDF] || [http://www.cp.jku.at Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Tim Pohle], [http://www.cp.jku.at Peter Knees]
|-
! TLN1
| MarsyasSimilarity || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TNL1.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://sness.net Steven Ness], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange]
|-
! TLN2
| Post-Processing 1 of Marsyas similarity results || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TLN1.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange], [http://sness.net Steven Ness]
|-
! TLN3
| Post-Processing 2 of Marsyas similarity results || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TLN2.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange], [http://sness.net Steven Ness]
|}

====Broad Categories====
'''NS''' = Not Similar 
'''SS''' = Somewhat Similar 
'''VS''' = Very Similar 

=====Understanding Summary Measures=====
'''Fine''' = Has a range from 0 (failure) to 100 (perfection). 
'''Broad''' = Has a range from 0 (failure) to 2 (perfection) as each query/candidate pair is scored with either NS=0, SS=1 or VS=2. 

==Human Evaluation==
===Overall Summary Results===

<csv p=3>2010/ams/AMS2010summary_evalutron.csv</csv>
 
'''Note:RZ1''' is the random result for comparing purpose.

===Friedman's Tests===
====Friedman's Test (FINE Scores)====
The Friedman test was run in MATLAB against the '''Fine''' summary data over the 100 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=3>2010/ams/evalutron.fine.friedman.tukeyKramerHSD.csv</csv>

[[File:2010AMS.evalutron.fine.friedman.tukeyKramerHSD.png|500px]]

====Friedman's Test (BROAD Scores)====
The Friedman test was run in MATLAB against the '''BROAD''' summary data over the 100 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=3>2010/ams/evalutron.cat.friedman.tukeyKramerHSD.csv</csv>

[[File:2010AMS.evalutron.cat.friedman.tukeyKramerHSD.png|500px]]

===Summary Results by Query===
====FINE Scores====
These are the mean FINE scores per query assigned by Evalutron graders. The FINE scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 and 100. A perfect score would be 100. Genre labels have been included for reference.

<csv p=1>2010/ams/fine_scores.csv</csv>

====BROAD Scores====
These are the mean BROAD scores per query assigned by Evalutron graders. The BROAD scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 (not similar) and 2 (very similar). A perfect score would be 2. Genre labels have been included for reference.

<csv p=1>2010/ams/cat_scores.csv</csv>

===Raw Scores===
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2010:Audio Music Similarity and Retrieval Raw Data]] page.

==Metadata and Distance Space Evaluation==
The following reports provide evaluation statistics based on analysis of the distance space and metadata matches and include:
* Neighbourhood clustering by artist, album and genre
* Artist-filtered genre clustering
* How often the triangular inequality holds
* Statistics on 'hubs' (tracks similar to many tracks) and orphans (tracks that are not similar to any other tracks at N results).

=== Reports ===

'''BWL1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/BWL1/report.txt Dmitry Bogdanov, Joan Serrà, Nicolas Wack, Perfecto Herrera] 
'''PS1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/PS1/report.txt Tim Pohle, Dominik Schnitzer] 
'''PSS1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/PSS1/report.txt Tim Pohle, Klaus Seyerlehner, Dominik Schnitzer] 
'''RZ1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/RZ1/report.txt Rainer Zufall] 
'''SSPK2''' = [https://music-ir.org/mirex/results/2010/ams/statistics/SSPK2/report.txt Klaus Seyerlehner, Markus Schedl, Tim Pohle, Peter Knees] 
'''TLN1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN1/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
'''TLN2''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN2/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
'''TLN3''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN3/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 

== Run Times ==
<csv>2010/ams/audiosim.runtime.csv</csv>

2011:MIREX Home

2010-08-30T16:42:52Z

Bfields: /* Welcome to MIREX 2011 */

==Welcome to MIREX 2011==
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2011.

The MIREX 2011 community will hold its annual meeting as part of [http://ismir2011.ismir.net/ The 12th International Conference on Music Information Retrieval], ISMIR 2011, which will be held in Miami, Florida, the week of October the 23rd, 2011. The MIREX plenary (working lunch) and poster sessions will be held at a time to be determined during the conference.

J. Stephen Downie 
Director, IMIRSEL 

===MIREX 2011 Evaluation Tasks===

* [[2011:Audio Classification (Train/Test) Tasks]], incorporating:
** Audio Artist Identification
** Audio US Pop Genre Classification
** Audio Latin Genre Classification
** Audio Music Mood Classification
** Audio Classical Composer Identification
* [[2011:Audio Cover Song Identification]]
* [[2011:Audio Tag Classification]]
* [[2011:Audio Music Similarity and Retrieval]]
* [[2011:Symbolic Melodic Similarity]]
* [[2011:Audio Onset Detection]]
* [[2011:Audio Key Detection]]
* [[2011:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2011:Query by Singing/Humming]]
* [[2011:Audio Melody Extraction]]
* [[2011:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2011:Audio Chord Estimation]]
* [[2011:Query by Tapping]]
* [[2011:Audio Beat Tracking]]
* [[2011:Structural Segmentation]]
* [[2011:Audio Tempo Estimation]]

===Note to New Participants===
Please take the time to read the following review article that explains the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

===Note to All Participants===
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2011 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)
# present a poster at the MIREX 2011 poster session at ISMIR 2011 (Wednesday, 11 August 2011)

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you.

Due to the high volume of submissions expected at MIREX 2011, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2011==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2011 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2011 tasks. Subscription information at:
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central].

If you are participating in MIREX 2011, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2011 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2011 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
Please create an account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2010 Wikis==
Content from MIREX 2005 - 2010 are available at:

'''[[2009:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

2011:Audio Music Similarity and Retrieval with Web access

2010-08-25T16:09:05Z

Bfields: initial write up

2011 talk:Audio Music Similarity and Retrieval

2010-08-25T11:31:13Z

Bfields: Created page with 'The new spin off task I'm proposing for next year (2011:Audio_Music_Similarity_and_Retrieval_with_Web_access) should almost certainly be merged through with this task (as th…'

The new spin off task I'm proposing for next year ([[2011:Audio_Music_Similarity_and_Retrieval_with_Web_access]]) should almost certainly be merged through with this task (as the evaluation will probably be done in unison and the i/o formats will be the same)

--[[User:Bfields|Bfields]] 11:31, 25 August 2010 (UTC)

2011:Audio Music Similarity and Retrieval

2010-08-25T11:24:49Z

Bfields: date updates and an addition of the pre flag up top

<pre>Note: I just moved this over from last year's task write up as a starting point - bfields </pre>
== Description ==

As the size of digitial music collections grow, music similarity has an increasingly important role as an aid to music discovery. A music similarity system can help a music consumer find new music by finding the music that is most musically similar to specific query songs (or is nearest to songs that the consumer already likes).

This page presents the Audio Music Similarity Evaluation, including the submission rules and formats. Additionally background information can be found here that should help explain some of the reasoning behind the approach taken in the evaluation. The intention of the Music Audio Search track is to evaluate music similarity searches (A music search engine that takes a single song as a query aka Query-by-example), not playlist generation or music recommendation.

The Audio Music Similarity and Retrieval task has been run in MIREX 2010, 2009, 2007, and 2006.

[[2010:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2010]] || [[2010:Audio_Music_Similarity_and_Retrieval_Results|Results]]

[[2009:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2009]] || [[2009:Audio_Music_Similarity_and_Retrieval_Results|Results]]

[[2007:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2007]] || [[2007:Audio_Music_Similarity_and_Retrieval_Results|Results]]

[[2006:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2006]] || [[2006:Audio_Music_Similarity_and_Retrieval_Results|Results]]

=== Task specific mailing list ===
In continuing with the changes that took place last year, task specific lists have been dropped. Instead we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.

== Data ==
Collection statistics: 7000 30-second audio clips drawn from 10 genres (700 clips from each genre).

The Genres that data was drawn from are:
*Blues
*Jazz
*Country/Western
*Baroque
*Classical
*Romantic
*Electronica
*Hip-Hop
*Rock
*HardRock/Metal

=== Audio formats ===
Participating algorithms will have to read audio in the following format:

* Sample rate: 22 KHz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV
* clip length: 30 secs from the middle of each file

== Evaluation ==
Two distinct evaluations will be performed
* Human Evaluation
* Objective statistics derived from the results lists

Note that at MIREX 2006 particpating algorithms were required to return full distance matrices showing the distance between all tracks, however, in subsequent years we have also supported sparse distance matrix format (detailed below) where only the distances of the top 100 results for each query in the collection are returned.

=== Human Evaluation ===
The primary evaluation will involve subjective judgments by human evaluators of the retrieved sets using IMIRSEL's Evalutron 6000 system. This year algorithms will be presented with the same 30 second preview clip that will be reviewed by the human evaluators.

* Evaluator question: Given a search based on track A, the following set of results was returned by all systems. Please place each returned track into one of three classes (not similar, somewhat similar, very similar) and provide an inidcation on a continuous scale of 0 - 10 of high similar the track is to the query.
* ~120 randomly selected queries, 5 results per query, 1 set of ears, ~10 participating labs
* Higher number of queries preferred as IR research indicates variance is in queries
* The songs by the same artist as the query will be filtered out of each result list (artist-filtering) to avoid colouring an evaluators judgement (a cover song or song by the same artist in a result list is likely to reduce the relative ranking of other similar but independent songs - use of songs by the same artist may allow over-fitting to affect the results)
* It will be possible for researchers to use this data for other types of system comparisons after MIREX 2011 results have been finalized.
* Human evaluation to be designed and led by IMIRSEL following a similar format to that used at MIREX 2006 (see: [[2006:Evalutron6000_Issues|Evalutron Issues in MIREX 2006]]).
* Human evaluators will be drawn from the participating labs (and any volunteers from IMIRSEL or on the MIREX lists)

=== Objective Statistics derived from the distance matrix ===
Statistics of each distance matrix will be calculated including:

* Average % of Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Precision at 5, 10, 20 & 50
* Average % of Genre matches in the top 5, 10, 20 & 50 results after artist filtering of results
* Average % of available Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Recall at 5, 10, 20 & 50 (just normalising scores when less than 20 matches for an artist, album or genre are available in the database)
* Always similar - Maximum # times a file was in the top 5, 10, 20 & 50 results
* % File never similar (never in a top 5, 10, 20 & 50 result list)
* % of 'test-able' song triplets where triangular inequality holds
** Note that as we are not requiring full distance matrices this year we will only be testing triangles that are found in the sparse distance matrix.
* Plot of the "number of times similar curve" - plot of song number vs. number of times it appeared in a top 20 list with songs sorted according to number times it appeared in a top 20 list (to produce the curve). Systems with a sharp rise at the end of this plot have "hubs", while a long 'zero' tail shows many never similar results.

=== Runtimes ===
In addition computation times for feature extraction/Index-building and querying
will be measured.

== Submission format ==
Submission to this task will have to conform to a specified format detailed below.

=== Implementation details ===
Scratch folders will be provided for all submissions for the storage of feature files and any model or index files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique filenames will be assigned to each audio track.

The audio files to be used in the task will be specified in a simple ASCII list file. This file will contain one path per line with no header line. Executables will have to accept the path to these list files as a command line parameter. The formats for the list files are specified below.

Multi-processor compute nodes (2, 4 or 8 cores) will be used to run this task. Hence, participants could attempt to use parrallelism. Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 2, 4 or 8 thread configurations. Single threaded submissions will, of course, be accepted but may be disadvantaged by time constraints.

Submissions will have to output either a full distance matrix or a search results file with the top 100 search results for each track in the collection. This list of results will be used to extract the artist-filtered results to present to the human evaluators and will facilitate the computation of the objective statistics.

=== I/O formats ===
In this section the input and output files used in this task are described as
are the command line calling format requirements for submissions.

==== Audio collection list file (input)====
The list file passed for feature extraction and indexing will be a simple ASCII list file. This file will contain one path per line with no header line, all paths will be absolute (full paths).

e.g.

/aDirectory/collectionFolder/b002342.wav
/aDirectory/collectionFolder/a005921.wav
...

==== Distance matrix output files ====
Participants should return one of two available output file formats, a full distance matrix or a sparse distance matrix. The sparse distance matrix format is preferred (as the dense distance matrices can be very large).

===== Sparse Distance Matrix =====
If computation or exhaustive search is a concern or not a normal output of the indexing algorithm employed, the sparse distance matric format detailed below may be used:

A simple ASCII file listing a name for the algorithm and the top 100 search results for every track in the collection.

This file should start with a header line with a name for the algorithm and should be followed by the results for one query per line, prefixed by the filename portion of the query path. This should be followed by a tab character and a tab separated, ordered list of the top 100 search results. Each result should include the result filename (e.g. a034728.wav) and the distance (e.g. 17.1 or 0.23) separated by a a comma.

<pre>
MyAlgorithm (my.email@address.com)
<example 1 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance>
<example 2 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance>
...
</pre>

which might look like:

<pre>
MyAlgorithm (my.email@address.com)
a009342.wav b229311.wav,0.16 a023821.wav,0.19 a001329,0.24 ... etc.
a009343.wav a661931.wav,0.12 a043322.wav,0.17 c002346,0.21 ... etc.
a009347.wav a671239.wav,0.13 c112393.wav,0.20 b083293,0.25 ... etc.
...
</pre>

The path to which this list file should be written must be accepted as a parameter on the command line.

===== Full Distance Matrix =====
Full distance matrix files should be generated in the the following format:

* A simple ASCII file listing a name for the algorithm on the first line,
* Numbered paths for each file appearing in the matrix, these can be in any order (i.e. the files don't have to be i the same order as they appeared in the list file) but should index into the columns/rows of of the distance matrix.
* A line beginning with 'Q/R' followed by a tab and tab separated list of the numbers 1 to N, where N is the files covered by the matrix.
* One line per file in the matrix give the distances of that files to each other file in the matrix. All distances should be zero or positive (0.0+) and should not be infinite or NaN. Values should be separated by a single tab character. Obviously the diagonal of the matrix (distance or a track to itself) should be zero.

<pre>
Distance matrix header text with system name
1\t</path/to/audio/file/1.wav>
2\t</path/to/audio/file/2.wav>
3\t</path/to/audio/file/3.wav>
...
N\t</path/to/audio/file/N.wav>
Q/R\t1\t2\t3\t...\tN
1\t0.0\t<dist 1 to 2>\t<dist 1 to 3>\t...\t<dist 1 to N>
2\t<dist 2 to 1>\t0.0\t<dist 2 to 3>\t...\t<dist 2 to N>
3\t<dist 3 to 2>\t<dist 3 to 2>\t0.0\t...\t<dist 3 to N>
...\t...\t...\t...\t...\t...
N\t<dist N to 1>\t<dist N to 2>\t<dist N to 3>\t...\t0.0
</pre>

which might look like:

<pre>
Example distance matrix 0.1
1 /path/to/audio/file/1.wav
2 /path/to/audio/file/2.wav
3 /path/to/audio/file/3.wav
4 /path/to/audio/file/4.wav
Q/R 1 2 3 4
1 0.00000 1.24100 0.2e-4 0.42559
2 1.24100 0.00000 0.62640 0.23564
3 50.2e-4 0.62640 0.00000 0.38000
4 0.42559 0.23567 0.38000 0.00000
</pre>

==== Example submission calling formats ====
extractFeatures.sh /path/to/scratch/folder /path/to/collectionListFile.txt
Query.sh /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt

or

doAudioSim.sh -numThreads 8 /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt

=== Packaging submissions ===
All submissions should be statically linked to all libraries (the presence of
dynamically linked libraries cannot be guarenteed).

All submissions should include a README file including the following the
information:

* Command line calling format for all executables and an example formatted set of commands
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Any required environments (and versions), e.g. python, java, bash, matlab.

== Time and hardware limits ==
Due to the potentially high number of particpants in this and other audio tasks,
hard limits on the runtime of submissions are specified.

A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.

== AMS evaluation software ==
The legacy software for performing various AMS related function is available [https://www.music-ir.org/mirex/results/2010/AMS_tools.zip here], [https://www.music-ir.org/mirex/results/2010/AMS_TOOLS_README%20.txt README file]. It maybe used to benchmark systems prior to submission and to check distance matrix file formats.

This tool set supports the following functions:
* the import of collection metadata from a delimited text file (e.g. TAB or CSV)
* the selection of a stratified random list of queries from the collection (i.e. an equal number of queries are chosen for each class of a particular metadata field, such as genre).
* the generation of results from distance matrices based on a list of pre-chosen queries.
* (pseudo-)objective statistical evaluation of distance matrices by comparing query metadata to the metadata of the top N results retrieved. Supports artist, album, genre and artist-filtered genre (where results form the same artist as query are skipped). Additionally, the number tracks never returned as results for all possible queries (orphans) and the largest hub (track similar to the most other tracks) are measured. Finally, the number of cases where the triangular inequality holds.
* preparation and post processing of results for the IMIRSEL Evalutron 6k human evaluation interface.

== Submission opening date ==

TBA

== Submission closing date ==
TBA

2011:Audio Music Similarity and Retrieval

2010-08-25T11:12:39Z

Bfields: Brought in page from MIREX 2010

== Description ==
As the size of digitial music collections grow, music similarity has an increasingly important role as an aid to music discovery. A music similarity system can help a music consumer find new music by finding the music that is most musically similar to specific query songs (or is nearest to songs that the consumer already likes).

This page presents the Audio Music Similarity Evaluation, including the submission rules and formats. Additionally background information can be found here that should help explain some of the reasoning behind the approach taken in the evaluation. The intention of the Music Audio Search track is to evaluate music similarity searches (A music search engine that takes a single song as a query aka Query-by-example), not playlist generation or music recommendation.

The Audio Music Similarity and Retrieval task has been run in MIREX 2009, 2007, and 2006.

[[2009:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2009]] || [[2009:Audio_Music_Similarity_and_Retrieval_Results|Results]]

[[2007:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2007]] || [[2007:Audio_Music_Similarity_and_Retrieval_Results|Results]]

[[2006:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2006]] || [[2006:Audio_Music_Similarity_and_Retrieval_Results|Results]]

=== Task specific mailing list ===
In the past we have use a specific mailing list for the discussion of this task and related tasks (e.g., [[2010:Audio Classification (Train/Test) Tasks]], [[2010:Audio Cover Song Identification]], [[2010:Audio Tag Classification]], [[2010:Audio Music Similarity and Retrieval]]). This year, however, we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.

== Data ==
Collection statistics: 7000 30-second audio clips drawn from 10 genres (700 clips from each genre).

The Genres that data was drawn from are:
*Blues
*Jazz
*Country/Western
*Baroque
*Classical
*Romantic
*Electronica
*Hip-Hop
*Rock
*HardRock/Metal

=== Audio formats ===
Participating algorithms will have to read audio in the following format:

* Sample rate: 22 KHz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV
* clip length: 30 secs from the middle of each file

== Evaluation ==
Two distinct evaluations will be performed
* Human Evaluation
* Objective statistics derived from the results lists

Note that at MIREX 2006 particpating algorithms were required to return full distance matrices showing the distance between all tracks, however, in subsequent years we have also supported sparse distance matrix format (detailed below) where only the distances of the top 100 results for each query in the collection are returned.

=== Human Evaluation ===
The primary evaluation will involve subjective judgments by human evaluators of the retrieved sets using IMIRSEL's Evalutron 6000 system. This year algorithms will be presented with the same 30 second preview clip that will be reviewed by the human evaluators.

* Evaluator question: Given a search based on track A, the following set of results was returned by all systems. Please place each returned track into one of three classes (not similar, somewhat similar, very similar) and provide an inidcation on a continuous scale of 0 - 10 of high similar the track is to the query.
* ~120 randomly selected queries, 5 results per query, 1 set of eyes, ~10 participating labs
* Higher number of queries preferred as IR research indicates variance is in queries
* The songs by the same artist as the query will be filtered out of each result list (artist-filtering) to avoid colouring an evaluators judgement (a cover song or song by the same artist in a result list is likely to reduce the relative ranking of other similar but independent songs - use of songs by the same artist may allow over-fitting to affect the results)
* It will be possible for researchers to use this data for other types of system comparisons after MIREX 2010 results have been finalized.
* Human evaluation to be designed and led by IMIRSEL following a similar format to that used at MIREX 2006 (see: [[2006:Evalutron6000_Issues|Evalutron Issues in MIREX 2006]]).
* Human evaluators will be drawn from the participating labs (and any volunteers from IMIRSEL or on the MIREX lists)

=== Objective Statistics derived from the distance matrix ===
Statistics of each distance matrix will be calculated including:

* Average % of Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Precision at 5, 10, 20 & 50
* Average % of Genre matches in the top 5, 10, 20 & 50 results after artist filtering of results
* Average % of available Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Recall at 5, 10, 20 & 50 (just normalising scores when less than 20 matches for an artist, album or genre are available in the database)
* Always similar - Maximum # times a file was in the top 5, 10, 20 & 50 results
* % File never similar (never in a top 5, 10, 20 & 50 result list)
* % of 'test-able' song triplets where triangular inequality holds
** Note that as we are not requiring full distance matrices this year we will only be testing triangles that are found in the sparse distance matrix.
* Plot of the "number of times similar curve" - plot of song number vs. number of times it appeared in a top 20 list with songs sorted according to number times it appeared in a top 20 list (to produce the curve). Systems with a sharp rise at the end of this plot have "hubs", while a long 'zero' tail shows many never similar results.

=== Runtimes ===
In addition computation times for feature extraction/Index-building and querying
will be measured.

== Submission format ==
Submission to this task will have to conform to a specified format detailed below.

=== Implementation details ===
Scratch folders will be provided for all submissions for the storage of feature files and any model or index files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique filenames will be assigned to each audio track.

The audio files to be used in the task will be specified in a simple ASCII list file. This file will contain one path per line with no header line. Executables will have to accept the path to these list files as a command line parameter. The formats for the list files are specified below.

Multi-processor compute nodes (2, 4 or 8 cores) will be used to run this task. Hence, participants could attempt to use parrallelism. Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 2, 4 or 8 thread configurations. Single threaded submissions will, of course, be accepted but may be disadvantaged by time constraints.

Submissions will have to output either a full distance matrix or a search results file with the top 100 search results for each track in the collection. This list of results will be used to extract the artist-filtered results to present to the human evaluators and will facilitate the computation of the objective statistics.

=== I/O formats ===
In this section the input and output files used in this task are described as
are the command line calling format requirements for submissions.

==== Audio collection list file (input)====
The list file passed for feature extraction and indexing will be a simple ASCII list file. This file will contain one path per line with no header line, all paths will be absolute (full paths).

e.g.

/aDirectory/collectionFolder/b002342.wav
/aDirectory/collectionFolder/a005921.wav
...

==== Distance matrix output files ====
Participants should return one of two available output file formats, a full distance matrix or a sparse distance matrix. The sparse distance matrix format is preferred (as the dense distance matrices can be very large).

===== Sparse Distance Matrix =====
If computation or exhaustive search is a concern or not a normal output of the indexing algorithm employed, the sparse distance matric format detailed below may be used:

A simple ASCII file listing a name for the algorithm and the top 100 search results for every track in the collection.

This file should start with a header line with a name for the algorithm and should be followed by the results for one query per line, prefixed by the filename portion of the query path. This should be followed by a tab character and a tab separated, ordered list of the top 100 search results. Each result should include the result filename (e.g. a034728.wav) and the distance (e.g. 17.1 or 0.23) separated by a a comma.

<pre>
MyAlgorithm (my.email@address.com)
<example 1 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance>
<example 2 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance>
...
</pre>

which might look like:

<pre>
MyAlgorithm (my.email@address.com)
a009342.wav b229311.wav,0.16 a023821.wav,0.19 a001329,0.24 ... etc.
a009343.wav a661931.wav,0.12 a043322.wav,0.17 c002346,0.21 ... etc.
a009347.wav a671239.wav,0.13 c112393.wav,0.20 b083293,0.25 ... etc.
...
</pre>

The path to which this list file should be written must be accepted as a parameter on the command line.

===== Full Distance Matrix =====
Full distance matrix files should be generated in the the following format:

* A simple ASCII file listing a name for the algorithm on the first line,
* Numbered paths for each file appearing in the matrix, these can be in any order (i.e. the files don't have to be i the same order as they appeared in the list file) but should index into the columns/rows of of the distance matrix.
* A line beginning with 'Q/R' followed by a tab and tab separated list of the numbers 1 to N, where N is the files covered by the matrix.
* One line per file in the matrix give the distances of that files to each other file in the matrix. All distances should be zero or positive (0.0+) and should not be infinite or NaN. Values should be separated by a single tab character. Obviously the diagonal of the matrix (distance or a track to itself) should be zero.

<pre>
Distance matrix header text with system name
1\t</path/to/audio/file/1.wav>
2\t</path/to/audio/file/2.wav>
3\t</path/to/audio/file/3.wav>
...
N\t</path/to/audio/file/N.wav>
Q/R\t1\t2\t3\t...\tN
1\t0.0\t<dist 1 to 2>\t<dist 1 to 3>\t...\t<dist 1 to N>
2\t<dist 2 to 1>\t0.0\t<dist 2 to 3>\t...\t<dist 2 to N>
3\t<dist 3 to 2>\t<dist 3 to 2>\t0.0\t...\t<dist 3 to N>
...\t...\t...\t...\t...\t...
N\t<dist N to 1>\t<dist N to 2>\t<dist N to 3>\t...\t0.0
</pre>

which might look like:

<pre>
Example distance matrix 0.1
1 /path/to/audio/file/1.wav
2 /path/to/audio/file/2.wav
3 /path/to/audio/file/3.wav
4 /path/to/audio/file/4.wav
Q/R 1 2 3 4
1 0.00000 1.24100 0.2e-4 0.42559
2 1.24100 0.00000 0.62640 0.23564
3 50.2e-4 0.62640 0.00000 0.38000
4 0.42559 0.23567 0.38000 0.00000
</pre>

==== Example submission calling formats ====
extractFeatures.sh /path/to/scratch/folder /path/to/collectionListFile.txt
Query.sh /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt

or

doAudioSim.sh -numThreads 8 /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt

=== Packaging submissions ===
All submissions should be statically linked to all libraries (the presence of
dynamically linked libraries cannot be guarenteed).

All submissions should include a README file including the following the
information:

* Command line calling format for all executables and an example formatted set of commands
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Any required environments (and versions), e.g. python, java, bash, matlab.

== Time and hardware limits ==
Due to the potentially high number of particpants in this and other audio tasks,
hard limits on the runtime of submissions are specified.

A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.

== AMS evaluation software ==
The legacy software for performing various AMS related function is available [https://www.music-ir.org/mirex/results/2010/AMS_tools.zip here], [https://www.music-ir.org/mirex/results/2010/AMS_TOOLS_README%20.txt README file]. It maybe used to benchmark systems prior to submission and to check distance matrix file formats.

This tool set supports the following functions:
* the import of collection metadata from a delimited text file (e.g. TAB or CSV)
* the selection of a stratified random list of queries from the collection (i.e. an equal number of queries are chosen for each class of a particular metadata field, such as genre).
* the generation of results from distance matrices based on a list of pre-chosen queries.
* (pseudo-)objective statistical evaluation of distance matrices by comparing query metadata to the metadata of the top N results retrieved. Supports artist, album, genre and artist-filtered genre (where results form the same artist as query are skipped). Additionally, the number tracks never returned as results for all possible queries (orphans) and the largest hub (track similar to the most other tracks) are measured. Finally, the number of cases where the triangular inequality holds.
* preparation and post processing of results for the IMIRSEL Evalutron 6k human evaluation interface.

== Submission opening date ==

Friday 4th June 2010

== Submission closing date ==
TBA

2011 talk:MIREX Home

2010-08-25T11:10:11Z

Bfields:

Just wanted to note that I cleaned up the welcome text to actually refer to ISMIR 2011's dates and locations instead of ISMIR 2010's (Utrecht -> Miami, August -> October)

--[[User:Bfields|Bfields]] 11:09, 25 August 2010 (UTC)

2011 talk:MIREX Home

2010-08-25T11:09:54Z

Bfields: Created page with 'Just wanted to note that I cleaned up the welcome text to actually refer to ISMIR 2011's dates and locations instead of ISMIR 2010's (Utrecht -> Miami, August -> October) --~~~~'

Just wanted to note that I cleaned up the welcome text to actually refer to ISMIR 2011's dates and locations instead of ISMIR 2010's (Utrecht -> Miami, August -> October)
--[[User:Bfields|Bfields]] 11:09, 25 August 2010 (UTC)

2011:MIREX Home

2010-08-25T11:08:23Z

Bfields: /* Welcome to MIREX 2011 */

==Welcome to MIREX 2011==
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2011.

The MIREX 2011 community will hold its annual meeting as part of [http://ismir2011.ismir.net/ The 11th International Conference on Music Information Retrieval], ISMIR 2011, which will be held in Miami, Florida, the week of October the 23rd, 2011. The MIREX plenary (working lunch) and poster sessions will be held at a time to be determined during the conference.

J. Stephen Downie 
Director, IMIRSEL 

===MIREX 2011 Evaluation Tasks===

* [[2011:Audio Classification (Train/Test) Tasks]], incorporating:
** Audio Artist Identification
** Audio US Pop Genre Classification
** Audio Latin Genre Classification
** Audio Music Mood Classification
** Audio Classical Composer Identification
* [[2011:Audio Cover Song Identification]]
* [[2011:Audio Tag Classification]]
* [[2011:Audio Music Similarity and Retrieval]]
* [[2011:Symbolic Melodic Similarity]]
* [[2011:Audio Onset Detection]]
* [[2011:Audio Key Detection]]
* [[2011:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2011:Query by Singing/Humming]]
* [[2011:Audio Melody Extraction]]
* [[2011:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2011:Audio Chord Estimation]]
* [[2011:Query by Tapping]]
* [[2011:Audio Beat Tracking]]
* [[2011:Structural Segmentation]]
* [[2011:Audio Tempo Estimation]]

===Note to New Participants===
Please take the time to read the following review article that explains the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

===Note to All Participants===
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2011 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)
# present a poster at the MIREX 2011 poster session at ISMIR 2011 (Wednesday, 11 August 2011)

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you.

Due to the high volume of submissions expected at MIREX 2011, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2011==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2011 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2011 tasks. Subscription information at:
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central].

If you are participating in MIREX 2011, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2011 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2011 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
Please create an account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2010 Wikis==
Content from MIREX 2005 - 2010 are available at:

'''[[2009:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''