MIREX Wiki - User contributions [en]

2010:MIREX2010 Results

2011-06-06T21:59:18Z

CameronJones: Reverted edits by Maldinii (Talk) to last revision by Kriswest

==OVERALL RESULTS POSTERS (First Version: Will need updating as last runs are completed)==
[https://www.music-ir.org/mirex/results/2010/mirex_2010_poster.pdf MIREX 2010 Overall Results Posters (PDF)]

==Results by Task ==
This year we ran many MIREX 2010 Tasks using the new [http://nema.lis.uiuc.edu/drupal/?q=nema/architecture NEMA MIREX DIY] infrastructure. Task results with "(DIY)" appended are those generated using the NEMA MIREX DIY system. Where appropriate, do explore the various new outputs that help visualize both individual and task-wide comparative performances. A demonstration [https://www.music-ir.org/diy-demo/ video] of the NEMA MIREX DIY system can be found is also available.

===Train-Test Task Set===
* [https://nema.lis.illinois.edu/nema_out/4ffcb482-b83c-4ba6-bc42-9b538b31143c/results/evaluation/ Audio Classical Composer Identification Results ]   (DIY)
* [https://nema.lis.illinois.edu/nema_out/6731c97a-240c-4d3d-8be9-90d715ea04e1/results/evaluation/ Audio Latin Genre Classification Results ]   (DIY)
* [https://nema.lis.illinois.edu/nema_out/9b11a5c8-9fcf-4029-95eb-51ed561cfb5f/results/evaluation/ Audio Music Mood Classification Results ]   (DIY)
* [https://nema.lis.illinois.edu/nema_out/2b5839b3-3012-4f76-8807-31823588ae25/results/evaluation/ Audio Mixed Popular Genre Classification Results ]   (DIY)

===Other Tasks===

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/mck/ MCK Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/maz/ MAZ Dataset]  (DIY)
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ace/ Audio Chord Detection]  (DIY)
* [[2010:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]]
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/akd/ Audio Key Detection Results]  (DIY)
* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/adc04/ ADC04 Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex05/ MIREX05 Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/indian08/ INDIAN08 Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_0dB/ MIREX09 0dB Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset]  (DIY)
* [[2010:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/aod/ Audio Onset Detection Results]  (DIY)
* Audio Tag Classification Results
** Major Miner Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)]  (DIY)
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/aff/ Affinity estimation evaluation]  (DIY)
** Mood Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)]  (DIY)
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/aff/ Affinity estimation evaluation]  (DIY)
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ate/ Audio Tempo Estimation Results]  (DIY)
* [[2010:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]
* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex09/ MIREX09 dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex10/ MIREX10 dataset]  (DIY)
* [[2010:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]]
* [[2010:Query-by-Tapping_Results | Query-by-Tapping Results]]
*[[2010:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]
* [[2010:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]

== Machine Specifications ==

== Runtime for Submissions Run by NEMA DIY ==

* [[2010:Runtime | Runtime]]

[[Category:Results]]

2011:MIREX Home

2011-06-06T21:59:11Z

CameronJones: Reverted edits by Maldinii (Talk) to last revision by Jdownie

==Welcome to MIREX 2011==
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2011.

The MIREX 2011 community will hold its annual meeting as part of [http://ismir2011.ismir.net/ The 12th International Conference on Music Information Retrieval], ISMIR 2011, which will be held in Miami, Florida, the week of October the 23rd, 2011. The MIREX plenary (working lunch) and poster sessions will be held at a time to be determined during the conference.

J. Stephen Downie 
Director, IMIRSEL 

===MIREX 2011 Evaluation Tasks===

* [[2011:Audio Classification (Train/Test) Tasks]], incorporating:
** Audio Artist Identification
** Audio US Pop Genre Classification
** Audio Latin Genre Classification
** Audio Music Mood Classification
** Audio Classical Composer Identification
* [[2011:Audio Cover Song Identification]]
* [[2011:Audio Tag Classification]]
* [[2011:Audio Music Similarity and Retrieval]]
* [[2011:Symbolic Melodic Similarity]]
* [[2011:Audio Onset Detection]]
* [[2011:Audio Key Detection]]
* [[2011:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2011:Query by Singing/Humming]]
* [[2011:Audio Melody Extraction]]
* [[2011:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2011:Audio Chord Estimation]]
* [[2011:Query by Tapping]]
* [[2011:Audio Beat Tracking]]
* [[2011:Structural Segmentation]]
* [[2011:Audio Tempo Estimation]]

===Note to New Participants===
Please take the time to read the following review article that explains the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

===Note to All Participants===
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2011 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)
# present a poster at the MIREX 2011 poster session at ISMIR 2011 (Wednesday, 11 August 2011)

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you.

Due to the high volume of submissions expected at MIREX 2011, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2011==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2011 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2011 tasks. Subscription information at:
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central].

If you are participating in MIREX 2011, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2011 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2011 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
Please create an account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2010 Wikis==
Content from MIREX 2005 - 2010 are available at:

'''[[2009:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

MediaWiki:Sidebar

2010-12-02T18:47:15Z

CameronJones:

MIREX HOME

2010-12-02T17:14:46Z

CameronJones: Undo revision 7799 by CoryDAllen (Talk)

__TOC__

== Current MIREX Wiki (2010) ==
You can view the current 2010 content here: [[2010:MIREX HOME]]

== Introduction ==
The Music Information Retrieval Evaluation eXchange (MIREX) is an annual evaluation campaign for Music Information Retrieval (MIR) algorithms, coupled to the [http://www.ismir.net International Society (and Conference) for Music Information Retrieval (ISMIR)]. MIREX is hosted by the [https://www.music-ir.org/evaluation/ International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL)] at the [http://www.lis.illinois.edu/ Graduate School of Library Information Sciences (GSLIS)] which is part of the [http://www.illinois.edu/ University of Illinois at Urbana-Champaign (UIUC)].

Current and Future MIREXs are in part facilitated by the work of the [https://nema.lis.illinois.edu/ Networked Environment for Music Analysis (NEMA) project]. The NEMA projects aims to automate and expose the workings of MIREX and MIR experimentation/evaluation to the MIR community, helping to deal with issues of collection/data/code sharing within the community by handling issues relating to copyright and IP restrictions by allowing MIR researchers to work (remotely) with resources without having to obtain licenses to the content/code.

MIR tasks evaluated at past MIREXs include:
* [[Audio Train/Test Tasks]]
** Audio Artist Identification
** Audio Genre Classification
** Audio Music Mood Classification
** Audio Classical Composer Identification
* [[Symbolic Genre Classification]]
* [[Audio Onset Detection]]
* [[Audio Key Detection]]
* [[Symbolic Key Detection]]
* [[Audio Tag Classification]]
* [[Audio Cover Song Identification]]
* [[Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[Query by Singing/Humming]]
* [[Multiple Fundamental Frequency Estimation & Tracking]]
* [[Audio Chord Estimation]]
* [[Audio Melody Extraction]]
* [[Query by Tapping]]
* [[Audio Beat Tracking]]
* [[Audio Music Similarity and Retrieval]]
* [[Symbolic Melodic Similarity]]
* [[Structural Segmentation]]
* [[Audio Drum Detection]]
* [[Audio Tempo Extraction]]

== Recent Changes ==
We recently have merged all current and previous iterations of the MIREX wiki into a single wiki installation to make it easier to manage. All the pages, images, abstracts, and images have been migrated, but some links and images may still be broken. We're currently manually inspecting all pages, but would appreciate your help in correcting any errors you see.

Content on the wiki is now organized into mediawiki namespaces, one for each year. You can view the current 2010 content here: [[2010:MIREX HOME]]

Similarly for previous content.
* [[2009:Main_Page]]
* [[2008:Main_Page]]
* [[2007:Main_Page]]
* [[2006:Main_Page]]
* [[2005:Main_Page]]

All links to older wiki content will be redirected to this new wiki, and should take you to the correct page on the new installation, but please update any bookmarks or links you may have which point into current or old wiki content.

2010:MIREX2010 Results

2010-08-26T04:49:44Z

CameronJones: fixed poster link

==OVERALL RESULTS POSTERS (NOT READY YET)==
[https://www.music-ir.org/mirex/results/2010/mirex_2010_poster.pdf MIREX 2010 Overall Results Posters (PDF)]

==Results by Task ==

===Train-Test Task Set===
* [http://nema.lis.uiuc.edu/nema_out/664ccbda-d5b6-48ae-8c47-c27e7c2372fe/results/evaluation/ Audio Classical Composer Identification Results ]
* [http://nema.lis.uiuc.edu/nema_out/d97c8282-883e-4e71-93b7-55283829ad21/results/evaluation/ Audio Latin Genre Classification Results ]
* [http://nema.lis.uiuc.edu/nema_out/0e2212ca-2c1a-4c4e-b164-de74974afe43/results/evaluation/ Audio Music Mood Classification Results ]
* [[2010:Audio_Mixed_Popular_Genre_Classification_Results | Audio Mixed Popular Genre Classification Results]]

===Other Tasks===

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/mck/ MCK Dataset]
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/maz/ MAZ Dataset]
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ace/ Audio Chord Detection]
* [[2010:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]]
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/akd/ Audio Key Detection Results]
* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/adc04/ ADC04 Dataset]
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex05/ MIREX05 Dataset]
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/indian08/ INDIAN08 Dataset]
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_0dB/ MIREX09 0dB Dataset]
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset]
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset]
* [[2010:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/aod/ Audio Onset Detection Results]
* Audio Tag Classification Results
** Major Miner Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)]
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/aff/ Affinity estimation evaluation]
** Mood Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)]
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/aff/ Affinity estimation evaluation]
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ate/ Audio Tempo Estimation Results]
* [[2010:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]
* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex09/ MIREX09 dataset]
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex10/ MIREX10 dataset]
* [[2010:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]]
* [[2010:Query-by-Tapping_Results | Query-by-Tapping Results]]
*[[2010:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]
* [[2010:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]

== Machine Specifications ==

== Runtime for Submissions Run by NEMA DIY ==

* [[2010:Runtime | Runtime]]

[[Category:Results]]

2011

2010-08-19T16:57:48Z

CameronJones: redirect

#REDIRECT [[2011:MIREX Home]]

2011:Main Page

2010-08-19T16:57:28Z

CameronJones: Redirect

#REDIRECT [[2011:MIREX Home]]

MediaWiki:Sidebar

2010-08-19T16:56:51Z

CameronJones:

2011:MIREX Home

2010-08-19T16:56:12Z

CameronJones:

==Welcome to MIREX 2011==
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2011.

The MIREX 2011 community will hold its annual meeting as part of [http://ismir2011.ismir.net/ The 11th International Conference on Music Information Retrieval], ISMIR 2011, which will be held in Utrecht, Netherlands, from August 9th to 13th, 2011. The MIREX plenary (working lunch) and poster sessions will be held Wednesday, 11 August 2011.

J. Stephen Downie 
Director, IMIRSEL 

===MIREX 2011 Evaluation Tasks===

* [[2011:Audio Classification (Train/Test) Tasks]], incorporating:
** Audio Artist Identification
** Audio US Pop Genre Classification
** Audio Latin Genre Classification
** Audio Music Mood Classification
** Audio Classical Composer Identification
* [[2011:Audio Cover Song Identification]]
* [[2011:Audio Tag Classification]]
* [[2011:Audio Music Similarity and Retrieval]]
* [[2011:Symbolic Melodic Similarity]]
* [[2011:Audio Onset Detection]]
* [[2011:Audio Key Detection]]
* [[2011:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2011:Query by Singing/Humming]]
* [[2011:Audio Melody Extraction]]
* [[2011:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2011:Audio Chord Estimation]]
* [[2011:Query by Tapping]]
* [[2011:Audio Beat Tracking]]
* [[2011:Structural Segmentation]]
* [[2011:Audio Tempo Estimation]]

===Note to New Participants===
Please take the time to read the following review article that explains the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

===Note to All Participants===
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2011 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)
# present a poster at the MIREX 2011 poster session at ISMIR 2011 (Wednesday, 11 August 2011)

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you.

Due to the high volume of submissions expected at MIREX 2011, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2011==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2011 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2011 tasks. Subscription information at:
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central].

If you are participating in MIREX 2011, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2011 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2011 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
Please create an account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2010 Wikis==
Content from MIREX 2005 - 2010 are available at:

'''[[2009:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

2011:MIREX Home

2010-08-19T16:55:40Z

CameronJones: Created page with ' ==Welcome to MIREX 2011== This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Re…'

==Welcome to MIREX 2011==
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2011.

The MIREX 2011 community will hold its annual meeting as part of [http://ismir2011.ismir.net/ The 11th International Conference on Music Information Retrieval], ISMIR 2011, which will be held in Utrecht, Netherlands, from August 9th to 13th, 2011. The MIREX plenary (working lunch) and poster sessions will be held Wednesday, 11 August 2011.

J. Stephen Downie 
Director, IMIRSEL 

===MIREX 2011 Evaluation Tasks===

* [[2011:Audio Classification (Train/Test) Tasks]], incorporating:
** Audio Artist Identification
** Audio US Pop Genre Classification
** Audio Latin Genre Classification
** Audio Music Mood Classification
** Audio Classical Composer Identification
* [[2011:Audio Cover Song Identification]]
* [[2011:Audio Tag Classification]]
* [[2011:Audio Music Similarity and Retrieval]]
* [[2011:Symbolic Melodic Similarity]]
* [[2011:Audio Onset Detection]]
* [[2011:Audio Key Detection]]
* [[2011:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2011:Query by Singing/Humming]]
* [[2011:Audio Melody Extraction]]
* [[2011:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2011:Audio Chord Estimation]]
* [[2011:Query by Tapping]]
* [[2011:Audio Beat Tracking]]
* [[2011:Structural Segmentation]]
* [[2011:Audio Tempo Estimation]]

===Note to New Participants===
Please take the time to read the following review article that explains the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

===Note to All Participants===
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2011 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)
# present a poster at the MIREX 2011 poster session at ISMIR 2011 (Wednesday, 11 August 2011)

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you.

Due to the high volume of submissions expected at MIREX 2011, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2011==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2011 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2011 tasks. Subscription information at:
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central].

If you are participating in MIREX 2011, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2011 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2011 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
'''''Please note that you may need to create a NEW login for this wiki even if you have a login that you previously used for editing the MIREX 2005, 2006, 2007, 2008 or 2009 wikis.'''''

However, starting in 2011 the MIREX wikis have been merged so that logins will persist for future iterations of MIREX.

Please create an account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2010 Wikis==
Content from MIREX 2005 - 2010 are available at:

'''[[2009:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

2010:Audio Music Similarity and Retrieval Results

2010-08-03T01:39:01Z

CameronJones: /* FINE Scores */

== Introduction ==
These are the results for the 2010 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.

Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 100 songs were randomly selected from the 10 genre groups (10 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical '''BROAD''' score with 3 categories: NS,SS,VS as explained below, and one '''FINE''' score (in the range from 0 to 10). A description and analysis is provided below.

The systems read in 30 second audio clips as their raw data. The same 30 second clips were used in the grading stage.

=== General Legend ===

==== Team ID ====
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BWL1
| MTG-AMS || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/BWL1.pdf PDF] || [http://mtg.upf.edu Dmitry Bogdanov], [http://mtg.upf.edu Nicolas Wack], [http://mtg.upf.edu Cyril Laurier]
|-
! PS1
| PS09 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/PS1.pdf PDF] || [http://www.cp.jku.at/ Tim Pohle], [http://www.cp.jku.at/ Dominik Schnitzer]
|-
! PSS1
| PSS10 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/PSS1.pdf PDF] || [http://www.cp.jku.at/ Tim Pohle], [http://www.cp.jku.at Klaus Seyerlehner], [http://www.cp.jku.at/ Dominik Schnitzer]
|-
! RZ1
| RND || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RZ1.pdf PDF] || [http://www.cp.jku.at Rainer Zufall]
|-
! SSPK2
| cbmr_sim || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/SSPK2.pdf PDF] || [http://www.cp.jku.at Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Tim Pohle], [http://www.cp.jku.at Peter Knees]
|-
! TLN1
| Post-Processing 1 of Marsyas similarity results || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TLN1.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange], [http://sness.net Steven Ness]
|-
! TLN2
| Post-Processing 2 of Marsyas similarity results || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TLN2.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange], [http://sness.net Steven Ness]
|-
! TNL1
| MarsyasSimilarity || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TNL1.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://sness.net Steven Ness], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange]
|}

====Broad Categories====
'''NS''' = Not Similar 
'''SS''' = Somewhat Similar 
'''VS''' = Very Similar 

=====Understanding Summary Measures=====
'''Fine''' = Has a range from 0 (failure) to 100 (perfection). 
'''Broad''' = Has a range from 0 (failure) to 2 (perfection) as each query/candidate pair is scored with either NS=0, SS=1 or VS=2. 

==Human Evaluation==
===Overall Summary Results===

<csv p=3>2010/ams/AMS2010summary_evalutron.csv</csv>
 
'''Note:RZ1''' is the random result for comparing purpose.

===Friedman's Tests===
====Friedman's Test (FINE Scores)====
The Friedman test was run in MATLAB against the '''Fine''' summary data over the 100 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=3>2010/ams/evalutron.fine.friedman.tukeyKramerHSD.csv</csv>

[[File:2010AMS.evalutron.fine.friedman.tukeyKramerHSD.png|500px]]

====Friedman's Test (BROAD Scores)====
The Friedman test was run in MATLAB against the '''BROAD''' summary data over the 100 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=3>2010/ams/evalutron.cat.friedman.tukeyKramerHSD.csv</csv>

[[File:2010AMS.evalutron.cat.friedman.tukeyKramerHSD.png|500px]]

===Summary Results by Query===
====FINE Scores====
These are the mean FINE scores per query assigned by Evalutron graders. The FINE scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 and 100. A perfect score would be 100. Genre labels have been included for reference.

<csv p=3>2010/ams/fine_scores.csv</csv>

====BROAD Scores====
These are the mean BROAD scores per query assigned by Evalutron graders. The BROAD scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 (not similar) and 2 (very similar). A perfect score would be 2. Genre labels have been included for reference.

<csv p=3>2010/ams/cat_scores.csv</csv>

===Raw Scores===
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2010:Audio Music Similarity and Retrieval Raw Data]] page.

==Metadata and Distance Space Evaluation==
The following reports provide evaluation statistics based on analysis of the distance space and metadata matches and include:
* Neighbourhood clustering by artist, album and genre
* Artist-filtered genre clustering
* How often the triangular inequality holds
* Statistics on 'hubs' (tracks similar to many tracks) and orphans (tracks that are not similar to any other tracks at N results).

=== Reports ===

'''BWL1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/BWL1/report.txt Dmitry Bogdanov, Nicolas Wack, Cyril Laurier] 
'''PS1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/PS1/report.txt Tim Pohle, Dominik Schnitzer] 
'''PSS1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/PSS1/report.txt Tim Pohle, Klaus Seyerlehner, Dominik Schnitzer] 
'''RZ1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/RZ1/report.txt Dmitry Rainer Zufall] 
'''SSPK2''' = [https://music-ir.org/mirex/results/2010/ams/statistics/SSPK2/report.txt Klaus Seyerlehner, Markus Schedl, Tim Pohle, Peter Knees] 
'''TLN1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN1/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
'''TLN2''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN2/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
'''TLN3''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN3/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
== Run Times ==
<csv>2010/ams/audiosim.runtime.csv</csv>

Sandbox

2010-07-24T05:46:11Z

CameronJones:

<math>\alpha + 5 * \beta</math>

é

2010:Evalutron6000 Walkthrough

2010-07-15T19:31:15Z

CameronJones: /* Evaluate */

==UPDATE 2010==
The 2010 Evalutron has a new look and implementation. It is easier to use than the original implementation. However, it is necessary to read through this document before you start using the system. Enjoy!

== Four Important Issues Before Proceeding ==

# This system is NOT open to the general public. Due to the legalities imposed upon us by the University of Illinois and the US Federal Government, we are allowed to accept only those graders that have a stake in Music Information Retrieval and Music Digital Library research. Acceptable persons include MIREX participants, subscribers to the music-ir[AT]ircam.fr list, ISMIR attendees, computational musicologists, music technology researchers, etc. If you are in doubt about your acceptability, please contact Prof. Downie at [mailto:jdownie@illinois.edu].
# Do NOT click "Get Assignments" link if you are merely curious to see what the Evalutron 6000 is set up to do. The assignment distribution process scientifically distributes Query and Candidate sets to each account asking for assignments.
# The Evalutron is set up for both Audio Music Similarity (AMS) task and Symbolic Melodic Similarity (SMS) task. Please do NOT click "Get Assignments" link under the task you do not intend to participate. Otherwise, you will get assignments for that task which will prevent other graders from getting the assignments you get.
# Please do NOT create multiple accounts for yourself Evalutron. The system is designed to track individuals based upon unique sign-in IDs. The creation of multiple accounts for one person causes the system to improperly distribute the assignments to your fellow graders. If you have difficulties, such as forgetting your sign-in password, please contact us at [mailto:mirproject@lists.lis.illinois.edu] or try the "Forgot Password" link found at the left side of the homepage (Fig 1).

==Welcome to the Evalutron 6000==

In order to use the Evalutron 6000 you will need to modern web browser (e.g., Firefox, Internet Explorer, Safari, Mozilla, etc) that supports JavaScript, Flash, and Cookies. Evalutron has been tested on Windows XP, Windows 7, MacOS X, and Ubuntu Linux. If you are using a different platform and having trouble, please try accessing Evalutron 6000 from another machine. If you are still having difficulty, contact us at [mailto:mirex@imirsel.org mirex@imirsel.org].

When first visiting the Evalutron 6000 homepage, you will see its home page (Fig. 1).

[[Image:2010_e6k_home.png|border]]

'''Figure 1. Evalutron 6000 home page.'''

==Register to the Evalutron 6000==

If you have an account with the MIREX submission system, then you can use the same account for Evalutron 6000. Otherwise, you must register a new account. Click on the "Register" link on the left side of the page to create an account.

The registration page is fairly straightforward (Fig. 2). All fields are required. You can create any username and password you wish. Username must be at least 5 characters long and passwords must be at least 8 characters long and are case-sensitive. Before completing the registration, you will receive an email with an activation link. Clicking that link will complete your registration.

[[Image:2010_e6k_register.png|border]]

'''Figure 2. Evalutron 6000 registration page.'''

==Agree to the Informed Consent ==

Before starting evaluation, every evaluator must read and agree to the terms of the Informed Consent document. Otherwise, you will be redirect to the informed consent page when you try to get evaluation assignments. Clicking the "Informed Consent" link on the left-side menu will show the form (Fig. 3). The evaluation, because it is using human judgments of similarity, is considered a human-subjects research project and the Evalutron is a survey instrument. To indicate your consent to participate in the evaluation, scroll down the page and check the checkbox below the informed consent document.

If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.

[[Image:2010_e6k_consent.png|border]]

'''Figure 3. Evalutron 6000 informed consent page.'''

==Get Your Assignments==

To start the evaluation process, click the "My Assignments" link on the left-side menu. The assignment page is similar to Fig. 4. This page shows all tasks available in the Evalutron 6000, and initially there is no assignment given. Be careful to select the task you intend to participate, and click the "Get Assignment" button under this task. The system will assign you a number of queries to evaluate.

Please note that once assignments are made, they cannot be changed or removed. Therefore, please NEVER click "Get Assignment" for the task you do not intend to participate.

[[Image:2010_e6k_getAssignment.png|border]]

'''Figure 4. Evalutron 6000 get assignment page.'''

==Evaluate==
Clicking the "Evaluate Query" button under a query will lead you to the evaluation page of that query (Fig. 5). This page consists of instructions on the top and a list of query-candidate pairs. Please read the instructions carefully. The query for all candidates on a single page is the same, it is included with each candidate so that you can easily compare it to each candidate. For the Audio Music Similarity (AMS) task, each query and candidate is 30 second long. For Symbolic Melody Similarity (SMS) task, the length varies. Clicking on the player button besides each query or candidate will load the song into the player and begin playing. Clicking the player button again will pause it. We recommend you listen to the entire query at least once before evaluating any candidate files.

[[Image:2010_e6k_eval_page.png|border]]

'''Figure 5. Sample evaluation page.'''

Please note there are more candidates than may be immediately visible on the page. Please scroll to the bottom of the candidate list to make sure you've evaluated each song.

Once you have a feeling for whether or not the candidate is similar to the query, click the "Not Similar", "Somewhat Similar" or "Very Similar" radio buttons to the right of the query-candidate pair. Each grader will also need to assign a fine-grained score for the similarity of the candidate to the query on a scale of 0-100, with 0 indicating completely different and 100 perfectly similar or identical). To input the fine score, you need to move the slider, and once you let go of the slider the system will automatically record the score.

[[Image:2010_e6k_qcq_detail.png|border]]

'''Figure 6. Close up image of a query-candidate pair and evaluation buttons.'''

Only after you input BOTH the broad category and the fine score can a query-candidate pair be marked as green, indicating you have completed evaluating this query-candidate pair.

[[Image:2010_e6k_qcq_done_ detail.png|border]]

'''Figure 7. Close up image of a completed query-candidate pair.'''

You can always change your evaluation for any candidate by toggling the radio buttons and adjust the Fine Score selection scale. Once an evaluation has been made, however, it cannot be retracted, only changed. (i.e., you cannot "unvote").

==Work on Another Query==

At the bottom of the evaluation page, there is a "View All Assignments" button (Fig. 8). At any time, clicking this button will direct you to your assignment page similar to the one shown in Fig. 4. When you have completed evaluating all of the candidates for this query, you may click this button to continue on another query.

[[Image:2010_e6k_eval_page_bottom.png|border]]

'''Figure 8. Close up image of "View All Assignments" button which loads the assignment page. '''

You may also click the "My Assignments" link on the left-side menu to go to the My Assignments page.

On the My Assignments page, you may click another "Evaluate Query" button to load evaluation page (like Fig. 5) of that query.

You will see a list of all of the queries you have evaluated (or are evaluating) at the My Assignments page. You can return to any query by clicking on the "Evaluate Query" underneath it (Fig. 4). You can re-evaluate any candidate for any query at any time, up to the closing of the evaluation system.

==Monitor Progress==

At any time a grader may monitor his/her progress on the My Assignment page (Fig. 9). Each query has a status bar where the completed portion will be marked as green and the unfinished part red. When all the bars become green, the assignments are all completed.

[[Image:2010_e6k_progress.png|border]]

'''Figure 9. Progress indicated on the My Assignment Page '''

==Grading Expectations and "Reasonableness"==

For each query-candidate pair, we need you to assign BOTH a Broad Category score AND a Fine Score (i.e., a numeric grade between 0 and 100, 0 is meant to represent complete different and 100 perfectly similar or identical.). You have the freedom to make whatever associations you desire between a particular Broad Category score and its related Fine Score. In fact, we expect to see variations across evaluators with regard to the relationships between Broad Categories and Fine Scores as this is a normal part of human subjectivity. However, we will be using the two different types of scores to do important inter-related post-Evalutron calculations so, please, do be thoughtful in selecting your Broad Categories and related Fine Scores. What we are really asking here is that you apply a level of "reasonableness" to both your scores and your associations. For example, if you score a candidate in the VERY SIMILAR category, a Fine Score of 21 would not be, by most standards, "reasonable". Same applies at the other extreme. For example, a Broad Category score of NOT SIMILAR should not be associated with a Fine Score of, say, 72 or 84, etc.

==Coming and Going: Avoiding Grader Fatigue==

Listening to, and then comparing, hundreds of audio files is very tiring. We have built into the back-end of the Evalutron 6000 system a rather robust database system that records in near-real-time your grading scores. These scores are saved along with information about which queries and candidates you have yet to review. All this information is stored in association with your personal sign-in ID. This means you can break up your grading over several days at times convenient and productive to you. In fact, we recommend that you not try to tackle your "assignment" in one big chunk as fresh ears are happy ears and happy ears make for better evaluations.

2010:Evalutron6000 Walkthrough

2010-07-15T19:26:56Z

CameronJones: /* Agree to the Informed Consent */

==UPDATE 2010==
The 2010 Evalutron has a new look and implementation. It is easier to use than the original implementation. However, it is necessary to read through this document before you start using the system. Enjoy!

== Four Important Issues Before Proceeding ==

# This system is NOT open to the general public. Due to the legalities imposed upon us by the University of Illinois and the US Federal Government, we are allowed to accept only those graders that have a stake in Music Information Retrieval and Music Digital Library research. Acceptable persons include MIREX participants, subscribers to the music-ir[AT]ircam.fr list, ISMIR attendees, computational musicologists, music technology researchers, etc. If you are in doubt about your acceptability, please contact Prof. Downie at [mailto:jdownie@illinois.edu].
# Do NOT click "Get Assignments" link if you are merely curious to see what the Evalutron 6000 is set up to do. The assignment distribution process scientifically distributes Query and Candidate sets to each account asking for assignments.
# The Evalutron is set up for both Audio Music Similarity (AMS) task and Symbolic Melodic Similarity (SMS) task. Please do NOT click "Get Assignments" link under the task you do not intend to participate. Otherwise, you will get assignments for that task which will prevent other graders from getting the assignments you get.
# Please do NOT create multiple accounts for yourself Evalutron. The system is designed to track individuals based upon unique sign-in IDs. The creation of multiple accounts for one person causes the system to improperly distribute the assignments to your fellow graders. If you have difficulties, such as forgetting your sign-in password, please contact us at [mailto:mirproject@lists.lis.illinois.edu] or try the "Forgot Password" link found at the left side of the homepage (Fig 1).

==Welcome to the Evalutron 6000==

In order to use the Evalutron 6000 you will need to modern web browser (e.g., Firefox, Internet Explorer, Safari, Mozilla, etc) that supports JavaScript, Flash, and Cookies. Evalutron has been tested on Windows XP, Windows 7, MacOS X, and Ubuntu Linux. If you are using a different platform and having trouble, please try accessing Evalutron 6000 from another machine. If you are still having difficulty, contact us at [mailto:mirex@imirsel.org mirex@imirsel.org].

When first visiting the Evalutron 6000 homepage, you will see its home page (Fig. 1).

[[Image:2010_e6k_home.png|border]]

'''Figure 1. Evalutron 6000 home page.'''

==Register to the Evalutron 6000==

If you have an account with the MIREX submission system, then you can use the same account for Evalutron 6000. Otherwise, you must register a new account. Click on the "Register" link on the left side of the page to create an account.

The registration page is fairly straightforward (Fig. 2). All fields are required. You can create any username and password you wish. Username must be at least 5 characters long and passwords must be at least 8 characters long and are case-sensitive. Before completing the registration, you will receive an email with an activation link. Clicking that link will complete your registration.

[[Image:2010_e6k_register.png|border]]

'''Figure 2. Evalutron 6000 registration page.'''

==Agree to the Informed Consent ==

Before starting evaluation, every evaluator must read and agree to the terms of the Informed Consent document. Otherwise, you will be redirect to the informed consent page when you try to get evaluation assignments. Clicking the "Informed Consent" link on the left-side menu will show the form (Fig. 3). The evaluation, because it is using human judgments of similarity, is considered a human-subjects research project and the Evalutron is a survey instrument. To indicate your consent to participate in the evaluation, scroll down the page and check the checkbox below the informed consent document.

If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.

[[Image:2010_e6k_consent.png|border]]

'''Figure 3. Evalutron 6000 informed consent page.'''

==Get Your Assignments==

To start the evaluation process, click the "My Assignments" link on the left-side menu. The assignment page is similar to Fig. 4. This page shows all tasks available in the Evalutron 6000, and initially there is no assignment given. Be careful to select the task you intend to participate, and click the "Get Assignment" button under this task. The system will assign you a number of queries to evaluate.

Please note that once assignments are made, they cannot be changed or removed. Therefore, please NEVER click "Get Assignment" for the task you do not intend to participate.

[[Image:2010_e6k_getAssignment.png|border]]

'''Figure 4. Evalutron 6000 get assignment page.'''

==Evaluate==
Clicking the "Evaluate Query" button under a query will lead you to the evaluation page of that query (Fig. 5). This page consists of instructions on the top and a list of query-candidate pairs. Please read the instructions carefully. The query in each of the query-candidate pairs is the same, and it is aligned with each candidate so that you can replay it at any time when you evaluate the candidate. For Audio Music Similarity (AMS) task, each query and candidate is 30 second long. For Symbolic Melody Similarity (SMS) task, the length varies. Clicking on the player button besides each query or candidate will load the song into the player and begin playing. Clicking the player button again will pause it. We recommend you listen to the entire query at least once before evaluating any candidate files.

[[Image:2010_e6k_eval_page.png|border]]

'''Figure 5. Sample evaluation page.'''

Please note there are more candidates than may be immediately visible on the page. Please scroll to the bottom of the candidate list to make sure you've evaluated each song.

Once you have a feeling for whether or not the candidate is similar to the query, click the "Not Similar", "Somewhat Similar" or "Very Similar" radio buttons to the right of the query-candidate pair. Each grader will also need to assign a fine-grained score for the similarity of the candidate to the query on a scale of 0-100, with 0 indicating completely different and 100 perfectly similar or identical). To input the fine score, you need to move the scaler, and once you let go of the scaler the system will automatically record the score.

[[Image:2010_e6k_qcq_detail.png|border]]

'''Figure 6. Close up image of a query-candidate pair and evaluation buttons.'''

Only after you input BOTH the broad category and the fine score can a query-candidate pair be marked as green, indicating you have complete evaluating this query-candidate pair.

[[Image:2010_e6k_qcq_done_ detail.png|border]]

'''Figure 7. Close up image of a completed query-candidate pair.'''

You can always change your evaluation for any candidate by toggling the radio buttons and adjust the Fine Score selection scale. Once an evaluation has been made, however, it cannot be retracted, only changed. (i.e., you cannot "unvote").

==Work on Another Query==

At the bottom of the evaluation page, there is a "View All Assignments" button (Fig. 8). At any time, clicking this button will direct you to your assignment page similar to the one shown in Fig. 4. When you have completed evaluating all of the candidates for this query, you may click this button to continue on another query.

[[Image:2010_e6k_eval_page_bottom.png|border]]

'''Figure 8. Close up image of "View All Assignments" button which loads the assignment page. '''

You may also click the "My Assignments" link on the left-side menu to go to the My Assignments page.

On the My Assignments page, you may click another "Evaluate Query" button to load evaluation page (like Fig. 5) of that query.

You will see a list of all of the queries you have evaluated (or are evaluating) at the My Assignments page. You can return to any query by clicking on the "Evaluate Query" underneath it (Fig. 4). You can re-evaluate any candidate for any query at any time, up to the closing of the evaluation system.

==Monitor Progress==

At any time a grader may monitor his/her progress on the My Assignment page (Fig. 9). Each query has a status bar where the completed portion will be marked as green and the unfinished part red. When all the bars become green, the assignments are all completed.

[[Image:2010_e6k_progress.png|border]]

'''Figure 9. Progress indicated on the My Assignment Page '''

==Grading Expectations and "Reasonableness"==

For each query-candidate pair, we need you to assign BOTH a Broad Category score AND a Fine Score (i.e., a numeric grade between 0 and 100, 0 is meant to represent complete different and 100 perfectly similar or identical.). You have the freedom to make whatever associations you desire between a particular Broad Category score and its related Fine Score. In fact, we expect to see variations across evaluators with regard to the relationships between Broad Categories and Fine Scores as this is a normal part of human subjectivity. However, we will be using the two different types of scores to do important inter-related post-Evalutron calculations so, please, do be thoughtful in selecting your Broad Categories and related Fine Scores. What we are really asking here is that you apply a level of "reasonableness" to both your scores and your associations. For example, if you score a candidate in the VERY SIMILAR category, a Fine Score of 21 would not be, by most standards, "reasonable". Same applies at the other extreme. For example, a Broad Category score of NOT SIMILAR should not be associated with a Fine Score of, say, 72 or 84, etc.

==Coming and Going: Avoiding Grader Fatigue==

Listening to, and then comparing, hundreds of audio files is very tiring. We have built into the back-end of the Evalutron 6000 system a rather robust database system that records in near-real-time your grading scores. These scores are saved along with information about which queries and candidates you have yet to review. All this information is stored in association with your personal sign-in ID. This means you can break up your grading over several days at times convenient and productive to you. In fact, we recommend that you not try to tackle your "assignment" in one big chunk as fresh ears are happy ears and happy ears make for better evaluations.

2010:Evalutron6000 Walkthrough

2010-07-15T19:25:13Z

CameronJones: /* Register to the Evalutron 6000 */

==UPDATE 2010==
The 2010 Evalutron has a new look and implementation. It is easier to use than the original implementation. However, it is necessary to read through this document before you start using the system. Enjoy!

== Four Important Issues Before Proceeding ==

# This system is NOT open to the general public. Due to the legalities imposed upon us by the University of Illinois and the US Federal Government, we are allowed to accept only those graders that have a stake in Music Information Retrieval and Music Digital Library research. Acceptable persons include MIREX participants, subscribers to the music-ir[AT]ircam.fr list, ISMIR attendees, computational musicologists, music technology researchers, etc. If you are in doubt about your acceptability, please contact Prof. Downie at [mailto:jdownie@illinois.edu].
# Do NOT click "Get Assignments" link if you are merely curious to see what the Evalutron 6000 is set up to do. The assignment distribution process scientifically distributes Query and Candidate sets to each account asking for assignments.
# The Evalutron is set up for both Audio Music Similarity (AMS) task and Symbolic Melodic Similarity (SMS) task. Please do NOT click "Get Assignments" link under the task you do not intend to participate. Otherwise, you will get assignments for that task which will prevent other graders from getting the assignments you get.
# Please do NOT create multiple accounts for yourself Evalutron. The system is designed to track individuals based upon unique sign-in IDs. The creation of multiple accounts for one person causes the system to improperly distribute the assignments to your fellow graders. If you have difficulties, such as forgetting your sign-in password, please contact us at [mailto:mirproject@lists.lis.illinois.edu] or try the "Forgot Password" link found at the left side of the homepage (Fig 1).

==Welcome to the Evalutron 6000==

In order to use the Evalutron 6000 you will need to modern web browser (e.g., Firefox, Internet Explorer, Safari, Mozilla, etc) that supports JavaScript, Flash, and Cookies. Evalutron has been tested on Windows XP, Windows 7, MacOS X, and Ubuntu Linux. If you are using a different platform and having trouble, please try accessing Evalutron 6000 from another machine. If you are still having difficulty, contact us at [mailto:mirex@imirsel.org mirex@imirsel.org].

When first visiting the Evalutron 6000 homepage, you will see its home page (Fig. 1).

[[Image:2010_e6k_home.png|border]]

'''Figure 1. Evalutron 6000 home page.'''

==Register to the Evalutron 6000==

If you have an account with the MIREX submission system, then you can use the same account for Evalutron 6000. Otherwise, you must register a new account. Click on the "Register" link on the left side of the page to create an account.

The registration page is fairly straightforward (Fig. 2). All fields are required. You can create any username and password you wish. Username must be at least 5 characters long and passwords must be at least 8 characters long and are case-sensitive. Before completing the registration, you will receive an email with an activation link. Clicking that link will complete your registration.

[[Image:2010_e6k_register.png|border]]

'''Figure 2. Evalutron 6000 registration page.'''

==Agree to the Informed Consent ==

Before starting evaluation, every evaluator must read and agree to the terms of the Informed Consent document. Otherwise, you will be redirect to the informed consent page when you try to get evaluation assignments. Clicking the "Informed Consent" link on the left-side menu will show the form (Fig. 3). The evaluation, because it is using human judgments of similarity, is considered a human-subjects research project and the Evaluatron is basically a survey instrument. To indicate your consent to participate in the evaluation, scroll down the page and check the "I Agree" checkbox below the informed consent document.

If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.

[[Image:2010_e6k_consent.png|border]]

'''Figure 3. Evalutron 6000 informed consent page.'''

==Get Your Assignments==

To start the evaluation process, click the "My Assignments" link on the left-side menu. The assignment page is similar to Fig. 4. This page shows all tasks available in the Evalutron 6000, and initially there is no assignment given. Be careful to select the task you intend to participate, and click the "Get Assignment" button under this task. The system will assign you a number of queries to evaluate.

Please note that once assignments are made, they cannot be changed or removed. Therefore, please NEVER click "Get Assignment" for the task you do not intend to participate.

[[Image:2010_e6k_getAssignment.png|border]]

'''Figure 4. Evalutron 6000 get assignment page.'''

==Evaluate==
Clicking the "Evaluate Query" button under a query will lead you to the evaluation page of that query (Fig. 5). This page consists of instructions on the top and a list of query-candidate pairs. Please read the instructions carefully. The query in each of the query-candidate pairs is the same, and it is aligned with each candidate so that you can replay it at any time when you evaluate the candidate. For Audio Music Similarity (AMS) task, each query and candidate is 30 second long. For Symbolic Melody Similarity (SMS) task, the length varies. Clicking on the player button besides each query or candidate will load the song into the player and begin playing. Clicking the player button again will pause it. We recommend you listen to the entire query at least once before evaluating any candidate files.

[[Image:2010_e6k_eval_page.png|border]]

'''Figure 5. Sample evaluation page.'''

Please note there are more candidates than may be immediately visible on the page. Please scroll to the bottom of the candidate list to make sure you've evaluated each song.

Once you have a feeling for whether or not the candidate is similar to the query, click the "Not Similar", "Somewhat Similar" or "Very Similar" radio buttons to the right of the query-candidate pair. Each grader will also need to assign a fine-grained score for the similarity of the candidate to the query on a scale of 0-100, with 0 indicating completely different and 100 perfectly similar or identical). To input the fine score, you need to move the scaler, and once you let go of the scaler the system will automatically record the score.

[[Image:2010_e6k_qcq_detail.png|border]]

'''Figure 6. Close up image of a query-candidate pair and evaluation buttons.'''

Only after you input BOTH the broad category and the fine score can a query-candidate pair be marked as green, indicating you have complete evaluating this query-candidate pair.

[[Image:2010_e6k_qcq_done_ detail.png|border]]

'''Figure 7. Close up image of a completed query-candidate pair.'''

You can always change your evaluation for any candidate by toggling the radio buttons and adjust the Fine Score selection scale. Once an evaluation has been made, however, it cannot be retracted, only changed. (i.e., you cannot "unvote").

==Work on Another Query==

At the bottom of the evaluation page, there is a "View All Assignments" button (Fig. 8). At any time, clicking this button will direct you to your assignment page similar to the one shown in Fig. 4. When you have completed evaluating all of the candidates for this query, you may click this button to continue on another query.

[[Image:2010_e6k_eval_page_bottom.png|border]]

'''Figure 8. Close up image of "View All Assignments" button which loads the assignment page. '''

You may also click the "My Assignments" link on the left-side menu to go to the My Assignments page.

On the My Assignments page, you may click another "Evaluate Query" button to load evaluation page (like Fig. 5) of that query.

You will see a list of all of the queries you have evaluated (or are evaluating) at the My Assignments page. You can return to any query by clicking on the "Evaluate Query" underneath it (Fig. 4). You can re-evaluate any candidate for any query at any time, up to the closing of the evaluation system.

==Monitor Progress==

At any time a grader may monitor his/her progress on the My Assignment page (Fig. 9). Each query has a status bar where the completed portion will be marked as green and the unfinished part red. When all the bars become green, the assignments are all completed.

[[Image:2010_e6k_progress.png|border]]

'''Figure 9. Progress indicated on the My Assignment Page '''

==Grading Expectations and "Reasonableness"==

For each query-candidate pair, we need you to assign BOTH a Broad Category score AND a Fine Score (i.e., a numeric grade between 0 and 100, 0 is meant to represent complete different and 100 perfectly similar or identical.). You have the freedom to make whatever associations you desire between a particular Broad Category score and its related Fine Score. In fact, we expect to see variations across evaluators with regard to the relationships between Broad Categories and Fine Scores as this is a normal part of human subjectivity. However, we will be using the two different types of scores to do important inter-related post-Evalutron calculations so, please, do be thoughtful in selecting your Broad Categories and related Fine Scores. What we are really asking here is that you apply a level of "reasonableness" to both your scores and your associations. For example, if you score a candidate in the VERY SIMILAR category, a Fine Score of 21 would not be, by most standards, "reasonable". Same applies at the other extreme. For example, a Broad Category score of NOT SIMILAR should not be associated with a Fine Score of, say, 72 or 84, etc.

==Coming and Going: Avoiding Grader Fatigue==

Listening to, and then comparing, hundreds of audio files is very tiring. We have built into the back-end of the Evalutron 6000 system a rather robust database system that records in near-real-time your grading scores. These scores are saved along with information about which queries and candidates you have yet to review. All this information is stored in association with your personal sign-in ID. This means you can break up your grading over several days at times convenient and productive to you. In fact, we recommend that you not try to tackle your "assignment" in one big chunk as fresh ears are happy ears and happy ears make for better evaluations.

2010:Evalutron6000 Walkthrough

2010-07-15T19:24:48Z

CameronJones: /* Welcome to the Evalutron 6000 */

==UPDATE 2010==
The 2010 Evalutron has a new look and implementation. It is easier to use than the original implementation. However, it is necessary to read through this document before you start using the system. Enjoy!

== Four Important Issues Before Proceeding ==

# This system is NOT open to the general public. Due to the legalities imposed upon us by the University of Illinois and the US Federal Government, we are allowed to accept only those graders that have a stake in Music Information Retrieval and Music Digital Library research. Acceptable persons include MIREX participants, subscribers to the music-ir[AT]ircam.fr list, ISMIR attendees, computational musicologists, music technology researchers, etc. If you are in doubt about your acceptability, please contact Prof. Downie at [mailto:jdownie@illinois.edu].
# Do NOT click "Get Assignments" link if you are merely curious to see what the Evalutron 6000 is set up to do. The assignment distribution process scientifically distributes Query and Candidate sets to each account asking for assignments.
# The Evalutron is set up for both Audio Music Similarity (AMS) task and Symbolic Melodic Similarity (SMS) task. Please do NOT click "Get Assignments" link under the task you do not intend to participate. Otherwise, you will get assignments for that task which will prevent other graders from getting the assignments you get.
# Please do NOT create multiple accounts for yourself Evalutron. The system is designed to track individuals based upon unique sign-in IDs. The creation of multiple accounts for one person causes the system to improperly distribute the assignments to your fellow graders. If you have difficulties, such as forgetting your sign-in password, please contact us at [mailto:mirproject@lists.lis.illinois.edu] or try the "Forgot Password" link found at the left side of the homepage (Fig 1).

==Welcome to the Evalutron 6000==

In order to use the Evalutron 6000 you will need to modern web browser (e.g., Firefox, Internet Explorer, Safari, Mozilla, etc) that supports JavaScript, Flash, and Cookies. Evalutron has been tested on Windows XP, Windows 7, MacOS X, and Ubuntu Linux. If you are using a different platform and having trouble, please try accessing Evalutron 6000 from another machine. If you are still having difficulty, contact us at [mailto:mirex@imirsel.org mirex@imirsel.org].

When first visiting the Evalutron 6000 homepage, you will see its home page (Fig. 1).

[[Image:2010_e6k_home.png|border]]

'''Figure 1. Evalutron 6000 home page.'''

==Register to the Evalutron 6000==

If you have an account with the submission system, then you can use the same account for Evalutron 6000. Otherwise, you must register a new account. Click on the "Register" link on the left side of the page to create an account.

The registration page is fairly straightforward (Fig. 2). All fields are required. You can create any username and password you wish. Username must be at least 5 characters long and passwords must be at least 8 characters long and are case-sensitive. Before completing the registration, you will receive an email with an activation link. Clicking that link will complete your registration.

[[Image:2010_e6k_register.png|border]]

'''Figure 2. Evalutron 6000 registration page.'''

==Agree to the Informed Consent ==

Before starting evaluation, every evaluator must read and agree to the terms of the Informed Consent document. Otherwise, you will be redirect to the informed consent page when you try to get evaluation assignments. Clicking the "Informed Consent" link on the left-side menu will show the form (Fig. 3). The evaluation, because it is using human judgments of similarity, is considered a human-subjects research project and the Evaluatron is basically a survey instrument. To indicate your consent to participate in the evaluation, scroll down the page and check the "I Agree" checkbox below the informed consent document.

If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.

[[Image:2010_e6k_consent.png|border]]

'''Figure 3. Evalutron 6000 informed consent page.'''

==Get Your Assignments==

To start the evaluation process, click the "My Assignments" link on the left-side menu. The assignment page is similar to Fig. 4. This page shows all tasks available in the Evalutron 6000, and initially there is no assignment given. Be careful to select the task you intend to participate, and click the "Get Assignment" button under this task. The system will assign you a number of queries to evaluate.

Please note that once assignments are made, they cannot be changed or removed. Therefore, please NEVER click "Get Assignment" for the task you do not intend to participate.

[[Image:2010_e6k_getAssignment.png|border]]

'''Figure 4. Evalutron 6000 get assignment page.'''

==Evaluate==
Clicking the "Evaluate Query" button under a query will lead you to the evaluation page of that query (Fig. 5). This page consists of instructions on the top and a list of query-candidate pairs. Please read the instructions carefully. The query in each of the query-candidate pairs is the same, and it is aligned with each candidate so that you can replay it at any time when you evaluate the candidate. For Audio Music Similarity (AMS) task, each query and candidate is 30 second long. For Symbolic Melody Similarity (SMS) task, the length varies. Clicking on the player button besides each query or candidate will load the song into the player and begin playing. Clicking the player button again will pause it. We recommend you listen to the entire query at least once before evaluating any candidate files.

[[Image:2010_e6k_eval_page.png|border]]

'''Figure 5. Sample evaluation page.'''

Please note there are more candidates than may be immediately visible on the page. Please scroll to the bottom of the candidate list to make sure you've evaluated each song.

Once you have a feeling for whether or not the candidate is similar to the query, click the "Not Similar", "Somewhat Similar" or "Very Similar" radio buttons to the right of the query-candidate pair. Each grader will also need to assign a fine-grained score for the similarity of the candidate to the query on a scale of 0-100, with 0 indicating completely different and 100 perfectly similar or identical). To input the fine score, you need to move the scaler, and once you let go of the scaler the system will automatically record the score.

[[Image:2010_e6k_qcq_detail.png|border]]

'''Figure 6. Close up image of a query-candidate pair and evaluation buttons.'''

Only after you input BOTH the broad category and the fine score can a query-candidate pair be marked as green, indicating you have complete evaluating this query-candidate pair.

[[Image:2010_e6k_qcq_done_ detail.png|border]]

'''Figure 7. Close up image of a completed query-candidate pair.'''

You can always change your evaluation for any candidate by toggling the radio buttons and adjust the Fine Score selection scale. Once an evaluation has been made, however, it cannot be retracted, only changed. (i.e., you cannot "unvote").

==Work on Another Query==

At the bottom of the evaluation page, there is a "View All Assignments" button (Fig. 8). At any time, clicking this button will direct you to your assignment page similar to the one shown in Fig. 4. When you have completed evaluating all of the candidates for this query, you may click this button to continue on another query.

[[Image:2010_e6k_eval_page_bottom.png|border]]

'''Figure 8. Close up image of "View All Assignments" button which loads the assignment page. '''

You may also click the "My Assignments" link on the left-side menu to go to the My Assignments page.

On the My Assignments page, you may click another "Evaluate Query" button to load evaluation page (like Fig. 5) of that query.

You will see a list of all of the queries you have evaluated (or are evaluating) at the My Assignments page. You can return to any query by clicking on the "Evaluate Query" underneath it (Fig. 4). You can re-evaluate any candidate for any query at any time, up to the closing of the evaluation system.

==Monitor Progress==

At any time a grader may monitor his/her progress on the My Assignment page (Fig. 9). Each query has a status bar where the completed portion will be marked as green and the unfinished part red. When all the bars become green, the assignments are all completed.

[[Image:2010_e6k_progress.png|border]]

'''Figure 9. Progress indicated on the My Assignment Page '''

==Grading Expectations and "Reasonableness"==

For each query-candidate pair, we need you to assign BOTH a Broad Category score AND a Fine Score (i.e., a numeric grade between 0 and 100, 0 is meant to represent complete different and 100 perfectly similar or identical.). You have the freedom to make whatever associations you desire between a particular Broad Category score and its related Fine Score. In fact, we expect to see variations across evaluators with regard to the relationships between Broad Categories and Fine Scores as this is a normal part of human subjectivity. However, we will be using the two different types of scores to do important inter-related post-Evalutron calculations so, please, do be thoughtful in selecting your Broad Categories and related Fine Scores. What we are really asking here is that you apply a level of "reasonableness" to both your scores and your associations. For example, if you score a candidate in the VERY SIMILAR category, a Fine Score of 21 would not be, by most standards, "reasonable". Same applies at the other extreme. For example, a Broad Category score of NOT SIMILAR should not be associated with a Fine Score of, say, 72 or 84, etc.

==Coming and Going: Avoiding Grader Fatigue==

Listening to, and then comparing, hundreds of audio files is very tiring. We have built into the back-end of the Evalutron 6000 system a rather robust database system that records in near-real-time your grading scores. These scores are saved along with information about which queries and candidates you have yet to review. All this information is stored in association with your personal sign-in ID. This means you can break up your grading over several days at times convenient and productive to you. In fact, we recommend that you not try to tackle your "assignment" in one big chunk as fresh ears are happy ears and happy ears make for better evaluations.

MediaWiki:Sidebar

2010-07-09T07:55:04Z

CameronJones:

MediaWiki:Sidebar

2010-07-09T07:51:01Z

CameronJones: Reverted edits by CameronJones (Talk) to last revision by Jdownie

MediaWiki:Sidebar

2010-07-09T07:50:36Z

CameronJones:

MIREX 2010 Submission Instructions

2010-06-25T06:22:20Z

CameronJones: Redirected page to 2010:MIREX 2010 Submission Instructions

#Redirect [[2010:MIREX 2010 Submission Instructions]]

Sandbox

2010-06-05T09:01:59Z

CameronJones: testing math extension after upgrade

<math>\alpha + 5 * \beta</math>

File:Korean keyboard layout.png

2010-06-05T09:01:20Z

CameronJones: Testing file upload after upgrade

Testing file upload after upgrade

MediaWiki:Sidebar

2010-06-04T15:43:41Z

CameronJones:

MediaWiki:Sidebar

2010-06-04T15:41:48Z

CameronJones:

Audio Drum Detection

2010-06-04T10:37:11Z

CameronJones: Created page with '* 2005:Audio Drum Det'

* [[2005:Audio Drum Det]]

Audio Classification

2010-06-04T09:53:47Z

CameronJones: Created page with '* 2005:Audio Artist Identification * 2005:Audio Genre Classification * 2007:Audio Genre Classification * 2007:Audio Music Mood Classification * [[2007:Audio Class…'

* [[2005:Audio Artist Identification]]
* [[2005:Audio Genre Classification]]
* [[2007:Audio Genre Classification]]
* [[2007:Audio Music Mood Classification]]
* [[2007:Audio Classical Composer Identification]]
* [[2008:Audio Genre Classification]]
* [[2008:Audio Music Mood Classification]]
* [[2009:Audio Genre Classification]]
* [[2009:Audio Music Mood Classification]]
* [[2010:Audio_Classification_(Train/Test)_Tasks]]

Audio Chord Estimation

2010-06-04T09:48:10Z

CameronJones:

* [[2008:Audio Chord Detection]]
* [[2009:Audio Chord Detection]]
* [[2010:Audio Chord Estimation]]

Audio Chord Estimation

2010-06-04T09:47:12Z

CameronJones: Created page with '* 2010:Audio Chord Estimation'

* [[2010:Audio Chord Estimation]]

Audio Beat Tracking

2010-06-04T09:46:27Z

CameronJones: Created page with '* 2006:Audio Beat Tracking * 2009:Audio Beat Tracking * 2010:Audio Beat Tracking'

* [[2006:Audio Beat Tracking]]
* [[2009:Audio Beat Tracking]]
* [[2010:Audio Beat Tracking]]

MediaWiki:Sidebar

2010-06-04T08:59:29Z

CameronJones:

MediaWiki:Sidebar

2010-06-04T08:48:43Z

CameronJones:

MediaWiki:Sidebar

2010-06-04T08:47:08Z

CameronJones:

MediaWiki:Sidebar

2010-06-04T08:29:46Z

CameronJones:

MediaWiki:Sidebar

2010-06-04T08:28:50Z

MediaWiki:Mainpage

2010-06-04T08:25:13Z

CameronJones: Created page with 'MIREX HOME'

MIREX HOME

2010:MIREX HOME

2010-06-04T04:59:56Z

CameronJones: /* MIREX 2010 Evaluation Tasks */

==Welcome to MIREX 2010==
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2010). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2010.

The MIREX 2010 community will hold its annual meeting as part of [http://ismir2010.ismir.net/ The 11th International Conference on Music Information Retrieval], ISMIR 2010, which will be held in Utrecht, Netherlands, from August 9th to 13th, 2010. The MIREX plenary (working lunch) and poster sessions will be held Wednesday, 11 August 2010.

J. Stephen Downie 
Director, IMIRSEL 

===MIREX 2010 Evaluation Tasks===

The IMIRSEL team at UIUC solicited proposals for evaluation tasks to be performed at the Music Information Retrieval Evaluation eXchange 2010 (MIREX 2010) and polled the community on their likelihood of participation in each task. A summary of the responses from the community is given below:

Results as of Monday 24th May 2010:

Total individual responses = 74

<csv p=0>2010/poll/MIREX_Task_Participation_Poll.csv</csv>

Hence, the IMIRSEL team has decided to attempt the running of the following tasks at MIREX 2010:
* [[2010:Audio Classification (Train/Test) Tasks]], incorporating:
** Audio Artist Identification
** Audio US Pop Genre Classification
** Audio Latin Genre Classification
** Audio Music Mood Classification
** Audio Classical Composer Identification
* [[2010:Audio Cover Song Identification]]
* [[2010:Audio Tag Classification]]
* [[2010:Audio Music Similarity and Retrieval]]
* [[2010:Symbolic Melodic Similarity]]
* [[2010:Audio Onset Detection]]
* [[2010:Audio Key Detection]]
* [[2010:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2010:Query by Singing/Humming]]
* [[2010:Audio Melody Extraction]]
* [[2010:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2010:Audio Chord Estimation]]
* <strike>[[2010:Query by Tapping]]</strike>
* [[2010:Audio Beat Tracking]]
* [[2010:Structural Segmentation]]

==== New 2010 Proposals ====
* <strike>[[2010:Harmonic Analysis]]</strike>

===Projected dates===
* 1st June 2010: MIREX submission system opens (target date)
* 22nd June - 1st July 2010: Rolling MIREX submission system closures (dates to be announced)
* 15th July 2010: MIREX results posting begins
* 1st August 2010: All MIREX results posted (somewhat hopeful target date)
* 2-6th August 2010: USMIR Summer School
* 9-13th August 2010: ISMIR conference

===Note to New Participants===
Please take the time to read the following review article that explains the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

===Note to All Participants===
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2010 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)
# present a poster at the MIREX 2010 poster session at ISMIR 2010 (exact date to be announced)

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you.

Due to the high volume of submissions expected at MIREX 2010, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2010==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2010 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2010 tasks. Subscription information at:
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest].

If you are participating in MIREX 2010, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2010 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2010 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
'''''Please note that you may need to create a NEW login for this wiki even if you have a login that you previously used for editing the MIREX 2005, 2006, 2007, 2008 or 2009 wikis.'''''

However, starting in 2010 the MIREX wikis have been merged so that logins will persist for future iterations of MIREX.

Please create an account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2010 Submission Instructions==

* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2010 Submission Instructions]]

==MIREX 2005 - 2009 Wikis==
This is the new wiki for MIREX 2010. The wikis for MIREX 2005 - 2009 are available at:

'''[[2009:Main_Page|MIREX 2009]]'''
https://www.music-ir.org/mirex/2009/

'''[[2008:Main_Page|MIREX 2008]]'''
https://www.music-ir.org/mirex/2008/

'''[[2007:Main_Page|MIREX 2007]]'''
https://www.music-ir.org/mirex/2007/

'''[[2006:Main_Page|MIREX 2006]]'''
https://www.music-ir.org/mirex/2006/

'''[[2005:Main_Page|MIREX 2005]]'''
https://www.music-ir.org/mirex/2005/

You can interlink between this wiki and the previous wikis using '''2005:''' prefix on links to connect to pages in MIREX 2005 and '''2006:''' for MIREX 2006 and '''2007:''' for MIREX 2007 and '''2008:''' for MIREX 2008 and '''2009:''' for MIREX 2009.

===ISMIR 2004 Audio Description Contest===
The Audio Description Contest held at ISMIR 2004 is a precursor to MIREX. Details of the ISMIR 2004 Audio Description Contest can be found at:

'''[http://ismir2004.ismir.net/ISMIR_Contest.html| ISMIR 2004 Audio Description Contest]'''
http://ismir2004.ismir.net/ISMIR_Contest.html

2010:Structural Segmentation

2010-05-29T00:44:31Z

CameronJones: /* Frame clustering */ mathified the equations

== Description ==

The aim of the MIREX structural segmentation evaluation is to identify the key structural sections in musical audio. The segment structure (or form) is one of the most important musical parameters. It is furthermore special because musical structure -- especially in popular music genres (e.g. verse, chorus, etc.) -- is accessible to everybody: it needs no particular musical knowledge. This task was first run in 2009.

== Data ==

=== Collections ===
The final MIREX data set for structural segmentation is comprised of 297 songs. The majority come from the Beatles collection. Works from other artists round out the evaluation dataset.

=== Audio Formats ===

* CD-quality (PCM, 16-bit, 44100 Hz)
* single channel (mono)

== Submission Format ==

Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.

=== Input Data ===
Participating algorithms will have to read audio in the following format:

* Sample rate: 44.1 KHz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

=== Output Data ===

The structural segmentation algorithms will return the segmentation in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.

=== Output File Format (Structural Segmentation) ===

The Structural Segmentation output file format is a tab-delimited ASCII text format. This is the same as Chris Harte's chord labelling files (.lab), and so is the same format as the ground truth as well. Onset and offset times are given in seconds, and the labels are simply letters: 'A', 'B', ... with segments referring to the same structural element having the same label.

Three column text file of the format

<onset_time(sec)>\t<offset_time(sec)>\t<label>\n
<onset_time(sec)>\t<offset_time(sec)>\t<label>\n
...

where \t denotes a tab, \n denotes the end of line. The < and > characters are not included. An example output file would look something like:

0.000 5.223 A
5.223 15.101 B
15.101 20.334 A

=== Algorithm Calling Format ===

The submitted algorithm must take as arguments a SINGLE .wav file to perform the structural segmentation on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:

foobar %input %output
foobar -i %input -o %output

Moreover, if your submission takes additional parameters, foobar could be called like:

foobar .1 %input %output
foobar -param1 .1 -i %input -o %output

If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example:

foobar('%input','%output')
foobar(.1,'%input','%output')

=== README File ===

A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.

For instance, to test the program foobar with a specific value for parameter param1, the README file would look like:

foobar -param1 .1 -i %input -o %output

For a submission using MATLAB, the README file could look like:

matlab -r "foobar(.1,'%input','%output');quit;"

== Evaluation Procedures ==
At the last ISMIR conference [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich] proposed a measure for segmentation evaluation. Because of the complexity of the structural segmentation task definition, several different evaluation measures will be employed to address different aspects. It should be noted that none of the evaluation measures cares about the true labels of the sections: they only denote the clustering. This means that it does not matter if the systems produce true labels such as "chorus" and "verse", or arbitrary labels such as "A" and "B".

=== Boundary retrieval ===
'''Hit rate''' Found segment boundaries are accepted to be correct if they are within 0.5s ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007]) or 3s ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008]) from a border in the ground truth. Based on the matched hits, ''boundary retrieval recall rate'', ''boundary retrieval precision rate'', and ''boundary retrieval F-measure'' are be calculated.

'''Median deviation''' Two median deviation measure between boundaries in the result and ground truth are calculated: ''median true-to-guess'' is the median time from boundaries in ground truth to the closest boundaries in the result, and ''median guess-to-true'' is similarly the median time from boundaries in the result to boundaries in ground truth. ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007])

=== Frame clustering ===
Both the result and the ground truth are handled in short frames (e.g., beat or fixed 100ms). All frame pairs in a structure description are handled. The pairs in which both frames are assigned to the same cluster (i.e., have the same label) form the sets <math>P_E</math> (for the system result) and <math>P_A</math> (for the ground truth). The ''pairwise precision rate'' can be calculated by <math>P = \frac{|P_E \cap P_A|}{|P_E|}</math>, ''pairwise recall rate'' by <math>R = \frac{|P_E \cap P_A|}{|P_A|}</math>, and ''pairwise F-measure'' by <math>F=\frac{2 P R}{P + R}</math>. ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008])

=== Normalised conditional entropies ===
Over- and under segmentation based evaluation measures proposed in [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich ISMIR2008].
Structure descriptions are represented as frame sequences with the associated cluster information (similar to the Frame clustering measure). Confusion matrix between the labels in ground truth and the result is calculated. The matrix C is of size |L_A| * |L_E|, i.e., number of unique labels in the ground truth times number of unique labels in the result. From the confusion matrix, the joint distribution is calculated by normalising the values with the total number of frames F:

<math>p_{i,j} = C_{i,j} / F</math>

Similarly, the two marginals are calculated:

<math>p_i^a = \sum_{j=1}^{|L_E|} C{i,j}/F</math>, and

<math>p_j^e = \sum_{i=1}^{|L_A|} C{i,j}/F</math>

Conditional distributions:

<math>p_{i,j}^{a|e} = C_{i,j} / \sum_{i=1}^{|L_A|} C{i,j}</math>, and

<math>p_{i,j}^{e|a} = C_{i,j} / \sum_{j=1}^{|L_E|} C{i,j}</math>

The conditional entropies will then be

<math>H(E|A) = - \sum_{i=1}^{|L_A|} p_i^a \sum_{j=1}^{|L_E|} p_{i,j}^{e|a} \log_2(p_{i,j}^{e|a})</math>, and

<math>H(A|E) = - \sum_{j=1}^{|L_E|} p_j^e \sum_{i=1}^{|L_A|} p_{i,j}^{a|e} \log_2(p_{i,j}^{a|e})</math>

The final evaluation measures will then be the oversegmentation score

<math>S_O = 1 - \frac{H(E|A)}{\log_2(|L_E|)}</math> , and the undersegmentation score

<math>S_U = 1 - \frac{H(A|E)}{\log_2(|L_A|)}</math>

== Relevant Development Collections ==
*Jouni Paulus's [http://www.cs.tut.fi/sgn/arg/paulus/structure.html structure analysis page] links to a corpus of 177 Beatles songs ([http://www.cs.tut.fi/sgn/arg/paulus/beatles_sections_TUT.zip zip file]). The Beatles annotations are not a part of the TUTstructure07 dataset. That dataset contains 557 songs, a list of which is available [http://www.cs.tut.fi/sgn/arg/paulus/TUTstructure07_files.html here].

*Ewald Peiszer's [http://www.ifs.tuwien.ac.at/mir/audiosegmentation.html thesis page] links to a portion of the corpus he used: 43 non-Beatles pop songs (including 10 J-pop songs) ([http://www.ifs.tuwien.ac.at/mir/audiosegmentation/dl/ep_groundtruth_excl_Paulus.zip zip file]).

These public corpora give a combined 220 songs.

2010:Structural Segmentation

2010-05-29T00:43:28Z

CameronJones: /* Normalised conditional entropies */ Hopefully TeX is working now...

== Description ==

The aim of the MIREX structural segmentation evaluation is to identify the key structural sections in musical audio. The segment structure (or form) is one of the most important musical parameters. It is furthermore special because musical structure -- especially in popular music genres (e.g. verse, chorus, etc.) -- is accessible to everybody: it needs no particular musical knowledge. This task was first run in 2009.

== Data ==

=== Collections ===
The final MIREX data set for structural segmentation is comprised of 297 songs. The majority come from the Beatles collection. Works from other artists round out the evaluation dataset.

=== Audio Formats ===

* CD-quality (PCM, 16-bit, 44100 Hz)
* single channel (mono)

== Submission Format ==

Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.

=== Input Data ===
Participating algorithms will have to read audio in the following format:

* Sample rate: 44.1 KHz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

=== Output Data ===

The structural segmentation algorithms will return the segmentation in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.

=== Output File Format (Structural Segmentation) ===

The Structural Segmentation output file format is a tab-delimited ASCII text format. This is the same as Chris Harte's chord labelling files (.lab), and so is the same format as the ground truth as well. Onset and offset times are given in seconds, and the labels are simply letters: 'A', 'B', ... with segments referring to the same structural element having the same label.

Three column text file of the format

<onset_time(sec)>\t<offset_time(sec)>\t<label>\n
<onset_time(sec)>\t<offset_time(sec)>\t<label>\n
...

where \t denotes a tab, \n denotes the end of line. The < and > characters are not included. An example output file would look something like:

0.000 5.223 A
5.223 15.101 B
15.101 20.334 A

=== Algorithm Calling Format ===

The submitted algorithm must take as arguments a SINGLE .wav file to perform the structural segmentation on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:

foobar %input %output
foobar -i %input -o %output

Moreover, if your submission takes additional parameters, foobar could be called like:

foobar .1 %input %output
foobar -param1 .1 -i %input -o %output

If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example:

foobar('%input','%output')
foobar(.1,'%input','%output')

=== README File ===

A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.

For instance, to test the program foobar with a specific value for parameter param1, the README file would look like:

foobar -param1 .1 -i %input -o %output

For a submission using MATLAB, the README file could look like:

matlab -r "foobar(.1,'%input','%output');quit;"

== Evaluation Procedures ==
At the last ISMIR conference [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich] proposed a measure for segmentation evaluation. Because of the complexity of the structural segmentation task definition, several different evaluation measures will be employed to address different aspects. It should be noted that none of the evaluation measures cares about the true labels of the sections: they only denote the clustering. This means that it does not matter if the systems produce true labels such as "chorus" and "verse", or arbitrary labels such as "A" and "B".

=== Boundary retrieval ===
'''Hit rate''' Found segment boundaries are accepted to be correct if they are within 0.5s ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007]) or 3s ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008]) from a border in the ground truth. Based on the matched hits, ''boundary retrieval recall rate'', ''boundary retrieval precision rate'', and ''boundary retrieval F-measure'' are be calculated.

'''Median deviation''' Two median deviation measure between boundaries in the result and ground truth are calculated: ''median true-to-guess'' is the median time from boundaries in ground truth to the closest boundaries in the result, and ''median guess-to-true'' is similarly the median time from boundaries in the result to boundaries in ground truth. ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007])

=== Frame clustering ===
Both the result and the ground truth are handled in short frames (e.g., beat or fixed 100ms). All frame pairs in a structure description are handled. The pairs in which both frames are assigned to the same cluster (i.e., have the same label) form the sets P_E (for the system result) and P_A (for the ground truth). The ''pairwise precision rate'' can be calculated by P = \frac{|P_E \cap P_A|}{|P_E|}, ''pairwise recall rate'' by R = \frac{|P_E \cap P_A|}{|P_A|}, and ''pairwise F-measure'' by F=\frac{2 P R}{P + R}. ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008])

=== Normalised conditional entropies ===
Over- and under segmentation based evaluation measures proposed in [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich ISMIR2008].
Structure descriptions are represented as frame sequences with the associated cluster information (similar to the Frame clustering measure). Confusion matrix between the labels in ground truth and the result is calculated. The matrix C is of size |L_A| * |L_E|, i.e., number of unique labels in the ground truth times number of unique labels in the result. From the confusion matrix, the joint distribution is calculated by normalising the values with the total number of frames F:

<math>p_{i,j} = C_{i,j} / F</math>

Similarly, the two marginals are calculated:

<math>p_i^a = \sum_{j=1}^{|L_E|} C{i,j}/F</math>, and

<math>p_j^e = \sum_{i=1}^{|L_A|} C{i,j}/F</math>

Conditional distributions:

<math>p_{i,j}^{a|e} = C_{i,j} / \sum_{i=1}^{|L_A|} C{i,j}</math>, and

<math>p_{i,j}^{e|a} = C_{i,j} / \sum_{j=1}^{|L_E|} C{i,j}</math>

The conditional entropies will then be

<math>H(E|A) = - \sum_{i=1}^{|L_A|} p_i^a \sum_{j=1}^{|L_E|} p_{i,j}^{e|a} \log_2(p_{i,j}^{e|a})</math>, and

<math>H(A|E) = - \sum_{j=1}^{|L_E|} p_j^e \sum_{i=1}^{|L_A|} p_{i,j}^{a|e} \log_2(p_{i,j}^{a|e})</math>

The final evaluation measures will then be the oversegmentation score

<math>S_O = 1 - \frac{H(E|A)}{\log_2(|L_E|)}</math> , and the undersegmentation score

<math>S_U = 1 - \frac{H(A|E)}{\log_2(|L_A|)}</math>

== Relevant Development Collections ==
*Jouni Paulus's [http://www.cs.tut.fi/sgn/arg/paulus/structure.html structure analysis page] links to a corpus of 177 Beatles songs ([http://www.cs.tut.fi/sgn/arg/paulus/beatles_sections_TUT.zip zip file]). The Beatles annotations are not a part of the TUTstructure07 dataset. That dataset contains 557 songs, a list of which is available [http://www.cs.tut.fi/sgn/arg/paulus/TUTstructure07_files.html here].

*Ewald Peiszer's [http://www.ifs.tuwien.ac.at/mir/audiosegmentation.html thesis page] links to a portion of the corpus he used: 43 non-Beatles pop songs (including 10 J-pop songs) ([http://www.ifs.tuwien.ac.at/mir/audiosegmentation/dl/ep_groundtruth_excl_Paulus.zip zip file]).

These public corpora give a combined 220 songs.

Sandbox

2010-05-29T00:38:34Z

CameronJones:

<math>\alpha</math>

Sandbox

2010-05-29T00:28:11Z

CameronJones:

Sandbox

2010-05-28T20:23:17Z

CameronJones: Created page with '<math>\alpha</math>'

<math>\alpha</math>

2010:Structural Segmentation

2010-05-28T20:22:02Z

CameronJones: /* Normalised conditional entropies */

== Description ==

The aim of the MIREX structural segmentation evaluation is to identify the key structural sections in musical audio. The segment structure (or form) is one of the most important musical parameters. It is furthermore special because musical structure -- especially in popular music genres (e.g. verse, chorus, etc.) -- is accessible to everybody: it needs no particular musical knowledge. This task was first run in 2009.

== Data ==

=== Collections ===
The final MIREX data set for structural segmentation is comprised of 297 songs. The majority come from the Beatles collection. Works from other artists round out the evaluation dataset.

=== Audio Formats ===

* CD-quality (PCM, 16-bit, 44100 Hz)
* single channel (mono)

== Submission Format ==

Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.

=== Input Data ===
Participating algorithms will have to read audio in the following format:

* Sample rate: 44.1 KHz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

=== Output Data ===

The structural segmentation algorithms will return the segmentation in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.

=== Output File Format (Structural Segmentation) ===

The Structural Segmentation output file format is a tab-delimited ASCII text format. This is the same as Chris Harte's chord labelling files (.lab), and so is the same format as the ground truth as well. Onset and offset times are given in seconds, and the labels are simply letters: 'A', 'B', ... with segments referring to the same structural element having the same label.

Three column text file of the format

<onset_time(sec)>\t<offset_time(sec)>\t<label>\n
<onset_time(sec)>\t<offset_time(sec)>\t<label>\n
...

where \t denotes a tab, \n denotes the end of line. The < and > characters are not included. An example output file would look something like:

0.000 5.223 A
5.223 15.101 B
15.101 20.334 A

=== Algorithm Calling Format ===

The submitted algorithm must take as arguments a SINGLE .wav file to perform the structural segmentation on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:

foobar %input %output
foobar -i %input -o %output

Moreover, if your submission takes additional parameters, foobar could be called like:

foobar .1 %input %output
foobar -param1 .1 -i %input -o %output

If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example:

foobar('%input','%output')
foobar(.1,'%input','%output')

=== README File ===

A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.

For instance, to test the program foobar with a specific value for parameter param1, the README file would look like:

foobar -param1 .1 -i %input -o %output

For a submission using MATLAB, the README file could look like:

matlab -r "foobar(.1,'%input','%output');quit;"

== Evaluation Procedures ==
At the last ISMIR conference [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich] proposed a measure for segmentation evaluation. Because of the complexity of the structural segmentation task definition, several different evaluation measures will be employed to address different aspects. It should be noted that none of the evaluation measures cares about the true labels of the sections: they only denote the clustering. This means that it does not matter if the systems produce true labels such as "chorus" and "verse", or arbitrary labels such as "A" and "B".

=== Boundary retrieval ===
'''Hit rate''' Found segment boundaries are accepted to be correct if they are within 0.5s ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007]) or 3s ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008]) from a border in the ground truth. Based on the matched hits, ''boundary retrieval recall rate'', ''boundary retrieval precision rate'', and ''boundary retrieval F-measure'' are be calculated.

'''Median deviation''' Two median deviation measure between boundaries in the result and ground truth are calculated: ''median true-to-guess'' is the median time from boundaries in ground truth to the closest boundaries in the result, and ''median guess-to-true'' is similarly the median time from boundaries in the result to boundaries in ground truth. ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007])

=== Frame clustering ===
Both the result and the ground truth are handled in short frames (e.g., beat or fixed 100ms). All frame pairs in a structure description are handled. The pairs in which both frames are assigned to the same cluster (i.e., have the same label) form the sets P_E (for the system result) and P_A (for the ground truth). The ''pairwise precision rate'' can be calculated by P = \frac{|P_E \cap P_A|}{|P_E|}, ''pairwise recall rate'' by R = \frac{|P_E \cap P_A|}{|P_A|}, and ''pairwise F-measure'' by F=\frac{2 P R}{P + R}. ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008])

=== Normalised conditional entropies ===
Over- and under segmentation based evaluation measures proposed in [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich ISMIR2008].
Structure descriptions are represented as frame sequences with the associated cluster information (similar to the Frame clustering measure). Confusion matrix between the labels in ground truth and the result is calculated. The matrix C is of size |L_A| * |L_E|, i.e., number of unique labels in the ground truth times number of unique labels in the result. From the confusion matrix, the joint distribution is calculated by normalising the values with the total number of frames F:

<math>p_{i,j} = C_{i,j} / F</math>

Similarly, the two marginals are calculated:

p_i^a = \sum_{j=1}^{|L_E|} C{i,j}/F , and

p_j^e = \sum_{i=1}^{|L_A|} C{i,j}/F

Conditional distributions:

p_{i,j}^{a|e} = C_{i,j} / \sum_{i=1}^{|L_A|} C{i,j} , and

p_{i,j}^{e|a} = C_{i,j} / \sum_{j=1}^{|L_E|} C{i,j}

The conditional entropies will then be

H(E|A) = - \sum_{i=1}^{|L_A|} p_i^a \sum_{j=1}^{|L_E|} p_{i,j}^{e|a} \log_2(p_{i,j}^{e|a}), and

H(A|E) = - \sum_{j=1}^{|L_E|} p_j^e \sum_{i=1}^{|L_A|} p_{i,j}^{a|e} \log_2(p_{i,j}^{a|e})

The final evaluation measures will then be the oversegmentation score

S_O = 1 - \frac{H(E|A)}{\log_2(|L_E|)} , and the undersegmentation score

S_U = 1 - \frac{H(A|E)}{\log_2(|L_A|)}

== Relevant Development Collections ==
*Jouni Paulus's [http://www.cs.tut.fi/sgn/arg/paulus/structure.html structure analysis page] links to a corpus of 177 Beatles songs ([http://www.cs.tut.fi/sgn/arg/paulus/beatles_sections_TUT.zip zip file]). The Beatles annotations are not a part of the TUTstructure07 dataset. That dataset contains 557 songs, a list of which is available [http://www.cs.tut.fi/sgn/arg/paulus/TUTstructure07_files.html here].

*Ewald Peiszer's [http://www.ifs.tuwien.ac.at/mir/audiosegmentation.html thesis page] links to a portion of the corpus he used: 43 non-Beatles pop songs (including 10 J-pop songs) ([http://www.ifs.tuwien.ac.at/mir/audiosegmentation/dl/ep_groundtruth_excl_Paulus.zip zip file]).

These public corpora give a combined 220 songs.

2005

2010-05-19T17:06:37Z

CameronJones: Redirected page to 2005:Main Page

#REDIRECT [[2005:Main_Page]]

2006

2010-05-19T17:06:09Z

CameronJones: Redirected page to 2006:Main Page

#REDIRECT [[2006:Main_Page]]

2007

2010-05-19T17:05:50Z

CameronJones: Redirected page to 2007:Main Page

#REDIRECT [[2007:Main_Page]]

2008

2010-05-19T17:05:35Z

CameronJones: Redirected page to 2008:Main Page

#REDIRECT [[2008:Main_Page]]

2009

2010-05-19T16:53:55Z

CameronJones: Redirected page to 2009:Main Page

#REDIRECT [[2009:Main_Page]]

2010

2010-05-19T16:53:35Z

CameronJones: Redirected page to 2010:Main Page

#REDIRECT [[2010:Main_Page]]

MIREX HOME

2010-05-17T22:16:28Z

CameronJones:

We recently have merged all current and previous iterations of the MIREX wiki into a single wiki installation to make it easier to manage. All the pages, images, abstracts, and images have been migrated, but some links and images may still be broken. We're currently manually inspecting all pages, but would appreciate your help in correcting any errors you see.

Content on the wiki is now organized into mediawiki namespaces, one for each year. You can view the current 2010 content here: [[2010:Main_Page]]

Similarly for previous content.
* [[2009:Main_Page]]
* [[2008:Main_Page]]
* [[2007:Main_Page]]
* [[2006:Main_Page]]
* [[2005:Main_Page]]

All links to older wiki content will be redirected to this new wiki, and should take you to the correct page on the new installation, but please update any bookmarks or links you may have which point into current or old wiki content.

2009:Query-by-Tapping Results

2010-05-14T03:39:06Z

CameronJones:

==Introduction==
These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the [[2009:Query by Singing/Humming]] page.

===Task Descriptions===

'''Task 1 [[#Task 1 Results|Goto Task 1 Results]]''': The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:

* [[#Task 1a, Jang's Dataset Results|Jang's Dataset]] Roger Jang's [http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries.
* [[#Task 1b, Hsiao's Dataset Results|Hsiao's Dataset]] Show Hsiao's [http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries.

'''Task 2 [[#Task 2 Results|Goto Task 2 Results]]''':The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.

===General Legend===
====Team ID====
'''CSJ''' = [https://www.music-ir.org/mirex/2009/results/qbt/QbtChenJang.pdf Chun-Ta Chen and Jyh-Shing Roger Jang] 
'''HAFR''' = [https://www.music-ir.org/mirex/2009/results/qbt/HAFR.pdf Pierre Hanna, Julien Allali, Pascal Ferraro,Matthias Robine] 
'''HL''' = [https://www.music-ir.org/mirex/2009/results/qbt/QBT_Show.pdf Shu-Jen Show Hsiao,Tyne Liang] 

===Task 1 Results===

===Task 1a, Jang's Dataset Results===

=====Task 1a Overall Results=====
<csv>2009/qbt/QbtFinalTask1a.csv</csv>

====Task 1a Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:
<csv>2009/qbt/friedman/QbtTask1aSimple_friedman_tukeyKramerHSD.csv</csv>

[[Image:2009_sqbttask1asimple_friedman_mean_ranks.png]]

MRR Method:
<csv>2009/qbt/friedman/Qbt1aTask2Mrr_friedman_tukeyKramerHSD.csv</csv>

[[Image:2009_sqbt1atask2mrr_friedman_mean_ranks.png]]

====Task 1a Summary Results by Query Group====
Simple Hit/Miss Counting
<csv p=2>2009/qbt/QbtTask1aSimpleByGroup.csv</csv>

MRR Method
<csv p=2>2009/qbt/QbtTask1aMrrByGroup.csv</csv>

====Task 1a Summary Results by Query ====
Simple Hit/Miss Counting
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask1aSimpleByQuery.csv]

MRR Method
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask1aMrrByQuery.csv]

===Task 1b, Hsiao's Dataset Results===

=====Task 1b Overall Results=====
<csv p=2>2009/qbt/QbtFinalTask1b.csv</csv>

====Task 1b Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:
<csv>2009/qbt/friedman/QbtTask1bSimple_friedman_tukeyKramerHSD.csv</csv>

[[Image:2009_sqbttask1bsimple_friedman_mean_ranks.png]]

MRR Method:
<csv>2009/qbt/friedman/QbtTask1bMrr_friedman_tukeyKramerHSD.csv</csv>

[[Image:2009_sqbttask1bmrr_friedman_mean_ranks.png ]]

====Task 1b Summary Results by Query Group====
Simple Hit/Miss Counting
<csv p=2>2009/qbt/QbtTask1bSimpleByGroup.csv</csv>

MRR Method
<csv p=2>2009/qbt/QbtTask1bMrrByGroup.csv</csv>

====Task 1b Summary Results by Query ====
Simple Hit/Miss Counting
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask1bSimpleByQuery.csv]

MRR Method
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask1bMrrByQuery.csv]

===Task 2 Results===

=====Task 2 Overall Results=====
<csv>2009/qbt/QbtFinalTask2.csv</csv>

====Task 2 Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:
<csv>2009/qbt/friedman/QbtTask2Simple_friedman_tukeyKramerHSD.csv</csv>

[[Image:2009_sqbttask2simple_friedman_mean_ranks.png]]

MRR Method:
<csv>2009/qbt/friedman/QbtTask2Mrr_friedman_tukeyKramerHSD.csv</csv>

[[Image:2009_sqbttask2mrr_friedman_mean_ranks.png]]

====Task 2 Summary Results by Query Group====
Simple Hit/Miss Counting
<csv p=2>2009/qbt/QbtTask2SimpleByGroup.csv</csv>

MRR Method
<csv p=2>2009/qbt/QbtTask2MrrByGroup.csv</csv>

====Task 2 Summary Results by Query ====
Simple Hit/Miss Counting
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask2SimpleByQuery.csv]

MRR Method
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask2MrrByQuery.csv]

===Runtime Results===

<csv>2009/qbt/qbtRunTime.csv</csv>

[[Category: Results]]