MIREX Wiki - User contributions [en]

2016:Task Captains

2016-02-22T18:17:27Z

Johan Pauwels: Added myself as team captain

Like ISMIR 2015, we are prepared to improve the distribution of tasks for the upcoming MIREX 2016. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead one or more tasks, please add your name in the "Captains" column.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

{| class="wikitable" style="margin-left: 20px"
!ID !! Task !! Captain(s)
|-
|abt
|[[2016:Audio Beat Tracking]]
|Sebastian Böck, Florian Krebs
|-
|ace
|[[2016:Audio Chord Estimation]]
|Johan Pauwels
|-
|act
|[[2016:Audio Classification (Train/Test) Tasks]]
|IMIRSEL
|-
|acs
|[[2016:Audio Cover Song Identification]]
|
|-
|ade
|[[2016:Audio Downbeat Estimation]]
|Florian Krebs, Sebastian Böck
|-
|akd
|[[2016:Audio Key Detection]]
|Johan Pauwels
|-
|ame
|[[2016:Audio Melody Extraction]]
|
|-
|ams
|[[2016:Audio Music Similarity and Retrieval]]
|IMIRSEL
|-
|aod
|[[2016:Audio Onset Detection]]
|Sebastian Böck
|-
|ate
|[[2016:Audio Tempo Estimation]]
|Aggelos Gkiokas
|-
|atg
|[[2016:Audio Tag Classification]]
|
|-
|mf0
|[[2016:Multiple Fundamental Frequency Estimation & Tracking]]
|
|-
|qbsh
|[[2016:Query by Singing/Humming]]
|
|-
|scofo
|[[2016:Real-time Audio to Score Alignment (a.k.a Score Following)]]
| Julio Carabias
|
|-
|sms
|[[2016:Symbolic Melodic Similarity]]
|
|-
|struct
|[[2016:Structural Segmentation]]
|IMIRSEL
|-
|drts
|[[2016:Discovery of Repeated Themes & Sections]]
|
|-
|sli
|[[2016:Set List Identification ]]
|
|-
|mscd
|[[2016:Music/Speech Classification and Detection]]
|
|-
|aod
|[[2016:Audio Offset Detection ]]
|
|-
|afp
|[[2016:Audio_Fingerprinting]]
|
|-
|svs
|[[2016:Singing Voice Separation]]
|
|}

2015:Audio Chord Estimation Results

2015-10-22T22:32:49Z

Johan Pauwels: Added note about CM3

==Introduction==

This page contains the results of the 2015 edition of the MIREX automatic chord estimation tasks. This edition was the third one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last two years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* A new data set, called "JayChou 2015" has been donated by [http://tangkk.net Junqi Deng] of the [http://www.hku.hk University of Hong Kong]. It consists of 29 Mandopop songs taken from various albums by [https://en.wikipedia.org/wiki/Jay_Chou Jay Chou]. Most of the songs are ballads and special attention has been paid to the annotation of extended chords and inversions. Because Junqi was kind enough to provide this set before official publication, the algorithmic output on these files and their ground-truth has been withheld for the time being. Also the file names in the per track results have been anonymized.
* The algorithmic output and per track results on the Isophonics set now display the unmasked song name, such that an evaluation per artist/album can be performed.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CM3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| DK4-DK9
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/DK4.pdf PDF]
| Junqi Deng, Yu-Kwong Kwok
|-
| KO1 (shineChords)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/KO1.pdf PDF]
| Maksim Khadkevich, Maurizio Omologo
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary.

=====Isophonics 2009=====
<csv>2015/ace/Isophonics2009.csv</csv>
=====Billboard 2012=====
<csv>2015/ace/Billboard2012.csv</csv>
=====Billboard 2013=====
<csv>2015/ace/Billboard2013.csv</csv>
=====JayChou 2015=====
<csv>2015/ace/JayChou2015.csv</csv>
=====CM3 update=====
Chris Cannam and Matthias Mauch have informed us that their intention was to resubmit last year's system. Unfortunately, a small change that wasn't supposed to affect the output did introduce a serious bug, which they only realised after seeing these results. They want the community to know that [[2014:Audio_Chord_Estimation_Results | last year's results]] are more representative of the capabilities of their system.

===Comparative Statistics===

An analysis of the statistical difference between all submissions can be found on the following pages:

* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Isophonics2009 | Isophonics 2009 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2012 | Billboard 2012 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2013 | Billboard 2013 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_JayChou2015 | JayChou 2015 Dataset]]

===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from these archives:

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Results.zip Isophonics2009Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Results.zip Billboard2012Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Results.zip Billboard2013Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/JayChou2015Results.zip JayChou2015Results.zip]

===Algorithmic Output===

The raw output of the algorithms are available in the archives below. This can be used to experiment with alternative evaluation measures and statistics.

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Output.zip Isophonics2009Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Output.zip Billboard2012Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Output.zip Billboard2013Output.zip]

2015:Audio Chord Estimation Results

2015-10-21T18:45:55Z

Johan Pauwels: Fix latest edit

==Introduction==

This page contains the results of the 2015 edition of the MIREX automatic chord estimation tasks. This edition was the third one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last two years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* A new data set, called "JayChou 2015" has been donated by [http://tangkk.net Junqi Deng] of the [http://www.hku.hk University of Hong Kong]. It consists of 29 Mandopop songs taken from various albums by [https://en.wikipedia.org/wiki/Jay_Chou Jay Chou]. Most of the songs are ballads and special attention has been paid to the annotation of extended chords and inversions. Because Junqi was kind enough to provide this set before official publication, the algorithmic output on these files and their ground-truth has been withheld for the time being. Also the file names in the per track results have been anonymized.
* The algorithmic output and per track results on the Isophonics set now display the unmasked song name, such that an evaluation per artist/album can be performed.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CM3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| DK4-DK9
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/DK4.pdf PDF]
| Junqi Deng, Yu-Kwong Kwok
|-
| KO1 (shineChords)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/KO1.pdf PDF]
| Maksim Khadkevich, Maurizio Omologo
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary.

=====Isophonics 2009=====
<csv>2015/ace/Isophonics2009.csv</csv>
=====Billboard 2012=====
<csv>2015/ace/Billboard2012.csv</csv>
=====Billboard 2013=====
<csv>2015/ace/Billboard2013.csv</csv>
=====JayChou 2015=====
<csv>2015/ace/JayChou2015.csv</csv>

===Comparative Statistics===

An analysis of the statistical difference between all submissions can be found on the following pages:

* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Isophonics2009 | Isophonics 2009 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2012 | Billboard 2012 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2013 | Billboard 2013 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_JayChou2015 | JayChou 2015 Dataset]]

===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from these archives:

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Results.zip Isophonics2009Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Results.zip Billboard2012Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Results.zip Billboard2013Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/JayChou2015Results.zip JayChou2015Results.zip]

===Algorithmic Output===

The raw output of the algorithms are available in the archives below. This can be used to experiment with alternative evaluation measures and statistics.

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Output.zip Isophonics2009Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Output.zip Billboard2012Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Output.zip Billboard2013Output.zip]

2015:Audio Chord Estimation Results

2015-10-21T18:44:08Z

Johan Pauwels: External link syntax cleanup

==Introduction==

This page contains the results of the 2015 edition of the MIREX automatic chord estimation tasks. This edition was the third one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last two years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 2013 edition]].

==What’s new?==
* A new data set, called "JayChou 2015" has been donated by [http://tangkk.net Junqi Deng] of the [http://www.hku.hk University of Hong Kong]. It consists of 29 Mandopop songs taken from various albums by [https://en.wikipedia.org/wiki/Jay_Chou | Jay Chou]. Most of the songs are ballads and special attention has been paid to the annotation of extended chords and inversions. Because Junqi was kind enough to provide this set before official publication, the algorithmic output on these files and their ground-truth has been withheld for the time being. Also the file names in the per track results have been anonymized.
* The algorithmic output and per track results on the Isophonics set now display the unmasked song name, such that an evaluation per artist/album can be performed.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CM3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| DK4-DK9
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/DK4.pdf PDF]
| Junqi Deng, Yu-Kwong Kwok
|-
| KO1 (shineChords)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/KO1.pdf PDF]
| Maksim Khadkevich, Maurizio Omologo
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary.

=====Isophonics 2009=====
<csv>2015/ace/Isophonics2009.csv</csv>
=====Billboard 2012=====
<csv>2015/ace/Billboard2012.csv</csv>
=====Billboard 2013=====
<csv>2015/ace/Billboard2013.csv</csv>
=====JayChou 2015=====
<csv>2015/ace/JayChou2015.csv</csv>

===Comparative Statistics===

An analysis of the statistical difference between all submissions can be found on the following pages:

* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Isophonics2009 | Isophonics 2009 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2012 | Billboard 2012 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2013 | Billboard 2013 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_JayChou2015 | JayChou 2015 Dataset]]

===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from these archives:

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Results.zip Isophonics2009Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Results.zip Billboard2012Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Results.zip Billboard2013Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/JayChou2015Results.zip JayChou2015Results.zip]

===Algorithmic Output===

The raw output of the algorithms are available in the archives below. This can be used to experiment with alternative evaluation measures and statistics.

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Output.zip Isophonics2009Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Output.zip Billboard2012Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Output.zip Billboard2013Output.zip]

2015:MIREX2015 Results

2015-10-21T18:42:47Z

Johan Pauwels: Fixed internal link

==Results by Task ==

==OVERALL RESULTS POSTERS ==

[https://www.music-ir.org/mirex/results/2015/mirex_2015_poster.pdf MIREX 2015 Overall Results Posters (PDF)]

===Train-Test Task Set===
* [https://www.music-ir.org/nema_out/mirex2015/results/act/composer_report/ Audio Classical Composer Identification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/latin_report/ Audio Latin Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/mood_report/index.html Audio Music Mood Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/kgenrek_report/ Audio KPOP Genre (Annotated by Korean Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/kgenrea_report/ Audio KPOP Genre (Annotated by American Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/kmoodk_report/ Audio KPOP Mood (Annotated by Korean Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/kmooda_report/ Audio KPOP Mood (Annotated by American Annotators) Classification Results ]  

===Other Tasks===

* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/struct/mrx09/ MIREX09 dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/struct/sal/ SALAMI dataset]  

* [[2015:Discovery of Repeated Themes & Sections Results | Discovery of Repeated Themes & Sections Results]]

* [https://nema.lis.illinois.edu/nema_out/mirex2015/results/aod/ Audio Onset Detection Results]  

* [[2015:Audio Cover Song Identification Results|Audio Cover Song Identification Results]]

* [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ate/ Audio Tempo Estimation Results]  

* Audio Key Detection Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/akd/mrx_05 MIREX 2015 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/akd/gsteps GiantSteps Dataset]  

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/abt/smc/ SMC Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/abt/maz/ MAZ Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/abt/mck/ MCK Dataset]  

* Audio Chord Estimation
** [[2015:Audio_Chord_Estimation_Results#Isophonics_2009 | Isophonics 2009 Dataset]]  
** [[2015:Audio_Chord_Estimation_Results#Billboard_2012 | Billboard 2012 Dataset]]  
** [[2015:Audio_Chord_Estimation_Results#Billboard_2013 | Billboard 2013 Dataset]]  
** [[2015:Audio_Chord_Estimation_Results#JayChou_2015 | JayChou 2015 Dataset]]  

* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/adc04/ ADC04 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/mrx05/ MIREX05 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/ind08/ INDIAN08 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/mrx09_0db/ MIREX09 0dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/orchset/ Orcheset15 Dataset]  

* Query-by-Singing/Humming Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task1_hidden/ Hidden Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task1_jang/ Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task1_thinkit/ ThinkIt Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task1_ioacas/ IOACAS Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task2_jang/ Subtask2 Dataset]  

* [[2015:Audio Downbeat Estimation Results|Audio Downbeat Estimation Results]]
* [[2015:Audio Fingerprinting Results|Audio Fingerprinting Results]]

* [https://www.music-ir.org/mirex/wiki/2015:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results#Summary_Results Real-time Audio to Score Alignment (a.k.a. Score Following) Results]  

* [https://www.music-ir.org/mirex/wiki/2015:Audio_Music_Similarity_and_Retrieval_Results Audio Music Similarity and Retrieval Results] 
* [[2015:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]
* [https://www.music-ir.org/mirex/wiki/2015:Singing_Voice_Separation_Results Singing Voice Separation] 

* Multiple Fundamental Frequency Estimation & Tracking Results
** [[2015:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_%2D_MIREX_Dataset | MIREX Dataset]]  
** [[2015:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_%2D_Su_Dataset |Su Dataset]]  

* [https://www.music-ir.org/mirex/wiki/2015:Music/Speech_Classification_and_Detection_Results Music/Speech Classification and Detection]

* [[2015:Set List Identification Results | Set List Identification Results]]

2015:Audio Chord Estimation Results

2015-10-21T18:42:06Z

Johan Pauwels: Improved spelling consistency

==Introduction==

This page contains the results of the 2015 edition of the MIREX automatic chord estimation tasks. This edition was the third one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last two years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* A new data set, called "JayChou 2015" has been donated by [http://tangkk.net | Junqi Deng] of the [http://www.hku.hk | University of Hong Kong]. It consists of 29 Mandopop songs taken from various albums by [https://en.wikipedia.org/wiki/Jay_Chou | Jay Chou]. Most of the songs are ballads and special attention has been paid to the annotation of extended chords and inversions. Because Junqi was kind enough to provide this set before official publication, the algorithmic output on these files and their ground-truth has been withheld for the time being. Also the file names in the per track results have been anonymized.
* The algorithmic output and per track results on the Isophonics set now display the unmasked song name, such that an evaluation per artist/album can be performed.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CM3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| DK4-DK9
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/DK4.pdf PDF]
| Junqi Deng, Yu-Kwong Kwok
|-
| KO1 (shineChords)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/KO1.pdf PDF]
| Maksim Khadkevich, Maurizio Omologo
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary.

=====Isophonics 2009=====
<csv>2015/ace/Isophonics2009.csv</csv>
=====Billboard 2012=====
<csv>2015/ace/Billboard2012.csv</csv>
=====Billboard 2013=====
<csv>2015/ace/Billboard2013.csv</csv>
=====JayChou 2015=====
<csv>2015/ace/JayChou2015.csv</csv>

===Comparative Statistics===

An analysis of the statistical difference between all submissions can be found on the following pages:

* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Isophonics2009 | Isophonics 2009 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2012 | Billboard 2012 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2013 | Billboard 2013 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_JayChou2015 | JayChou 2015 Dataset]]

===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from these archives:

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Results.zip Isophonics2009Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Results.zip Billboard2012Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Results.zip Billboard2013Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/JayChou2015Results.zip JayChou2015Results.zip]

===Algorithmic Output===

The raw output of the algorithms are available in the archives below. This can be used to experiment with alternative evaluation measures and statistics.

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Output.zip Isophonics2009Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Output.zip Billboard2012Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Output.zip Billboard2013Output.zip]

2015:MIREX2015 Results

2015-10-21T18:39:22Z

Johan Pauwels: Added new ACE dataset

==Results by Task ==

==OVERALL RESULTS POSTERS ==

[https://www.music-ir.org/mirex/results/2015/mirex_2015_poster.pdf MIREX 2015 Overall Results Posters (PDF)]

===Train-Test Task Set===
* [https://www.music-ir.org/nema_out/mirex2015/results/act/composer_report/ Audio Classical Composer Identification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/latin_report/ Audio Latin Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/mood_report/index.html Audio Music Mood Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/kgenrek_report/ Audio KPOP Genre (Annotated by Korean Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/kgenrea_report/ Audio KPOP Genre (Annotated by American Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/kmoodk_report/ Audio KPOP Mood (Annotated by Korean Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/kmooda_report/ Audio KPOP Mood (Annotated by American Annotators) Classification Results ]  

===Other Tasks===

* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/struct/mrx09/ MIREX09 dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/struct/sal/ SALAMI dataset]  

* [[2015:Discovery of Repeated Themes & Sections Results | Discovery of Repeated Themes & Sections Results]]

* [https://nema.lis.illinois.edu/nema_out/mirex2015/results/aod/ Audio Onset Detection Results]  

* [[2015:Audio Cover Song Identification Results|Audio Cover Song Identification Results]]

* [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ate/ Audio Tempo Estimation Results]  

* Audio Key Detection Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/akd/mrx_05 MIREX 2015 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/akd/gsteps GiantSteps Dataset]  

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/abt/smc/ SMC Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/abt/maz/ MAZ Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/abt/mck/ MCK Dataset]  

* Audio Chord Estimation
** [[2015:Audio_Chord_Estimation_Results#Isophonics_2009 | Isophonics 2009 Dataset]]  
** [[2015:Audio_Chord_Estimation_Results#Billboard_2012 | Billboard 2012 Dataset]]  
** [[2015:Audio_Chord_Estimation_Results#Billboard_2013 | Billboard 2013 Dataset]]  
** [[2015:Audio_Chord_Estimation_Results#Jay_Chou_2015 | Jay Chou 2015 Dataset]]  

* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/adc04/ ADC04 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/mrx05/ MIREX05 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/ind08/ INDIAN08 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/mrx09_0db/ MIREX09 0dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/orchset/ Orcheset15 Dataset]  

* Query-by-Singing/Humming Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task1_hidden/ Hidden Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task1_jang/ Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task1_thinkit/ ThinkIt Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task1_ioacas/ IOACAS Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task2_jang/ Subtask2 Dataset]  

* [[2015:Audio Downbeat Estimation Results|Audio Downbeat Estimation Results]]
* [[2015:Audio Fingerprinting Results|Audio Fingerprinting Results]]

* [https://www.music-ir.org/mirex/wiki/2015:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results#Summary_Results Real-time Audio to Score Alignment (a.k.a. Score Following) Results]  

* [https://www.music-ir.org/mirex/wiki/2015:Audio_Music_Similarity_and_Retrieval_Results Audio Music Similarity and Retrieval Results] 
* [[2015:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]
* [https://www.music-ir.org/mirex/wiki/2015:Singing_Voice_Separation_Results Singing Voice Separation] 

* Multiple Fundamental Frequency Estimation & Tracking Results
** [[2015:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_%2D_MIREX_Dataset | MIREX Dataset]]  
** [[2015:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_%2D_Su_Dataset |Su Dataset]]  

* [https://www.music-ir.org/mirex/wiki/2015:Music/Speech_Classification_and_Detection_Results Music/Speech Classification and Detection]

* [[2015:Set List Identification Results | Set List Identification Results]]

2015:Audio Chord Estimation Results

2015-10-21T18:37:27Z

Johan Pauwels: Fixed internal link

==Introduction==

This page contains the results of the 2015 edition of the MIREX automatic chord estimation tasks. This edition was the third one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last two years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* A new data set, called "Jay Chou 2015" has been donated by [http://tangkk.net | Junqi Deng] of the [http://www.hku.hk | University of Hong Kong]. It consists of 29 Mandopop songs taken from various albums by [https://en.wikipedia.org/wiki/Jay_Chou | Jay Chou]. Most of the songs are ballads and special attention has been paid to the annotation of extended chords and inversions. Because Junqi was kind enough to provide this set before official publication, the algorithmic output on these files and their ground-truth has been withheld for the time being. Also the file names in the per track results have been anonymized.
* The algorithmic output and per track results on the Isophonics set now display the unmasked song name, such that an evaluation per artist/album can be performed.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CM3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| DK4-DK9
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/DK4.pdf PDF]
| Junqi Deng, Yu-Kwong Kwok
|-
| KO1 (shineChords)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/KO1.pdf PDF]
| Maksim Khadkevich, Maurizio Omologo
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary.

=====Isophonics 2009=====
<csv>2015/ace/Isophonics2009.csv</csv>
=====Billboard 2012=====
<csv>2015/ace/Billboard2012.csv</csv>
=====Billboard 2013=====
<csv>2015/ace/Billboard2013.csv</csv>
=====JayChou 2015=====
<csv>2015/ace/JayChou2015.csv</csv>

===Comparative Statistics===

An analysis of the statistical difference between all submissions can be found on the following pages:

* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Isophonics2009 | Isophonics 2009 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2012 | Billboard 2012 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2013 | Billboard 2013 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_JayChou2015 | Jay Chou 2015 Dataset]]

===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from these archives:

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Results.zip Isophonics2009Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Results.zip Billboard2012Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Results.zip Billboard2013Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/JayChou2015Results.zip JayChou2015Results.zip]

===Algorithmic Output===

The raw output of the algorithms are available in the archives below. This can be used to experiment with alternative evaluation measures and statistics.

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Output.zip Isophonics2009Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Output.zip Billboard2012Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Output.zip Billboard2013Output.zip]

2015:Audio Chord Estimation Results

2015-10-21T18:35:53Z

Johan Pauwels: Creation of the 2015 ace results page

==Introduction==

This page contains the results of the 2015 edition of the MIREX automatic chord estimation tasks. This edition was the third one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last two years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results | 2013 edition]].

==What’s new?==
* A new data set, called "Jay Chou 2015" has been donated by [http://tangkk.net | Junqi Deng] of the [http://www.hku.hk | University of Hong Kong]. It consists of 29 Mandopop songs taken from various albums by [https://en.wikipedia.org/wiki/Jay_Chou | Jay Chou]. Most of the songs are ballads and special attention has been paid to the annotation of extended chords and inversions. Because Junqi was kind enough to provide this set before official publication, the algorithmic output on these files and their ground-truth has been withheld for the time being. Also the file names in the per track results have been anonymized.
* The algorithmic output and per track results on the Isophonics set now display the unmasked song name, such that an evaluation per artist/album can be performed.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CM3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| DK4-DK9
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/DK4.pdf PDF]
| Junqi Deng, Yu-Kwong Kwok
|-
| KO1 (shineChords)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/KO1.pdf PDF]
| Maksim Khadkevich, Maurizio Omologo
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary.

=====Isophonics 2009=====
<csv>2015/ace/Isophonics2009.csv</csv>
=====Billboard 2012=====
<csv>2015/ace/Billboard2012.csv</csv>
=====Billboard 2013=====
<csv>2015/ace/Billboard2013.csv</csv>
=====JayChou 2015=====
<csv>2015/ace/JayChou2015.csv</csv>

===Comparative Statistics===

An analysis of the statistical difference between all submissions can be found on the following pages:

* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Isophonics2009 | Isophonics 2009 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2012 | Billboard 2012 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2013 | Billboard 2013 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_JayChou2015 | Jay Chou 2015 Dataset]]

===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from these archives:

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Results.zip Isophonics2009Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Results.zip Billboard2012Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Results.zip Billboard2013Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/JayChou2015Results.zip JayChou2015Results.zip]

===Algorithmic Output===

The raw output of the algorithms are available in the archives below. This can be used to experiment with alternative evaluation measures and statistics.

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Output.zip Isophonics2009Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Output.zip Billboard2012Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Output.zip Billboard2013Output.zip]

2015:Task Captains

2015-03-30T14:30:36Z

Johan Pauwels:

Like ISMIR 2014, we are prepared to improve the distribution of tasks for the upcoming MIREX 2015. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead one or more tasks, please add your name in the "Captains" column.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

{| class="wikitable" style="margin-left: 20px"
!ID !! Task !! Captain(s)
|-
|abt
|[[2015:Audio Beat Tracking]]
|, Sebastian Böck
|-
|ace
|[[2015:Audio Chord Estimation]]
|Johan Pauwels
|-
|act
|[[2015:Audio Classification (Train/Test) Tasks]]
|
|-
|acs
|[[2015:Audio Cover Song Identification]]
|
|-
|ade
|[[2015:Audio Downbeat Estimation]]
|Florian Krebs, Sebastian Böck
|-
|akd
|[[2015:Audio Key Detection]]
|Johan Pauwels
|-
|ame
|[[2015:Audio Melody Extraction]]
|
|-
|ams
|[[2015:Audio Music Similarity and Retrieval]]
|
|-
|aod
|[[2015:Audio Onset Detection]]
|Sebastian Böck
|-
|ate
|[[2015:Audio Tempo Estimation]]
|
|-
|atg
|[[2015:Audio Tag Classification]]
|
|-
|mf0
|[[2015:Multiple Fundamental Frequency Estimation & Tracking]]
|
|-
|qbsh
|[[2015:Query by Singing/Humming]]
|
|-
|qbt
|[[2015:Query by Tapping]]
|
|-
|scofo
|[[2015:Real-time Audio to Score Alignment (a.k.a Score Following)]]
|
|-
|sms
|[[2015:Symbolic Melodic Similarity]]
|
|-
|struct
|[[2015:Structural Segmentation]]
|
|-
|drts
|[[2015:Discovery of Repeated Themes & Sections]]
|
|-
|afp
|[[2015:Audio_Fingerprinting]]
|
|-
|svs
|[[2015:Singing_Voice_Separation]]
|
|-
|kgc
|[[2015:Audio K-POP Genre Classification]]
|
|-
|kmc
|[[2015:Audio K-POP Mood Classification]]
|
|}

2014:Audio Chord Estimation Statistical Analysis MirexChord2009

2014-11-24T21:59:52Z

Johan Pauwels: Added segmentation plot

[[File:MirexChord2009-Root.png]]
[[File:MirexChord2009-MajMin.png]]
[[File:MirexChord2009-MajMinBass.png]]
[[File:MirexChord2009-Sevenths.png]]
[[File:MirexChord2009-SeventhsBass.png]]
[[File:MirexChord2009-Segmentation.png]]

2014:Audio Chord Estimation Statistical Analysis Billboard2013

2014-11-24T21:58:57Z

Johan Pauwels: Added segmentation plot

[[File:Billboard2013-Root.png]]
[[File:Billboard2013-MajMin.png]]
[[File:Billboard2013-MajMinBass.png]]
[[File:Billboard2013-Sevenths.png]]
[[File:Billboard2013-SeventhsBass.png]]
[[File:Billboard2013-Segmentation.png]]

2014:Audio Chord Estimation Statistical Analysis Billboard2012

2014-11-24T21:58:17Z

Johan Pauwels: Added segmentation plot

[[File:Billboard2012-Root.png]]
[[File:Billboard2012-MajMin.png]]
[[File:Billboard2012-MajMinBass.png]]
[[File:Billboard2012-Sevenths.png]]
[[File:Billboard2012-SeventhsBass.png]]
[[File:Billboard2012-Segmentation.png]]

File:MirexChord2009-SeventhsBass.png

2014-11-24T21:57:05Z

Johan Pauwels:

File:MirexChord2009-Sevenths.png

2014-11-24T21:56:51Z

Johan Pauwels:

File:MirexChord2009-Segmentation.png

2014-11-24T21:56:36Z

Johan Pauwels:

File:MirexChord2009-Root.png

2014-11-24T21:56:23Z

Johan Pauwels:

File:MirexChord2009-MajMinBass.png

2014-11-24T21:55:56Z

Johan Pauwels:

File:MirexChord2009-MajMin.png

2014-11-24T21:55:37Z

Johan Pauwels:

File:Billboard2013-SeventhsBass.png

2014-11-24T21:55:25Z

Johan Pauwels:

File:Billboard2013-Sevenths.png

2014-11-24T21:55:12Z

Johan Pauwels:

File:Billboard2013-Segmentation.png

2014-11-24T21:54:59Z

Johan Pauwels:

File:Billboard2013-Root.png

2014-11-24T21:54:49Z

Johan Pauwels:

File:Billboard2013-MajMinBass.png

2014-11-24T21:54:37Z

Johan Pauwels:

File:Billboard2013-MajMin.png

2014-11-24T21:54:25Z

Johan Pauwels:

File:Billboard2012-SeventhsBass.png

2014-11-24T21:54:06Z

Johan Pauwels:

File:Billboard2012-Sevenths.png

2014-11-24T21:53:53Z

Johan Pauwels:

File:Billboard2012-Segmentation.png

2014-11-24T21:53:42Z

Johan Pauwels:

File:Billboard2012-Root.png

2014-11-24T21:53:32Z

Johan Pauwels:

File:Billboard2012-MajMinBass.png

2014-11-24T21:53:19Z

Johan Pauwels:

File:Billboard2012-MajMin.png

2014-11-24T21:52:35Z

Johan Pauwels:

2014:Audio Chord Estimation Results

2014-11-12T22:13:47Z

Johan Pauwels:

==Introduction==

This page contains the results of these new evaluations for the Isophonics dataset, a.k.a. the MIREX 2009 dataset. It comprises the collected Beatles, Queen, and Zweieck datasets from Queen Mary, University of London, and has been used for audio chord estimation in MIREX for many years.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

=====MIREX Chord 2009=====
<csv>2014/ace/mirex09.csv</csv>
=====Billboard 2012=====
<csv>2014/ace/billboard12.csv</csv>
=====Billboard 2013=====
<csv>2014/ace/billboard13.csv</csv>

===Comparative Statistics===

An analysis of the statistical difference between all submissions can be found on the following pages:

* [[2014:Audio_Chord_Estimation_Statistical_Analysis_MirexChord2009 | MIREX Chord 2009 Dataset]]
* [[2014:Audio_Chord_Estimation_Statistical_Analysis_Billboard2012 | Billboard2012 Dataset]]
* [[2014:Audio_Chord_Estimation_Statistical_Analysis_Billboard2013 | Billboard2013 Dataset]]

===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from these archives:

* [https://music-ir.org/mirex/results/2014/ace/MirexChord2009Results.zip MirexChord2009Results.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2012Results.zip Billboard2012Results.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2013Results.zip Billboard2013Results.zip]

===Algorithmic Output===

The raw output of the algorithms are available in the archives below. This can be used to experiment with alternative evaluation measures and statistics.

* [https://music-ir.org/mirex/results/2014/ace/MirexChord2009Output.zip MirexChord2009Output.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2012Output.zip Billboard2012Output.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2013Output.zip Billboard2013Output.zip]

===Notes===
The evaluation procedure of this year was exactly the same as that of last year, so the results can be compared with each other. Even more, two of the three submissions were resubmissions of last year: KO1 (2014) = KO1 (2013) and CM3 (2014) = CF2 (2013) and consequently have the same scores.

2014:Audio Chord Estimation Results

2014-11-12T22:02:36Z

Johan Pauwels: Added description and links to software

==Introduction==

This page contains the results of these new evaluations for the Isophonics dataset, a.k.a. the MIREX 2009 dataset. It comprises the collected Beatles, Queen, and Zweieck datasets from Queen Mary, University of London, and has been used for audio chord estimation in MIREX for many years.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

==Software==
All software used for the evaluation has been made open-source. The evaluation procedure itself is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub]. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

=====MIREX Chord 2009=====
<csv>2014/ace/mirex09.csv</csv>
=====Billboard 2012=====
<csv>2014/ace/billboard12.csv</csv>
=====Billboard 2013=====
<csv>2014/ace/billboard13.csv</csv>

===Comparative Statistics===

An analysis of the statistical difference between all submissions can be found on the following pages:

* [[2014:Audio_Chord_Estimation_Statistical_Analysis_MirexChord2009 | MIREX Chord 2009 Dataset]]
* [[2014:Audio_Chord_Estimation_Statistical_Analysis_Billboard2012 | Billboard2012 Dataset]]
* [[2014:Audio_Chord_Estimation_Statistical_Analysis_Billboard2013 | Billboard2013 Dataset]]

===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from these archives:

* [https://music-ir.org/mirex/results/2014/ace/MirexChord2009Results.zip MirexChord2009Results.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2012Results.zip Billboard2012Results.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2013Results.zip Billboard2013Results.zip]

===Algorithmic Output===

The raw output of the algorithms are available in the archives below. This can be used to experiment with alternative evaluation measures and statistics.

* [https://music-ir.org/mirex/results/2014/ace/MirexChord2009Output.zip MirexChord2009Output.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2012Output.zip Billboard2012Output.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2013Output.zip Billboard2013Output.zip]

===Notes===
The evaluation procedure of this year was exactly the same as that of last year, so the results can be compared with each other. Even more, two of the three submissions were resubmissions of last year: KO1 (2014) = KO1 (2013) and CM3 (2014) = CF2 (2013) and consequently have the same scores.

2014:Audio Chord Estimation Statistical Analysis Billboard2013

2014-11-12T21:42:58Z

Johan Pauwels: Added links to comparative statistics pages

[[File:Billboard2013-Root.png]]

[[File:Billboard2013-MajMin.png]]

[[File:Billboard2013-MajMinBass.png]]

[[File:Billboard2013-Sevenths.png]]

[[File:Billboard2013-SeventhsBass.png]]

2014:Audio Chord Estimation Statistical Analysis Billboard2012

2014-11-12T21:42:18Z

Johan Pauwels: Added statistical significance plots

[[File:Billboard2012-Root.png]]

[[File:Billboard2012-MajMin.png]]

[[File:Billboard2012-MajMinBass.png]]

[[File:Billboard2012-Sevenths.png]]

[[File:Billboard2012-SeventhsBass.png]]

2014:Audio Chord Estimation Statistical Analysis MirexChord2009

2014-11-12T21:39:17Z

Johan Pauwels: Added statistical significance plots

[[File:MirexChord2009-Root.png]]
[[File:MirexChord2009-MajMin.png]]
[[File:MirexChord2009-MajMinBass.png]]
[[File:MirexChord2009-Sevenths.png]]
[[File:MirexChord2009-SeventhsBass.png]]

2014:Audio Chord Estimation Results

2014-11-12T21:31:46Z

Johan Pauwels: Added links to comparative statistics pages

==Introduction==

This page contains the results of these new evaluations for the Isophonics dataset, a.k.a. the MIREX 2009 dataset. It comprises the collected Beatles, Queen, and Zweieck datasets from Queen Mary, University of London, and has been used for audio chord estimation in MIREX for many years.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

=====MIREX Chord 2009=====
<csv>2014/ace/mirex09.csv</csv>
=====Billboard 2012=====
<csv>2014/ace/billboard12.csv</csv>
=====Billboard 2013=====
<csv>2014/ace/billboard13.csv</csv>

===Comparative Statistics===

An analysis of the statistical difference between all submissions can be found on the following pages:

* [[2014:Audio_Chord_Estimation_Statistical_Analysis_MirexChord2009 | MIREX Chord 2009 Dataset]]
* [[2014:Audio_Chord_Estimation_Statistical_Analysis_Billboard2012 | Billboard2012 Dataset]]
* [[2014:Audio_Chord_Estimation_Statistical_Analysis_Billboard2013 | Billboard2013 Dataset]]

===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from these archives:

* [https://music-ir.org/mirex/results/2014/ace/MirexChord2009Results.zip MirexChord2009Results.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2012Results.zip Billboard2012Results.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2013Results.zip Billboard2013Results.zip]

===Algorithmic Output===

The raw output of the algorithms are available in the archives below. This can be used to experiment with alternative evaluation measures and statistics.

* [https://music-ir.org/mirex/results/2014/ace/MirexChord2009Output.zip MirexChord2009Output.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2012Output.zip Billboard2012Output.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2013Output.zip Billboard2013Output.zip]

===Notes===
The evaluation procedure of this year was exactly the same as those of last year, so the results can be compared with each other. Even more, two of the three submissions were resubmissions of last year: KO1 (2014) = KO1 (2013) and CM3 (2014) = CF2 (2013) and consequently have the same scores.

2014:Audio Chord Estimation Results

2014-11-12T21:15:46Z

Johan Pauwels: Publish detailled results and output

==Introduction==

This page contains the results of these new evaluations for the Isophonics dataset, a.k.a. the MIREX 2009 dataset. It comprises the collected Beatles, Queen, and Zweieck datasets from Queen Mary, University of London, and has been used for audio chord estimation in MIREX for many years.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

=====MIREX Chord 2009=====
<csv>2014/ace/mirex09.csv</csv>
=====Billboard 2012=====
<csv>2014/ace/billboard12.csv</csv>
=====Billboard 2013=====
<csv>2014/ace/billboard13.csv</csv>



===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from these archives:

* [https://music-ir.org/mirex/results/2014/ace/MirexChord2009Results.zip MirexChord2009Results.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2012Results.zip Billboard2012Results.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2013Results.zip Billboard2013Results.zip]

===Algorithmic Output===

The raw output of the algorithms are available in the archives below. This can be used to experiment with alternative evaluation measures and statistics.

* [https://music-ir.org/mirex/results/2014/ace/MirexChord2009Output.zip MirexChord2009Output.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2012Output.zip Billboard2012Output.zip]
* [https://music-ir.org/mirex/results/2014/ace/Billboard2013Output.zip Billboard2013Output.zip]

2014:Audio Chord Estimation Results

2014-10-31T20:03:20Z

Johan Pauwels:

==Introduction==

This page contains the results of these new evaluations for the Isophonics dataset, a.k.a. the MIREX 2009 dataset. It comprises the collected Beatles, Queen, and Zweieck datasets from Queen Mary, University of London, and has been used for audio chord estimation in MIREX for many years.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

=====MIREX Chord 2009=====
<csv>2014/ace/mirex09.csv</csv>
=====Billboard 2012=====
<csv>2014/ace/billboard12.csv</csv>
=====Billboard 2013=====
<csv>2014/ace/billboard13.csv</csv>

2014:MIREX2014 Results

2014-10-31T17:56:02Z

Johan Pauwels: Redirected three data set pages of ace to new unified page

==Results by Task ==

==OVERALL RESULTS POSTERS ==

[https://www.music-ir.org/mirex/results/2014/mirex_2014_poster.pdf MIREX 2014 Overall Results Posters (PDF)]

===Train-Test Task Set===
* [https://www.music-ir.org/nema_out/mirex2014/results/act/composer_report/ Audio Classical Composer Identification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/latin_report/ Audio Latin Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/mood_report/index.html Audio Music Mood Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/kgenrek_report/ Audio KPOP Genre (Annotated by Korean Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/kgenrea_report/ Audio KPOP Genre (Annotated by American Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/kmoodk_report/ Audio KPOP Mood (Annotated by Korean Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/kmooda_report/ Audio KPOP Mood (Annotated by American Annotators) Classification Results ]  

===Other Tasks===

* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/mrx09/ MIREX09 dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/sal/ SALAMI dataset]  

* [[2014:Discovery of Repeated Themes & Sections Results | Discovery of Repeated Themes & Sections Results]]

* [https://nema.lis.illinois.edu/nema_out/mirex2014/results/aod/ Audio Onset Detection Results]  

* [[2014:Audio Cover Song Identification Results|Audio Cover Song Identification Results]]

* [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ate/ Audio Tempo Estimation Results]  
* [https://nema.lis.illinois.edu/nema_out/mirex2014/results/akd/ Audio Key Detection Results]  

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/abt/smc/ SMC Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/abt/maz/ MAZ Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/abt/mck/ MCK Dataset]  

* Audio Chord Estimation
** [[2014:Audio_Chord_Estimation_Results#MIREX_Chord_2009 | MIREX Chord ’09 Dataset]]  
** [[2014:Audio_Chord_Estimation_Results#Billboard_2012 | Billboard ’12 Dataset]]  
** [[2014:Audio_Chord_Estimation_Results#Billboard_2013 | Billboard ’13 Dataset]]  

* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/adc04/ ADC04 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/mrx05/ MIREX05 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/ind08/ INDIAN08 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/mrx09_0db/ MIREX09 0dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset]  

* Audio Tag Classification Results
** Major Miner Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)]  
*** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/atg/subtask1_report/aff/ Affinity estimation evaluation]  
** Mood Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)]  
*** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/atg/subtask2_report/aff/ Affinity estimation evaluation]  

* Query-by-Singing/Humming Results
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/qbsh/qbsh_task1_hidden/ Hidden Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/qbsh/qbsh_task1_jang/ Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/qbsh/qbsh_task1_thinkit/ ThinkIt Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/qbsh/qbsh_task1_ioacas/ IOACAS Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/qbsh/qbsh_task2_jang/ Subtask2 Dataset]  

* Query-by-Tapping Results
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/qbt/qbt_task1_jang/ Subtask 1, Jang dataset]
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/qbt/qbt_task1_hsiao/ Subtask 1, Hsiao dataset]
** Subtask 1, QBT-Extended dataset
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/qbt/qbt_task2_jang/ Subtask 2, Jang dataset]
** Subtask 3, QBT-Extended dataset

* [[2014:Audio Downbeat Estimation Results|Audio Downbeat Estimation Results]]
* [[2014:Audio Fingerprinting Results|Audio Fingerprinting Results]]

* [https://www.music-ir.org/mirex/wiki/2014:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results#Summary_Results Real-time Audio to Score Alignment (a.k.a. Score Following) Results]  

* [https://www.music-ir.org/mirex/wiki/2014:Audio_Music_Similarity_and_Retrieval_Results Audio Music Similarity and Retrieval Results] 
* [[2014:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]
* [https://www.music-ir.org/mirex/wiki/2014:Singing_Voice_Separation_Results Singing Voice Separation] 

* [https://www.music-ir.org/mirex/wiki/2014:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results Multiple Fundamental Frequency Estimation & Tracking Results]

2014:Audio Chord Estimation Results

2014-10-31T17:51:55Z

Johan Pauwels: moved 2014:Audio Chord Estimation Results MIREX 2009 to 2014:Audio Chord Estimation Results: Merging three pages into one for easier maintenance

==Introduction==

This page contains the results of these new evaluations for the Isophonics dataset, a.k.a. the MIREX 2009 dataset. It comprises the collected Beatles, Queen, and Zweieck datasets from Queen Mary, University of London, and has been used for audio chord estimation in MIREX for many years.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

====Mirex Chord 2009====
<csv>2014/ace/mirex09.csv</csv>
====Billboard 2012====
<csv>2014/ace/billboard12.csv</csv>
====Billboard2013====
<csv>2014/ace/billboard13.csv</csv>

2014:Audio Chord Estimation Results MIREX 2009

2014-10-31T17:51:55Z

Johan Pauwels: moved 2014:Audio Chord Estimation Results MIREX 2009 to 2014:Audio Chord Estimation Results: Merging three pages into one for easier maintenance

#REDIRECT [[2014:Audio Chord Estimation Results]]

2014:Audio Chord Estimation Results

2014-10-31T17:51:24Z

Johan Pauwels: Merging three pages into one for easier maintenance

==Introduction==

This page contains the results of these new evaluations for the Isophonics dataset, a.k.a. the MIREX 2009 dataset. It comprises the collected Beatles, Queen, and Zweieck datasets from Queen Mary, University of London, and has been used for audio chord estimation in MIREX for many years.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

====Mirex Chord 2009====
<csv>2014/ace/mirex09.csv</csv>
====Billboard 2012====
<csv>2014/ace/billboard12.csv</csv>
====Billboard2013====
<csv>2014/ace/billboard13.csv</csv>

2013:MIREX2013 Results

2014-10-21T18:07:05Z

Johan Pauwels: Removed links to partial and buggy old style evaluations for ace

==OVERALL RESULTS POSTERS ==

This page is under construction.

[https://www.music-ir.org/mirex/results/2013/mirex_2013_poster.pdf MIREX 2013 Overall Results Posters (PDF)]

==Results by Task ==

===Train-Test Task Set===
* [https://www.music-ir.org/nema_out/mirex2013/results/act/composer_report/ Audio Classical Composer Identification Results ]  
* [https://www.music-ir.org/nema_out/mirex2013/results/act/latin_report/ Audio Latin Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2013/results/act/mood_report/index.html Audio Music Mood Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2013/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]  
===Other Tasks===

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/abt/dav/ DAV Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/abt/maz/ MAZ Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/abt/mck/ MCK Dataset]  
* Audio Chord Detection Results
** [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | MIREX ’09 Dataset]]  
** [[2013:Audio_Chord_Estimation_Results_Billboard_2012 | Billboard ’12 Dataset]]  
** [[2013:Audio_Chord_Estimation_Results_Billboard_2013 | Billboard ’13 Dataset]]  
* [https://nema.lis.illinois.edu/nema_out/mirex2013/results/akd/ Audio Key Detection Results]  
* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ame/adc04/ ADC04 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ame/mrx05/ MIREX05 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ame/ind08/ INDIAN08 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ame/mrx09_0db/ MIREX09 0dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset]  
* [[2013:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]
* [https://nema.lis.illinois.edu/nema_out/mirex2013/results/aod/ Audio Onset Detection Results]  
* Audio Tag Classification Results
** Major Miner Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)]  
*** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/atg/subtask1_report/aff/ Affinity estimation evaluation]  
** Mood Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)]  
*** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/atg/subtask2_report/aff/ Affinity estimation evaluation]  
* [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ate/ Audio Tempo Estimation Results]  
* [[2013:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]
* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/struct/mrx09/ MIREX09 dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/struct/sal/ SALAMI dataset]  
* Query-by-Singing/Humming Results
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbsh/qbsh_task1_hidden/ Hidden Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbsh/qbsh_task1a_jang/ Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbsh/qbsh_task1b_thinkit/ ThinkIt Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbsh/qbsh_task1c_ioacas/ IOACAS Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbsh/qbsh_task2_jang/ Subtask2 Dataset]  
* Query-by-Tapping Results
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbt/qbt_task1_jang/ Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbt/qbt_task1_hsiao/ HSIAO Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbt/qbt_task2_jang/ Subtask2 Dataset]  
*[[2013:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]
* [[2013:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]
* [[2013:Discovery of Repeated Themes & Sections Results | Discovery of Repeated Themes & Sections Results]]
* [[2013:Audio Cover Song Identification Results]]

2014:Audio Chord Estimation Results Billboard 2013

2014-10-21T17:52:31Z

Johan Pauwels: Read reformatted results from table

==Introduction==

This page contains the results of these new evaluations for a special subset of the ''Billboard'' dataset from McGill University that has never been made available to the public. Further subsets have been withheld to support the ACE task through MIREX 2015.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2014/ace/billboard13.csv</csv>

2014:Audio Chord Estimation Results Billboard 2013

2014-10-21T17:18:02Z

Johan Pauwels:

==Introduction==

This page contains the results of these new evaluations for a special subset of the ''Billboard'' dataset from McGill University that has never been made available to the public. Further subsets have been withheld to support the ACE task through MIREX 2015.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2014/ace/billboard13.csv</csv>

2013:Task Captains

2013-07-09T12:41:23Z

Johan Pauwels:

In response to discussions at ISMIR 2012, we are prepared to improve the distribution of tasks for the upcoming MIREX 2013. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead one or more tasks, please add your name in the "Captains" column.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

{| class="wikitable" style="margin-left: 20px"
!ID !! Task !! Captain(s)
|-
|abt
|[[2013:Audio Beat Tracking]]
|Fu-Hai Frank Wu
|-
|ace
|[[2013:Audio Chord Estimation]]
|John Ashley Burgoyne, W. Bas de Haas, Johan Pauwels
|-
|act
|[[2013:Audio Classification (Train/Test) Tasks]]
|IMIRSEL
|-
|acs
|[[2013:Audio Cover Song Identification]]
|
|-
|akd
|[[2013:Audio Key Detection]]
|Johan Pauwels
|-
|ame
|[[2013:Audio Melody Extraction]]
|KETI
|-
|ams
|[[2013:Audio Music Similarity and Retrieval]]
|IMIRSEL
|-
|aod
|[[2013:Audio Onset Detection]]
|Sebastian Böck
|-
|ate
|[[2013:Audio Tempo Estimation]]
|Aggelos Gkiokas, Anders Elowsson
|-
|atg
|[[2013:Audio Tag Classification]]
|
|-
|mf0
|[[2013:Multiple Fundamental Frequency Estimation & Tracking]]
|Mert Bay
|-
|qbsh
|[[2013:Query by Singing/Humming]]
|KETI
|-
|qbt
|[[2013:Query by Tapping]]
|KETI
|-
|scofo
|[[2013:Real-time Audio to Score Alignment (a.k.a Score Following)]]
|
|-
|sms
|[[2013:Symbolic Melodic Similarity]]
|IMIRSEL
|-
|struct
|[[2013:Structural Segmentation]]
|
|-
|drts
|[[2013:Discovery of Repeated Themes & Sections]]
|Tom Collins
|}

2013:Task Captains

2013-06-14T09:17:02Z

Johan Pauwels:

In response to discussions at ISMIR 2012, we are prepared to improve the distribution of tasks for the upcoming MIREX 2013. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead one or more tasks, please add your name in the "Captains" column.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

{| class="wikitable" style="margin-left: 20px"
!ID !! Task !! Captain(s)
|-
|abt
|[[2013:Audio Beat Tracking]]
|
|-
|ace
|[[2013:Audio Chord Estimation]]
|Johan Pauwels
|-
|act
|[[2013:Audio Classification (Train/Test) Tasks]]
|IMIRSEL
|-
|acs
|[[2013:Audio Cover Song Identification]]
|
|-
|akd
|[[2013:Audio Key Detection]]
|Johan Pauwels
|-
|ame
|[[2013:Audio Melody Extraction]]
|KETI
|-
|ams
|[[2013:Audio Music Similarity and Retrieval]]
|IMIRSEL
|-
|aod
|[[2013:Audio Onset Detection]]
|Sebastian Böck
|-
|ate
|[[2013:Audio Tempo Estimation]]
|
|-
|atg
|[[2013:Audio Tag Classification]]
|
|-
|mf0
|[[2013:Multiple Fundamental Frequency Estimation & Tracking]]
|Mert Bay
|-
|qbsh
|[[2013:Query by Singing/Humming]]
|KETI
|-
|qbt
|[[2013:Query by Tapping]]
|KETI
|-
|scofo
|[[2013:Real-time Audio to Score Alignment (a.k.a Score Following)]]
|
|-
|sms
|[[2013:Symbolic Melodic Similarity]]
|IMIRSEL
|-
|struct
|[[2013:Structural Segmentation]]
|
|-
|drts
|[[2013:Discovery of Repeated Themes & Sections]]
|Tom Collins
|}

2013:Task Captains

2013-06-14T09:16:26Z

Johan Pauwels:

In response to discussions at ISMIR 2012, we are prepared to improve the distribution of tasks for the upcoming MIREX 2013. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead one or more tasks, please add your name in the "Captains" column.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

{| class="wikitable" style="margin-left: 20px"
!ID !! Task !! Captain(s)
|-
|abt
|[[2013:Audio Beat Tracking]]
|
|-
|ace
|[[2013:Audio Chord Estimation]]
|Johan Pauwels
|-
|act
|[[2013:Audio Classification (Train/Test) Tasks]]
|IMIRSEL
|-
|acs
|[[2013:Audio Cover Song Identification]]
|
|-
|akd
|[[2013:Audio Key Detection]]
|
|-
|ame
|[[2013:Audio Melody Extraction]]
|KETI
|-
|ams
|[[2013:Audio Music Similarity and Retrieval]]
|IMIRSEL
|-
|aod
|[[2013:Audio Onset Detection]]
|Sebastian Böck
|-
|ate
|[[2013:Audio Tempo Estimation]]
|
|-
|atg
|[[2013:Audio Tag Classification]]
|
|-
|mf0
|[[2013:Multiple Fundamental Frequency Estimation & Tracking]]
|Mert Bay
|-
|qbsh
|[[2013:Query by Singing/Humming]]
|KETI
|-
|qbt
|[[2013:Query by Tapping]]
|KETI
|-
|scofo
|[[2013:Real-time Audio to Score Alignment (a.k.a Score Following)]]
|
|-
|sms
|[[2013:Symbolic Melodic Similarity]]
|IMIRSEL
|-
|struct
|[[2013:Structural Segmentation]]
|
|-
|drts
|[[2013:Discovery of Repeated Themes & Sections]]
|Tom Collins
|}