MIREX Wiki - User contributions [en]

2019:MIREX2019 Results

2020-10-14T10:37:46Z

JohanPauwels: Add direct link to ACE CASD results

==Overall Results Poster==
Coming soon

==Results by Task (More results are coming) ==
* Multiple Fundamental Frequency Estimation & Tracking Results
** [[2019:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_%2D_MIREX_Dataset | MIREX Dataset]]  
** [[2019:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_%2D_Su_Dataset | Su Dataset]]  
* [https://www.music-ir.org/mirex/wiki/2019:Music_Detection_Results Music Detection Results]  
* [https://www.music-ir.org/mirex/wiki/2019:Patterns_for_Prediction_Results Patterns for Prediction Results]  
* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2019/results/ame/adc04/ ADC04 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2019/results/ame/mrx05/ MIREX05 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2019/results/ame/ind08/ INDIAN08 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2019/results/ame/mrx09_0db/ MIREX09 0dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2019/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2019/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2019/results/ame/orchset/ ORCHSET15 Dataset]  
* [[2019:Automatic Lyrics-to-Audio Alignment Results | Automatic Lyrics-to-Audio Alignment Results]]  
* [[2019:Drum_Transcription_Results | Drum Transcription]]  
* [[2019:Audio Fingerprinting Results|Audio Fingerprinting Results]]
* [[2019:Audio_Key_Detection_Results | Audio Key Detection]]  
* [[2019:Audio_Chord_Estimation_Results | Audio Chord Estimation]]
** [[2019:Audio_Chord_Estimation_Results#Isophonics2009 | Isophonics2009 Dataset]]  
** [[2019:Audio_Chord_Estimation_Results#Billboard2012 | Billboard2012 Dataset]]  
** [[2019:Audio_Chord_Estimation_Results#Billboard_013 | Billboard2013 Dataset]]  
** [[2019:Audio_Chord_Estimation_Results#JayChou29 | JayChou29 Dataset]]  
** [[2019:Audio_Chord_Estimation_Results#RobbieWilliams | RobbieWilliams Dataset]]  
** [[2019:Audio_Chord_Estimation_Results#RWC-Popular | RWC-Popular Dataset]]  
** [[2019:Audio_Chord_Estimation_Results#USPOP2002Chords | USPOP2002Chords Dataset]]  
** [[2019:Audio_Chord_Estimation_Results#CASD-Annotator1 | CASD Dataset]]  
*Train-Test Task Set
** [https://www.music-ir.org/nema_out/mirex2019/results/act/composer_report/ Audio Classical Composer Identification Results ]  
** [https://www.music-ir.org/nema_out/mirex2019/results/act/latin_report/ Audio Latin Genre Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2019/results/act/mood_report/index.html Audio Music Mood Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2019/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2019/results/act/kgenrek_report/ Audio KPOP Genre (Annotated by Korean Annotators) Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2019/results/act/kgenrea_report/ Audio KPOP Genre (Annotated by American Annotators) Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2019/results/act/kmoodk_report/ Audio KPOP Mood (Annotated by Korean Annotators) Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2019/results/act/kmooda_report/ Audio KPOP Mood (Annotated by American Annotators) Classification Results ]  

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2019/results/abt/smc/ SMC Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2019/results/abt/maz/ MAZ Dataset]  

* [[2019:Audio Cover Song Identification Results|Audio Cover Song Identification Results]]

2019:Audio Chord Estimation Results

2020-10-14T10:27:04Z

JohanPauwels: Update year count

==Introduction==

This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the seventh one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last six years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* All datasets and evaluation procedures are the same as last year's MIREX.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CLSYJ1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/CLSYJ1.pdf PDF]
| [http://mirlab.org/users/eden.chien/ i Chien], [http://mirlab.org/users/hubert.lee/ Song Rong Lee], [http://mirlab.org/ Yeh Ssuhung], [http://mirlab.org/users/kenshincs/index.htm Tzu-Chun Yeh], [http://mirlab.org/jang Jyh-Shing Roger Jang]
|-
| CM1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/CM1.pdf PDF]
| [https://code.soundsoftware.ac.uk/users/3 Chris Cannam], [http://matthiasmauch.net Matthias Mauch]
|-
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====Isophonics2009=====
<csv>2019/ace/2019-Isophonics2009.csv</csv>
=====Billboard2012=====
<csv>2019/ace/2019-Billboard2012.csv</csv>
=====Billboard2013=====
<csv>2019/ace/2019-Billboard2013.csv</csv>
=====JayChou29=====
<csv>2019/ace/2019-JayChou29.csv</csv>
=====RobbieWilliams=====
<csv>2019/ace/2019-RobbieWilliams.csv</csv>
=====RWC-Popular=====
<csv>2019/ace/2019-RWC-Popular.csv</csv>
=====USPOP2002Chords=====
<csv>2019/ace/2019-USPOP2002Chords.csv</csv>
=====CASD-Annotator1=====
<csv>2019/ace/2019-CASD-Annotator1.csv</csv>
=====CASD-Annotator2=====
<csv>2019/ace/2019-CASD-Annotator2.csv</csv>
=====CASD-Annotator3=====
<csv>2019/ace/2019-CASD-Annotator3.csv</csv>
=====CASD-Annotator4=====
<csv>2019/ace/2019-CASD-Annotator4.csv</csv>



===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from [https://github.com/ismir-mirex/ace-results/tree/master/2019 this repository].
.

===Algorithmic Output===

The raw output of the algorithms are available in [https://github.com/ismir-mirex/ace-output/tree/master/2019 this repository]. They can be used to experiment with alternative evaluation measures and statistics.

2018:Audio Chord Estimation Results

2020-10-14T10:26:07Z

JohanPauwels: Add link to Github repositories

==Introduction==

This page contains the results of the 2018 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* All datasets and evaluation procedures are the same as last year's MIREX.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/CM1.pdf PDF]
| [https://code.soundsoftware.ac.uk/users/3 Chris Cannam], [http://c4dm.eecs.qmul.ac.uk/ Matthias Mauch]
|-
| JLCX1, JLCX2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/JLCX1.pdf PDF]
| [https://github.com/instr3/ Junyan Jiang], [https://github.com/RetroCirce Ke Chen], [http://www.cs.fudan.edu.cn/ Wei Li], [http://www.cs.cmu.edu/~gxia Guangyu Xia]
|-
| SG1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/SG1.pdf PDF]
| [https://www.fsit.services Franz Strasser], [http://www.jku.at/ Stefan Gaser]
|-
| FK2
| style="text-align: center;" | [PDF]
| [http://www.cp.jku.at Florian Krebs], [http://www.cp.jku.at Filip Korzeniowski], [http://www.ofai.at Sebastian Böck]
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====Isophonics2009=====
<csv>2018/ace/2018-Isophonics2009.csv</csv>
=====Billboard2012=====
<csv>2018/ace/2018-Billboard2012.csv</csv>
=====Billboard2013=====
<csv>2018/ace/2018-Billboard2013.csv</csv>
=====JayChou29=====
<csv>2018/ace/2018-JayChou29.csv</csv>
=====RobbieWilliams=====
<csv>2018/ace/2018-RobbieWilliams.csv</csv>
=====RWC-Popular=====
<csv>2018/ace/2018-RWC-Popular.csv</csv>
=====USPOP2002Chords=====
<csv>2018/ace/2018-USPOP2002Chords.csv</csv>



===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from [https://github.com/ismir-mirex/ace-results/tree/master/2018 this repository].

===Algorithmic Output===

The raw output of the algorithms are available in [https://github.com/ismir-mirex/ace-output/tree/master/2018 this repository]. They can be used to experiment with alternative evaluation measures and statistics.

2019:Audio Chord Estimation Results

2020-10-14T10:23:20Z

JohanPauwels: Undo overzealous correction

==Introduction==

This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* All datasets and evaluation procedures are the same as last year's MIREX.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CLSYJ1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/CLSYJ1.pdf PDF]
| [http://mirlab.org/users/eden.chien/ i Chien], [http://mirlab.org/users/hubert.lee/ Song Rong Lee], [http://mirlab.org/ Yeh Ssuhung], [http://mirlab.org/users/kenshincs/index.htm Tzu-Chun Yeh], [http://mirlab.org/jang Jyh-Shing Roger Jang]
|-
| CM1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/CM1.pdf PDF]
| [https://code.soundsoftware.ac.uk/users/3 Chris Cannam], [http://matthiasmauch.net Matthias Mauch]
|-
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====Isophonics2009=====
<csv>2019/ace/2019-Isophonics2009.csv</csv>
=====Billboard2012=====
<csv>2019/ace/2019-Billboard2012.csv</csv>
=====Billboard2013=====
<csv>2019/ace/2019-Billboard2013.csv</csv>
=====JayChou29=====
<csv>2019/ace/2019-JayChou29.csv</csv>
=====RobbieWilliams=====
<csv>2019/ace/2019-RobbieWilliams.csv</csv>
=====RWC-Popular=====
<csv>2019/ace/2019-RWC-Popular.csv</csv>
=====USPOP2002Chords=====
<csv>2019/ace/2019-USPOP2002Chords.csv</csv>
=====CASD-Annotator1=====
<csv>2019/ace/2019-CASD-Annotator1.csv</csv>
=====CASD-Annotator2=====
<csv>2019/ace/2019-CASD-Annotator2.csv</csv>
=====CASD-Annotator3=====
<csv>2019/ace/2019-CASD-Annotator3.csv</csv>
=====CASD-Annotator4=====
<csv>2019/ace/2019-CASD-Annotator4.csv</csv>



===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from [https://github.com/ismir-mirex/ace-results/tree/master/2019 this repository].
.

===Algorithmic Output===

The raw output of the algorithms are available in [https://github.com/ismir-mirex/ace-output/tree/master/2019 this repository]. They can be used to experiment with alternative evaluation measures and statistics.

2019:Audio Chord Estimation Results

2020-10-14T10:20:48Z

JohanPauwels: Remove outdated reference

==Introduction==

This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2013 | 2013 edition]].

==What’s new?==
* All datasets and evaluation procedures are the same as last year's MIREX.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CLSYJ1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/CLSYJ1.pdf PDF]
| [http://mirlab.org/users/eden.chien/ i Chien], [http://mirlab.org/users/hubert.lee/ Song Rong Lee], [http://mirlab.org/ Yeh Ssuhung], [http://mirlab.org/users/kenshincs/index.htm Tzu-Chun Yeh], [http://mirlab.org/jang Jyh-Shing Roger Jang]
|-
| CM1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/CM1.pdf PDF]
| [https://code.soundsoftware.ac.uk/users/3 Chris Cannam], [http://matthiasmauch.net Matthias Mauch]
|-
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====Isophonics2009=====
<csv>2019/ace/2019-Isophonics2009.csv</csv>
=====Billboard2012=====
<csv>2019/ace/2019-Billboard2012.csv</csv>
=====Billboard2013=====
<csv>2019/ace/2019-Billboard2013.csv</csv>
=====JayChou29=====
<csv>2019/ace/2019-JayChou29.csv</csv>
=====RobbieWilliams=====
<csv>2019/ace/2019-RobbieWilliams.csv</csv>
=====RWC-Popular=====
<csv>2019/ace/2019-RWC-Popular.csv</csv>
=====USPOP2002Chords=====
<csv>2019/ace/2019-USPOP2002Chords.csv</csv>
=====CASD-Annotator1=====
<csv>2019/ace/2019-CASD-Annotator1.csv</csv>
=====CASD-Annotator2=====
<csv>2019/ace/2019-CASD-Annotator2.csv</csv>
=====CASD-Annotator3=====
<csv>2019/ace/2019-CASD-Annotator3.csv</csv>
=====CASD-Annotator4=====
<csv>2019/ace/2019-CASD-Annotator4.csv</csv>



===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from [https://github.com/ismir-mirex/ace-results/tree/master/2019 this repository].
.

===Algorithmic Output===

The raw output of the algorithms are available in [https://github.com/ismir-mirex/ace-output/tree/master/2019 this repository]. They can be used to experiment with alternative evaluation measures and statistics.

2017:Audio Chord Estimation Results

2020-10-14T10:18:43Z

JohanPauwels: Remove outdated reference

==Introduction==

This page contains the results of the 2017 edition of the MIREX automatic chord estimation tasks. This edition was the fifth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last four years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* This year the algorithms have additionally been evaluated on the "RWC-Popular" and "USPOP2002Chords" dataset annotated at the [http://steinhardt.nyu.edu/marl/ Music and Audio Research Lab] of NYU, whose annotations are [https://github.com/tmc323/Chord-Annotations publicly available]. The [https://staff.aist.go.jp/m.goto/RWC-MDB/rwc-mdb-p.html RWC-Popular dataset] contains 100 pop songs recorded specifically for music information retrieval research. The USPOP2002Chords set is the 195 file subset of the [https://labrosa.ee.columbia.edu/projects/musicsim/uspop2002.html USPOP2002 dataset] that have been annotated with chord sequences.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/CM2.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JLW1, JLW2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/JLW1.pdf PDF]
| Junyan Jiang, Wei Li, Yiming Wu
|-
| KBK1, KBK2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/KBK1.pdf PDF]
| Filip Korzeniowski, Sebastian Böck, Florian Krebs
|-
| WL1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/WL1.pdf PDF]
| Yiming Wu, Wei Li
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====Isophonics2009=====
<csv>2017/ace/2017-Isophonics2009.csv</csv>
=====Billboard2012=====
<csv>2017/ace/2017-Billboard2012.csv</csv>
=====Billboard2013=====
<csv>2017/ace/2017-Billboard2013.csv</csv>
=====JayChou29=====
<csv>2017/ace/2017-JayChou29.csv</csv>
=====RobbieWilliams=====
<csv>2017/ace/2017-RobbieWilliams.csv</csv>
=====RWC-Popular=====
<csv>2017/ace/2017-RWC-Popular.csv</csv>
=====USPOP2002Chords=====
<csv>2017/ace/2017-USPOP2002Chords.csv</csv>



===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from [https://github.com/ismir-mirex/ace-results/tree/master/2017 this repository].

===Algorithmic Output===

The raw output of the algorithms are available in [https://github.com/ismir-mirex/ace-output/tree/master/2017 this repository]. They can be used to experiment with alternative evaluation measures and statistics.

2019:Audio Chord Estimation Results

2020-10-14T10:11:45Z

JohanPauwels: Add link to Github repositories

==Introduction==

This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* All datasets and evaluation procedures are the same as last year's MIREX.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output will be provided later  More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CLSYJ1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/CLSYJ1.pdf PDF]
| [http://mirlab.org/users/eden.chien/ i Chien], [http://mirlab.org/users/hubert.lee/ Song Rong Lee], [http://mirlab.org/ Yeh Ssuhung], [http://mirlab.org/users/kenshincs/index.htm Tzu-Chun Yeh], [http://mirlab.org/jang Jyh-Shing Roger Jang]
|-
| CM1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/CM1.pdf PDF]
| [https://code.soundsoftware.ac.uk/users/3 Chris Cannam], [http://matthiasmauch.net Matthias Mauch]
|-
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====Isophonics2009=====
<csv>2019/ace/2019-Isophonics2009.csv</csv>
=====Billboard2012=====
<csv>2019/ace/2019-Billboard2012.csv</csv>
=====Billboard2013=====
<csv>2019/ace/2019-Billboard2013.csv</csv>
=====JayChou29=====
<csv>2019/ace/2019-JayChou29.csv</csv>
=====RobbieWilliams=====
<csv>2019/ace/2019-RobbieWilliams.csv</csv>
=====RWC-Popular=====
<csv>2019/ace/2019-RWC-Popular.csv</csv>
=====USPOP2002Chords=====
<csv>2019/ace/2019-USPOP2002Chords.csv</csv>
=====CASD-Annotator1=====
<csv>2019/ace/2019-CASD-Annotator1.csv</csv>
=====CASD-Annotator2=====
<csv>2019/ace/2019-CASD-Annotator2.csv</csv>
=====CASD-Annotator3=====
<csv>2019/ace/2019-CASD-Annotator3.csv</csv>
=====CASD-Annotator4=====
<csv>2019/ace/2019-CASD-Annotator4.csv</csv>



===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from [https://github.com/ismir-mirex/ace-results/tree/master/2019 this repository].
.

===Algorithmic Output===

The raw output of the algorithms are available in [https://github.com/ismir-mirex/ace-output/tree/master/2019 this repository]. They can be used to experiment with alternative evaluation measures and statistics.

2019:Audio Chord Estimation Results

2020-10-14T10:03:00Z

JohanPauwels: Make CASD results visible

==Introduction==

This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* All datasets and evaluation procedures are the same as last year's MIREX.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output will be provided later  More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CLSYJ1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/CLSYJ1.pdf PDF]
| [http://mirlab.org/users/eden.chien/ i Chien], [http://mirlab.org/users/hubert.lee/ Song Rong Lee], [http://mirlab.org/ Yeh Ssuhung], [http://mirlab.org/users/kenshincs/index.htm Tzu-Chun Yeh], [http://mirlab.org/jang Jyh-Shing Roger Jang]
|-
| CM1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/CM1.pdf PDF]
| [https://code.soundsoftware.ac.uk/users/3 Chris Cannam], [http://matthiasmauch.net Matthias Mauch]
|-
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====Isophonics2009=====
<csv>2019/ace/2019-Isophonics2009.csv</csv>
=====Billboard2012=====
<csv>2019/ace/2019-Billboard2012.csv</csv>
=====Billboard2013=====
<csv>2019/ace/2019-Billboard2013.csv</csv>
=====JayChou29=====
<csv>2019/ace/2019-JayChou29.csv</csv>
=====RobbieWilliams=====
<csv>2019/ace/2019-RobbieWilliams.csv</csv>
=====RWC-Popular=====
<csv>2019/ace/2019-RWC-Popular.csv</csv>
=====USPOP2002Chords=====
<csv>2019/ace/2019-USPOP2002Chords.csv</csv>
=====CASD-Annotator1=====
<csv>2019/ace/2019-CASD-Annotator1.csv</csv>
=====CASD-Annotator2=====
<csv>2019/ace/2019-CASD-Annotator2.csv</csv>
=====CASD-Annotator3=====
<csv>2019/ace/2019-CASD-Annotator3.csv</csv>
=====CASD-Annotator4=====
<csv>2019/ace/2019-CASD-Annotator4.csv</csv>



===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from ?.

===Algorithmic Output===

The raw output of the algorithms are available in ?. They can be used to experiment with alternative evaluation measures and statistics.

2018:Audio Chord Estimation Results

2018-09-22T09:09:57Z

JohanPauwels: Add note about comparison with last year

==Introduction==

This page contains the results of the 2018 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* All datasets and evaluation procedures are the same as last year's MIREX.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output will be provided later  More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/CM1.pdf PDF]
| [https://code.soundsoftware.ac.uk/users/3 Chris Cannam], [http://c4dm.eecs.qmul.ac.uk/ Matthias Mauch]
|-
| JLCX1, JLCX2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/JLCX1.pdf PDF]
| [https://github.com/instr3/ Junyan Jiang], [https://github.com/RetroCirce Ke Chen], [http://www.cs.fudan.edu.cn/ Wei Li], [http://www.cs.cmu.edu/~gxia Guangyu Xia]
|-
| SG1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/SG1.pdf PDF]
| [https://www.fsit.services Franz Strasser], [http://www.jku.at/ Stefan Gaser]
|-
| FK2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/WL1.pdf PDF]
| [http://www.cp.jku.at Florian Krebs], [http://www.cp.jku.at Filip Korzeniowski], [http://www.ofai.at Sebastian Böck]
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====Isophonics2009=====
<csv>2018/ace/2018-Isophonics2009.csv</csv>
=====Billboard2012=====
<csv>2018/ace/2018-Billboard2012.csv</csv>
=====Billboard2013=====
<csv>2018/ace/2018-Billboard2013.csv</csv>
=====JayChou29=====
<csv>2018/ace/2018-JayChou29.csv</csv>
=====RobbieWilliams=====
<csv>2018/ace/2018-RobbieWilliams.csv</csv>
=====RWC-Popular=====
<csv>2018/ace/2018-RWC-Popular.csv</csv>
=====USPOP2002Chords=====
<csv>2018/ace/2018-USPOP2002Chords.csv</csv>



===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from ?.

===Algorithmic Output===

The raw output of the algorithms are available in ?. They can be used to experiment with alternative evaluation measures and statistics.

2018:Audio Key Detection Results

2018-09-22T09:05:44Z

JohanPauwels: Add note about comparison with last year

==Introduction==

This page contains the results of the 2018 edition of the MIREX automatic key detection estimation task.

==What’s new?==
* Nothing has changed, which means the results can be directly compared to last year's.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CG1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/CG1.pdf PDF]
| [http://www.uab.cat David Castells-Rufas], [http://www.uab.cat/enginyeria/ Adria Galin]
|-
| GC1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/GC1.pdf PDF]
| [http://www.uab.cat/enginyeria/ Adria Galin], [http://www.uab.cat David Castells-Rufas]
|-
| CN1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/CN1.pdf PDF]
| [https://code.soundsoftware.ac.uk/users/3 Chris Cannam], [http://c4dm.eecs.qmul.ac.uk/ Katy Noland]
|-
| NA1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/NA1.pdf PDF]
| [http://ddmal.music.mcgill.ca/ Nestor Napoles Lopez] , [https://music.gatech.edu/ Claire Arthur]
|-
| FK1, FK3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/FK1.pdf PDF]
| [http://www.cp.jku.at Filip Korzeniowski]
|-
| OM1-OM3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/OM1.pdf PDF]
| [http://JamesOwers.github.io James Owers], [http://homepages.inf.ed.ac.uk/amcleod8 Andrew McLeod]
|-
|}

==Results==

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

====MIREX2005Key====
<csv>2018/akd/2018-MIREX2005Key.csv</csv>
====GiantStepsKey====
<csv>2018/akd/2018-GiantStepsKey.csv</csv>
====PresegmentedKeyIsophonics====
<csv>2018/akd/2018-PresegmentedKeyIsophonics.csv</csv>
====PresegmentedKeyRobbieWilliams====
<csv>2018/akd/2018-PresegmentedKeyRobbieWilliams.csv</csv>
====Billboard2012Key====
<csv>2018/akd/2018-Billboard2012Key.csv</csv>

2017:Audio Chord Estimation Results

2017-11-28T17:01:01Z

JohanPauwels: Add links to repositories

==Introduction==

This page contains the results of the 2017 edition of the MIREX automatic chord estimation tasks. This edition was the fifth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last four years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* This year the algorithms have additionally been evaluated on the "RWC-Popular" and "USPOP2002Chords" dataset annotated at the [http://steinhardt.nyu.edu/marl/ Music and Audio Research Lab] of NYU, whose annotations are [https://github.com/tmc323/Chord-Annotations publicly available]. The [https://staff.aist.go.jp/m.goto/RWC-MDB/rwc-mdb-p.html RWC-Popular dataset] contains 100 pop songs recorded specifically for music information retrieval research. The USPOP2002Chords set is the 195 file subset of the [https://labrosa.ee.columbia.edu/projects/musicsim/uspop2002.html USPOP2002 dataset] that have been annotated with chord sequences.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output will be provided later  More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/CM2.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JLW1, JLW2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/JLW1.pdf PDF]
| Junyan Jiang, Wei Li, Yiming Wu
|-
| KBK1, KBK2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/KBK1.pdf PDF]
| Filip Korzeniowski, Sebastian Böck, Florian Krebs
|-
| WL1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/WL1.pdf PDF]
| Yiming Wu, Wei Li
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====Isophonics2009=====
<csv>2017/ace/2017-Isophonics2009.csv</csv>
=====Billboard2012=====
<csv>2017/ace/2017-Billboard2012.csv</csv>
=====Billboard2013=====
<csv>2017/ace/2017-Billboard2013.csv</csv>
=====JayChou29=====
<csv>2017/ace/2017-JayChou29.csv</csv>
=====RobbieWilliams=====
<csv>2017/ace/2017-RobbieWilliams.csv</csv>
=====RWC-Popular=====
<csv>2017/ace/2017-RWC-Popular.csv</csv>
=====USPOP2002Chords=====
<csv>2017/ace/2017-USPOP2002Chords.csv</csv>



===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from [https://github.com/ismir-mirex/ace-results/tree/master/2017 this repository].

===Algorithmic Output===

The raw output of the algorithms are available in [https://github.com/ismir-mirex/ace-output/tree/master/2017 this repository]. They can be used to experiment with alternative evaluation measures and statistics.

2017:Audio Key Detection Results

2017-11-21T17:28:06Z

JohanPauwels:

==Introduction==

This page contains the results of the 2017 edition of the MIREX automatic key detection estimation task.

==What’s new?==
* The NEMA system was retired this year, since a bug has been found in the calculation of the results. Keys with tonics related by a fifth and the same mode (a.k.a. adjacent keys) are supposed to get a score of 0.5, but only ascending fifths (going from ground-truth to estimation) were counted, not descending ones. It has been brought to my attention that the description of the measure on the wiki has been ambiguous for years, and probably the NEMA implementer got confused by this. However, the intention has always been to count ascending and descending fifth (or fourth) relationships between the tonics (in my humble opinion).
* New datasets: "PresegmentedKeyIsophonics" and "PresegmentedKeyRobbieWilliams" use the local key annotations for the [http://isophonics.net/content/reference-annotations Isophonics set] and the [http://ispg.deib.polimi.it/mir-software.html Robbie Williams set], but have been split into separate files according to the local key annotations. The segments annotated with major and minor modes have been retained and were presented to the submissions. Therefore their results are slightly optimistic in the sense that the segments are guaranteed to contain just a single key, which is not the case for real-world songs. Keep also in mind that some files are strongly correlated (different segments or even repeated chorusses of the same song). Any statistical analysis of the results (e.g. pairwise significance tests) that relies on independence between files is consequently invalid.
* New dataset: "Billboard2012Key" is the subset of the Billboard2012 chord dataset for which it was possible to derive the key automatically from the chord annotations (using the procedure outlined by Korzeniowski & Widmer in their [https://arxiv.org/abs/1706.02921 2017 EUSIPCO paper]). The annotations are [http://www.cp.jku.at/people/korzeniowski/bb.zip freely available]

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| BD1, BD2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/BD1.pdf PDF]
| Gilberto Bernardes, Matthew Davies
|-
| CN1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/CN1.pdf PDF]
| Chris Cannam, Katy Noland
|-
| FK1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/FK1.pdf PDF]
| Filip Korzeniowski
|-
| HS1-HS3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/HS1.pdf PDF]
| Hendrik Schreiber
|-
| PRGR5
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/PRGR5.pdf PDF]
| Adam Pluta, Marcin Gawrysz
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

Note: until the table display problems are resolved, you can download the results from my [http://eecs.qmul.ac.uk/~johan/akd17results.tar.gz personal website]. --Johan

=====MIREX2005Key=====
<csv>2017/akd/2017-MIREX2005Key.csv</csv>
=====GiantStepsKey=====
<csv>2017/akd/2017-GiantStepsKey.csv</csv>
=====PresegmentedKeyIsophonics=====
<csv>2017/akd/2017-PresegmentedKeyIsophonics.csv</csv>
=====PresegmentedKeyRobbieWilliams=====
<csv>2017/akd/2017-PresegmentedKeyRobbieWilliams.csv</csv>
=====Billboard2012Key=====
<csv>2017/akd/2017-Billboard2012Key.csv</csv>

The submission PRGR5 is currently not able to complete the task without crashing, but hopefully this can still be remedied. These tables will be updated as soon as this is the case.

==Note==
This page will be further updated with more detailed info and extended results (extra statistics, per-file results, confusion matrices) once I get back to a country where the wifi is better and Google's services aren't blocked (which includes the captcha's for this bloody wiki). That will be around November 10th. --Johan

2017:Audio Key Detection Results

2017-11-21T17:27:37Z

JohanPauwels: Temporarily host result csvs at QMUL

==Introduction==

This page contains the results of the 2017 edition of the MIREX automatic key detection estimation task.

==What’s new?==
* The NEMA system was retired this year, since a bug has been found in the calculation of the results. Keys with tonics related by a fifth and the same mode (a.k.a. adjacent keys) are supposed to get a score of 0.5, but only ascending fifths (going from ground-truth to estimation) were counted, not descending ones. It has been brought to my attention that the description of the measure on the wiki has been ambiguous for years, and probably the NEMA implementer got confused by this. However, the intention has always been to count ascending and descending fifth (or fourth) relationships between the tonics (in my humble opinion).
* New datasets: "PresegmentedKeyIsophonics" and "PresegmentedKeyRobbieWilliams" use the local key annotations for the [http://isophonics.net/content/reference-annotations Isophonics set] and the [http://ispg.deib.polimi.it/mir-software.html Robbie Williams set], but have been split into separate files according to the local key annotations. The segments annotated with major and minor modes have been retained and were presented to the submissions. Therefore their results are slightly optimistic in the sense that the segments are guaranteed to contain just a single key, which is not the case for real-world songs. Keep also in mind that some files are strongly correlated (different segments or even repeated chorusses of the same song). Any statistical analysis of the results (e.g. pairwise significance tests) that relies on independence between files is consequently invalid.
* New dataset: "Billboard2012Key" is the subset of the Billboard2012 chord dataset for which it was possible to derive the key automatically from the chord annotations (using the procedure outlined by Korzeniowski & Widmer in their [https://arxiv.org/abs/1706.02921 2017 EUSIPCO paper]). The annotations are [http://www.cp.jku.at/people/korzeniowski/bb.zip freely available]

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| BD1, BD2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/BD1.pdf PDF]
| Gilberto Bernardes, Matthew Davies
|-
| CN1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/CN1.pdf PDF]
| Chris Cannam, Katy Noland
|-
| FK1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/FK1.pdf PDF]
| Filip Korzeniowski
|-
| HS1-HS3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/HS1.pdf PDF]
| Hendrik Schreiber
|-
| PRGR5
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/PRGR5.pdf PDF]
| Adam Pluta, Marcin Gawrysz
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

Note: until the table display problems are resolved, you can download the results from my [http://eecs.qmul.ac.uk/~johan/akd2017results.tar.gz personal website]. --Johan

=====MIREX2005Key=====
<csv>2017/akd/2017-MIREX2005Key.csv</csv>
=====GiantStepsKey=====
<csv>2017/akd/2017-GiantStepsKey.csv</csv>
=====PresegmentedKeyIsophonics=====
<csv>2017/akd/2017-PresegmentedKeyIsophonics.csv</csv>
=====PresegmentedKeyRobbieWilliams=====
<csv>2017/akd/2017-PresegmentedKeyRobbieWilliams.csv</csv>
=====Billboard2012Key=====
<csv>2017/akd/2017-Billboard2012Key.csv</csv>

The submission PRGR5 is currently not able to complete the task without crashing, but hopefully this can still be remedied. These tables will be updated as soon as this is the case.

==Note==
This page will be further updated with more detailed info and extended results (extra statistics, per-file results, confusion matrices) once I get back to a country where the wifi is better and Google's services aren't blocked (which includes the captcha's for this bloody wiki). That will be around November 10th. --Johan

2017:Audio Key Detection Results

2017-11-02T03:09:23Z

JohanPauwels: First addition of AKD results

==Introduction==

This page contains the results of the 2017 edition of the MIREX automatic key detection estimation task.

==What’s new?==
* The NEMA system was retired this year, since a bug has been found in the calculation of the results. Keys with tonics related by a fifth and the same mode (a.k.a. adjacent keys) are supposed to get a score of 0.5, but only ascending fifths (going from ground-truth to estimation) were counted, not descending ones. It has been brought to my attention that the description of the measure on the wiki has been ambiguous for years, and probably the NEMA implementer got confused by this. However, the intention has always been to count ascending and descending fifth (or fourth) relationships between the tonics (in my humble opinion).
* New datasets: "PresegmentedKeyIsophonics" and "PresegmentedKeyRobbieWilliams" use the local key annotations for the [http://isophonics.net/content/reference-annotations Isophonics set] and the [http://ispg.deib.polimi.it/mir-software.html Robbie Williams set], but have been split into separate files according to the local key annotations. The segments annotated with major and minor modes have been retained and were presented to the submissions. Therefore their results are slightly optimistic in the sense that the segments are guaranteed to contain just a single key, which is not the case for real-world songs. Keep also in mind that some files are strongly correlated (different segments or even repeated chorusses of the same song). Any statistical analysis of the results (e.g. pairwise significance tests) that relies on independence between files is consequently invalid.
* New dataset: "Billboard2012Key" is the subset of the Billboard2012 chord dataset for which it was possible to derive the key automatically from the chord annotations (using the procedure outlined by Korzeniowski & Widmer in their [https://arxiv.org/abs/1706.02921 2017 EUSIPCO paper]). The annotations are [http://www.cp.jku.at/people/korzeniowski/bb.zip freely available]

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| BD1, BD2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/BD1.pdf PDF]
| Gilberto Bernardes, Matthew Davies
|-
| CN1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/CN1.pdf PDF]
| Chris Cannam, Katy Noland
|-
| FK1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/FK1.pdf PDF]
| Filip Korzeniowski
|-
| HS1-HS3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/HS1.pdf PDF]
| Hendrik Schreiber
|-
| PRGR5
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/PRGR5.pdf PDF]
| Adam Pluta, Marcin Gawrysz
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====MIREX2005Key=====
<csv>2017/akd/2017-MIREX2005Key.csv</csv>
=====GiantStepsKey=====
<csv>2017/akd/2017-GiantStepsKey.csv</csv>
=====PresegmentedKeyIsophonics=====
<csv>2017/akd/2017-PresegmentedKeyIsophonics.csv</csv>
=====PresegmentedKeyRobbieWilliams=====
<csv>2017/akd/2017-PresegmentedKeyRobbieWilliams.csv</csv>
=====Billboard2012Key=====
<csv>2017/akd/2017-Billboard2012Key.csv</csv>

The submission PRGR5 is currently not able to complete the task without crashing, but hopefully this can still be remedied. These tables will be updated as soon as this is the case.

==Note==
This page will be further updated with more detailed info and extended results (extra statistics, per-file results, confusion matrices) once I get back to a country where the wifi is better and Google's services aren't blocked (which includes the captcha's for this bloody wiki). That will be around November 10th. --Johan

2017:MIREX2017 Results

2017-11-02T03:08:24Z

JohanPauwels: Add AKD results

==Overall Results Poster==
Coming soon

==Results by Task (More results are coming) ==
* [[2017:Audio Fingerprinting Results|Audio Fingerprinting Results]]
* [[2017:Set List Identification Results | Set List Identification Results]]

*Train-Test Task Set
** [https://www.music-ir.org/nema_out/mirex2017/results/act/composer_report/ Audio Classical Composer Identification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/latin_report/ Audio Latin Genre Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/mood_report/index.html Audio Music Mood Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/kgenrek_report/ Audio KPOP Genre (Annotated by Korean Annotators) Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/kgenrea_report/ Audio KPOP Genre (Annotated by American Annotators) Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/kmoodk_report/ Audio KPOP Mood (Annotated by Korean Annotators) Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/kmooda_report/ Audio KPOP Mood (Annotated by American Annotators) Classification Results ]  
* [https://nema.lis.illinois.edu/nema_out/mirex2017/results/aod/ Audio Onset Detection Results]  
* [https://nema.lis.illinois.edu/nema_out/mirex2017/results/ate/ Audio Tempo Estimation Results]  

* [[2017:Automatic Lyrics-to-Audio Alignment Results | Automatic Lyrics-to-Audio Alignment Results]]
* [[2017:Discovery of Repeated Themes & Sections Results|Discovery of Repeated Themes & Sections Results]]

* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/struct/mrx09/ MIREX09 dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/struct/salami/ SALAMI dataset] (partial)  

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/abt/smc/ SMC Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/abt/maz/ MAZ Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/abt/mck/ MCK Dataset]  

* [[2017:Audio_Chord_Estimation_Results | Audio Chord Estimation]]
** [[2017:Audio_Chord_Estimation_Results#Isophonics2009 | Isophonics2009 Dataset]]  
** [[2017:Audio_Chord_Estimation_Results#Billboard2012 | Billboard2012 Dataset]]  
** [[2017:Audio_Chord_Estimation_Results#Billboard_013 | Billboard2013 Dataset]]  
** [[2017:Audio_Chord_Estimation_Results#JayChou29 | JayChou29 Dataset]]  
** [[2017:Audio_Chord_Estimation_Results#RobbieWilliams | RobbieWilliams Dataset]]  
** [[2017:Audio_Chord_Estimation_Results#RWC-Popular | RWC-Popular Dataset]]  
** [[2017:Audio_Chord_Estimation_Results#USPOP2002Chords | USPOP2002Chords Dataset]]  

* Multiple Fundamental Frequency Estimation & Tracking Results
** [[2017:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_%2D_MIREX_Dataset | MIREX Dataset]]  
** [[2017:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_%2D_Su_Dataset | Su Dataset]]  

* [[2017:Drum_Transcription_Results | Drum Transcription]]  
* [https://www.music-ir.org/mirex/wiki/2017:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results Real-time Audio to Score Alignment (a.k.a. Score Following) Results]  
* [[2017:Audio_Key_Detection_Results | Audio Key Detection]]  

2017:Audio Chord Estimation Results

2017-10-16T17:33:44Z

JohanPauwels: Creation of 2017 ACE results

==Introduction==

This page contains the results of the 2017 edition of the MIREX automatic chord estimation tasks. This edition was the fifth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last four years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* This year the algorithms have additionally been evaluated on the "RWC-Popular" and "USPOP2002Chords" dataset annotated at the [http://steinhardt.nyu.edu/marl/ Music and Audio Research Lab] of NYU, whose annotations are [https://github.com/tmc323/Chord-Annotations publicly available]. The [https://staff.aist.go.jp/m.goto/RWC-MDB/rwc-mdb-p.html RWC-Popular dataset] contains 100 pop songs recorded specifically for music information retrieval research. The USPOP2002Chords set is the 195 file subset of the [https://labrosa.ee.columbia.edu/projects/musicsim/uspop2002.html USPOP2002 dataset] that have been annotated with chord sequences.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output will be provided later  More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/CM2.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JLW1, JLW2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/JLW1.pdf PDF]
| Junyan Jiang, Wei Li, Yiming Wu
|-
| KBK1, KBK2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/KBK1.pdf PDF]
| Filip Korzeniowski, Sebastian Böck, Florian Krebs
|-
| WL1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/WL1.pdf PDF]
| Yiming Wu, Wei Li
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====Isophonics2009=====
<csv>2017/ace/2017-Isophonics2009.csv</csv>
=====Billboard2012=====
<csv>2017/ace/2017-Billboard2012.csv</csv>
=====Billboard2013=====
<csv>2017/ace/2017-Billboard2013.csv</csv>
=====JayChou29=====
<csv>2017/ace/2017-JayChou29.csv</csv>
=====RobbieWilliams=====
<csv>2017/ace/2017-RobbieWilliams.csv</csv>
=====RWC-Popular=====
<csv>2017/ace/2017-RWC-Popular.csv</csv>
=====USPOP2002Chords=====
<csv>2017/ace/2017-USPOP2002Chords.csv</csv>



===Detailed Results===

Coming later


===Algorithmic Output===

Coming later

2017:MIREX2017 Results

2017-10-12T11:43:09Z

JohanPauwels: /* Results by Task (More results are coming) */

==Overall Results Poster==
Coming soon

==Results by Task (More results are coming) ==
* [[2017:Audio Fingerprinting Results|Audio Fingerprinting Results]]
* [[2017:Set List Identification Results | Set List Identification Results]]

*Train-Test Task Set
** [https://www.music-ir.org/nema_out/mirex2017/results/act/composer_report/ Audio Classical Composer Identification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/latin_report/ Audio Latin Genre Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/mood_report/index.html Audio Music Mood Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/kgenrek_report/ Audio KPOP Genre (Annotated by Korean Annotators) Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/kgenrea_report/ Audio KPOP Genre (Annotated by American Annotators) Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/kmoodk_report/ Audio KPOP Mood (Annotated by Korean Annotators) Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2017/results/act/kmooda_report/ Audio KPOP Mood (Annotated by American Annotators) Classification Results ]  
* [https://nema.lis.illinois.edu/nema_out/mirex2017/results/aod/ Audio Onset Detection Results]  
* [https://nema.lis.illinois.edu/nema_out/mirex2017/results/ate/ Audio Tempo Estimation Results]  

* [[2017:Discovery of Repeated Themes & Sections Results|Discovery of Repeated Themes & Sections Results]]

* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/struct/mrx09/ MIREX09 dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/struct/salami/ SALAMI dataset] (partial)  

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/abt/smc/ SMC Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/abt/maz/ MAZ Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2017/results/abt/mck/ MCK Dataset]  

* [[2017:Audio_Chord_Estimation_Results | Audio Chord Estimation]]
** [[2017:Audio_Chord_Estimation_Results#Isophonics2009 | Isophonics2009 Dataset]]  
** [[2017:Audio_Chord_Estimation_Results#Billboard2012 | Billboard2012 Dataset]]  
** [[2017:Audio_Chord_Estimation_Results#Billboard_013 | Billboard2013 Dataset]]  
** [[2017:Audio_Chord_Estimation_Results#JayChou29 | JayChou29 Dataset]]  
** [[2017:Audio_Chord_Estimation_Results#RobbieWilliams | RobbieWilliams Dataset]]  
** [[2017:Audio_Chord_Estimation_Results#RWC-Popular | RWC-Popular Dataset]]  
** [[2017:Audio_Chord_Estimation_Results#USPOP2002Chords | USPOP2002Chords Dataset]]  

2013:Audio Chord Estimation Results Billboard 2013

2016-08-31T10:37:09Z

JohanPauwels: Add significant figures

==Introduction==

This year, we have started a new evaluation battery for audio chord estimation. This page contains the results of these new evaluations for a special subset of the ''Billboard'' dataset from McGill University that has never been made available to the public. Further subsets have been withheld to support the ACE task through MIREX 2015.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CB3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB3.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CB4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB4.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CF2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CF2.pdf PDF]
| Chris Cannam, Matthias Mauch, Matthew E. P. Davies, Simon Dixon, Christian Landone, Katy Noland, Mark Levy, Massimiliano Zanoni, Dan Stowell & Luís A. Figueira
|-
| KO1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO1.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| KO2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO2.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| NG1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG1.pdf PDF]
| Nikolay Glazyrin
|-
| NG2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG2.pdf PDF]
| Nikolay Glazyrin
|-
| NMSD1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD1.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| NMSD2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD2.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| PP3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP3.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| PP4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP4.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| SB8
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/SB8.pdf PDF]
| Nikolaas Steenbergen & John Ashley Burgoyne
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2013/ace/2013-Billboard2013.csv</csv>

===Comparative Statistics===

* ''coming soon...''

===Complete Results===

More detailed about the performance of the algorithms, including per-song performance and the breakdown of the WCSR calculations, is available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2013.zip BillboardTest2013.zip]

===Algorithmic Output===

The recognition output and the ground-truth files are available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2013Output.zip BillboardTest2013Output.zip]

We hope to generate a graphical comparison of all algorithms against the ground truth early in 2014.

2013:Audio Chord Estimation Results Billboard 2012

2016-08-31T10:36:27Z

JohanPauwels: Add significant figures

==Introduction==

This year, we have started a new evaluation battery for audio chord estimation. This page contains the results of these new evaluations for an abridged version of the ''Billboard'' dataset from McGill University, including a representative sample of American popular music from the 1950s through the 1990s, as used for MIREX 2012.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CB3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB3.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CB4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB4.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CF2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CF2.pdf PDF]
| Chris Cannam, Matthias Mauch, Matthew E. P. Davies, Simon Dixon, Christian Landone, Katy Noland, Mark Levy, Massimiliano Zanoni, Dan Stowell & Luís A. Figueira
|-
| KO1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO1.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| KO2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO2.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| NG1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG1.pdf PDF]
| Nikolay Glazyrin
|-
| NG2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG2.pdf PDF]
| Nikolay Glazyrin
|-
| NMSD1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD1.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| NMSD2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD2.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| PP3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP3.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| PP4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP4.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| SB8
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/SB8.pdf PDF]
| Nikolaas Steenbergen & John Ashley Burgoyne
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2013/ace/2013-Billboard2012.csv</csv>

===Comparative Statistics===

* ''coming soon...''

===Complete Results===

More detailed about the performance of the algorithms, including per-song performance and the breakdown of the WCSR calculations, is available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2012.zip BillboardTest2012.zip]

===Algorithmic Output===

The recognition output and the ground-truth files are available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2012Output.zip BillboardTest2012Output.zip]

We hope to generate a graphical comparison of all algorithms against the ground truth early in 2014.

2013:Audio Chord Estimation Results MIREX 2009

2016-08-31T10:35:33Z

JohanPauwels: Add significant figures

==Introduction==

This year, we have started a new evaluation battery for audio chord estimation. This page contains the results of these new evaluations for the Isophonics dataset, a.k.a. the MIREX 2009 dataset. It comprises the collected Beatles, Queen, and Zweieck datasets from Queen Mary, University of London, and has been used for audio chord estimation in MIREX for many years.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CB3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB3.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CB4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB4.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CF2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CF2.pdf PDF]
| Chris Cannam, Matthias Mauch, Matthew E. P. Davies, Simon Dixon, Christian Landone, Katy Noland, Mark Levy, Massimiliano Zanoni, Dan Stowell & Luís A. Figueira
|-
| KO1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO1.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| KO2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO2.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| NG1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG1.pdf PDF]
| Nikolay Glazyrin
|-
| NG2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG2.pdf PDF]
| Nikolay Glazyrin
|-
| NMSD1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD1.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| NMSD2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD2.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| PP3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP3.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| PP4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP4.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| SB8
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/SB8.pdf PDF]
| Nikolaas Steenbergen & John Ashley Burgoyne
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2013/ace/2013-Isophonics2009.csv</csv>

===Comparative Statistics===

* ''coming soon...''

===Complete Results===

More detailed about the performance of the algorithms, including per-song performance and the breakdown of the WCSR calculations, is available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/MirexChord2009.zip MirexChord2009.zip]

===Algorithmic Output===

The recognition output and the ground-truth files are available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/MirexChord2009Output.zip MirexChord2009Output.zip]

We hope to generate a graphical comparison of all algorithms against the ground truth early in 2014.

2016:Audio Chord Estimation Results

2016-08-30T12:29:32Z

JohanPauwels: /* Detailed Results */

==Introduction==

This page contains the results of the 2016 edition of the MIREX automatic chord estimation tasks. This edition was the fourth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last three years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* This year the algorithms have been evaluated on the "Robbie Williams" dataset annotated at the [http://ispg.deib.polimi.it/ Image and Sound Processing Group] of Politecnico di Milano, which is [http://ispg.deib.polimi.it/mir-software.html publicly available]. A detailed description of this set can be found in [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=6623838 DiGiorgi et al. (2013)].

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM1 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/CM1.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| DK1-DK4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/DK1.pdf PDF]
| Junqi Deng, Yu-Kwong Kwok
|-
| FK2, FK4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/FK2.pdf PDF]
| Filip Korzeniowski
|-
| KO1 (shineChords)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/KO1.pdf PDF]
| Maksim Khadkevich, Maurizio Omologo
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). 

=====Isophonics 2009=====
<csv>2016/ace/2016-Isophonics2009.csv</csv>
=====Billboard 2012=====
<csv>2016/ace/2016-Billboard2012.csv</csv>
=====Billboard 2013=====
<csv>2016/ace/2016-Billboard2013.csv</csv>
=====JayChou 2015=====
<csv>2016/ace/2016-JayChou2015.csv</csv>
=====RobbieWilliams 2016=====
<csv>2016/ace/2016-RobbieWilliams2016.csv</csv>

===Comparative Statistics===

In progress

===Detailed Results===

More details about the performance of the algorithms, including per-song performance, confusion matrices and supplementary statistics, are available in this [https://music-ir.org/mirex/results/2016/ace/detailled-results-2016.zip zip-file].

===Algorithmic Output===

The raw output of the algorithms are available on [https://github.com/ismir-mirex/ace-output/tree/master/2016 GitHub]. They can be used to experiment with alternative evaluation measures and statistics.

2016:Audio Chord Estimation Results

2016-08-30T12:24:19Z

JohanPauwels: Add algorithmic output link

==Introduction==

This page contains the results of the 2016 edition of the MIREX automatic chord estimation tasks. This edition was the fourth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last three years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* This year the algorithms have been evaluated on the "Robbie Williams" dataset annotated at the [http://ispg.deib.polimi.it/ Image and Sound Processing Group] of Politecnico di Milano, which is [http://ispg.deib.polimi.it/mir-software.html publicly available]. A detailed description of this set can be found in [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=6623838 DiGiorgi et al. (2013)].

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM1 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/CM1.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| DK1-DK4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/DK1.pdf PDF]
| Junqi Deng, Yu-Kwong Kwok
|-
| FK2, FK4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/FK2.pdf PDF]
| Filip Korzeniowski
|-
| KO1 (shineChords)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/KO1.pdf PDF]
| Maksim Khadkevich, Maurizio Omologo
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). 

=====Isophonics 2009=====
<csv>2016/ace/2016-Isophonics2009.csv</csv>
=====Billboard 2012=====
<csv>2016/ace/2016-Billboard2012.csv</csv>
=====Billboard 2013=====
<csv>2016/ace/2016-Billboard2013.csv</csv>
=====JayChou 2015=====
<csv>2016/ace/2016-JayChou2015.csv</csv>
=====RobbieWilliams 2016=====
<csv>2016/ace/2016-RobbieWilliams2016.csv</csv>

===Comparative Statistics===

In progress

===Detailed Results===

In progress

===Algorithmic Output===

The raw output of the algorithms are available on [https://github.com/ismir-mirex/ace-output/tree/master/2016 GitHub]. They can be used to experiment with alternative evaluation measures and statistics.

2016:Task Captains

2016-08-10T20:12:43Z

JohanPauwels: Update email address

Like ISMIR 2015, we are prepared to improve the distribution of tasks for the upcoming MIREX 2016. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead one or more tasks, please add your name in the "Captains" column.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

{| class="wikitable" style="margin-left: 20px"
!ID !! Task !! Captain(s)
|-
|abt
|[[2016:Audio Beat Tracking]]
|Sebastian Böck (sebastian.boeck@jku.at), Florian Krebs (florian.krebs@jku.at)
|-
|ace
|[[2016:Audio Chord Estimation]]
|Johan Pauwels (j.pauwels@qmul.ac.uk)
|-
|act
|[[2016:Audio Classification (Train/Test) Tasks]]
|IMIRSEL (mirproject@lists.lis.illinois.edu)
|-
|acs
|[[2016:Audio Cover Song Identification]]
|Chris Tralie (chris.tralie@gmail.com)
|-
|ade
|[[2016:Audio Downbeat Estimation]]
|Florian Krebs (florian.krebs@jku.at), Sebastian Böck (sebastian.boeck@jku.at)
|-
|akd
|[[2016:Audio Key Detection]]
|Johan Pauwels (j.pauwels@qmul.ac.uk)
|-
|ame
|[[2016:Audio Melody Extraction]]
|KETI (dalwon@keti.re.kr)
|-
|ams
|[[2016:Audio Music Similarity and Retrieval]]
|IMIRSEL (mirproject@lists.lis.illinois.edu)
|-
|aod
|[[2016:Audio Onset Detection]]
|Sebastian Böck (sebastian.boeck@jku.at)
|-
|ate
|[[2016:Audio Tempo Estimation]]
|Aggelos Gkiokas (agkiokas@ilsp.gr)
|-
|mf0
|[[2016:Multiple Fundamental Frequency Estimation & Tracking]]
|Yun Hao (yunhao2@illinois.edu)
|-
|qbsh
|[[2016:Query by Singing/Humming]]
|KETI (dalwon@keti.re.kr)
|-
|scofo
|[[2016:Real-time Audio to Score Alignment (a.k.a Score Following)]]
|Julio Carabias (julio.carabias@upf.edu)
|-
|struct
|[[2016:Structural Segmentation]]
|Piotr Organisciak (organis2@illinois.edu)
|-
|drts
|[[2016:Discovery of Repeated Themes & Sections]]
|Tom Collins (tom.collins@jku.at)
|-
|sli
|[[2016:Set List Identification ]]
|Ming-Chi Yen (ymchiqq@gmail.com)
|-
|aod
|[[2016:Audio Offset Detection ]]
|David Heise (HeiseD@lincolnu.edu)
|-
|afp
|[[2016:Audio_Fingerprinting]]
|Chung-Che Wang (geniusturtle@mirlab.org)
|-
|svs
|[[2016:Singing Voice Separation]]
|Tak-Shing Chan (takshingchan@gmail.com)
|}

2016:Audio Chord Estimation Results

2016-08-06T21:00:58Z

JohanPauwels: typo fix

==Introduction==

This page contains the results of the 2016 edition of the MIREX automatic chord estimation tasks. This edition was the fourth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last three years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* This year the algorithms have been evaluated on the "Robbie Williams" dataset annotated at the [http://ispg.deib.polimi.it/ Image and Sound Processing Group] of Politecnico di Milano, which is [http://ispg.deib.polimi.it/mir-software.html publicly available]. A detailed description of this set can be found in [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=6623838 DiGiorgi et al. (2013)].

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM1 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/CM1.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| DK1-DK4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/DK1.pdf PDF]
| Junqi Deng, Yu-Kwong Kwok
|-
| FK2, FK4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/FK2.pdf PDF]
| Filip Korzeniowski
|-
| KO1 (shineChords)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/KO1.pdf PDF]
| Maksim Khadkevich, Maurizio Omologo
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). 

=====Isophonics 2009=====
<csv>2016/ace/2016-Isophonics2009.csv</csv>
=====Billboard 2012=====
<csv>2016/ace/2016-Billboard2012.csv</csv>
=====Billboard 2013=====
<csv>2016/ace/2016-Billboard2013.csv</csv>
=====JayChou 2015=====
<csv>2016/ace/2016-JayChou2015.csv</csv>
=====RobbieWilliams 2016=====
<csv>2016/ace/2016-RobbieWilliams2016.csv</csv>

===Comparative Statistics===

In progress

===Detailed Results===

In progress

===Algorithmic Output===

In progress

2016:Audio Chord Estimation Results

2016-07-30T16:42:20Z

JohanPauwels: Correct external link formatting

==Introduction==

This page contains the results of the 2016 edition of the MIREX automatic chord estimation tasks. This edition was the fourth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last three years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* This year the algorithms have been evaluated on the "Robbie Williams" dataset annotated at the [http://ispg.deib.polimi.it/ Image and Sound Processing Group] of Politecnico di Milano, which is [http://ispg.deib.polimi.it/mir-software.html publicly available]. A detailed description of this set can be found in [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=6623838 DiGiorgi et al. (2013)].

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM1 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/CM1.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| DK1-DK4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/DK1.pdf PDF]
| Junqi Deng, Yu-Kwong Kwok
|-
| FK2, FK4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/FK2.pdf PDF]
| Filip Korzeniowski
|-
| KO1 (shineChords)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/KO1.pdf PDF]
| Maksim Khadkevich, Maurizio Omologo
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). 

=====Isophonics 2009=====
<csv>2016/ace/2016-Isophonics2009.csv</csv>
=====Billboard 2012=====
<csv>2016/ace/2016-Billboard2012.csv</csv>
=====Bllboard 2013=====
<csv>2016/ace/2016-Billboard2013.csv</csv>
=====JayChou 2015=====
<csv>2016/ace/2016-JayChou2015.csv</csv>
=====RobbieWilliams 2016=====
<csv>2016/ace/2016-RobbieWilliams2016.csv</csv>

===Comparative Statistics===

In progress

===Detailed Results===

In progress

===Algorithmic Output===

In progress

2016:Audio Chord Estimation Results

2016-07-30T16:36:59Z

JohanPauwels: Create ACE results page

==Introduction==

This page contains the results of the 2016 edition of the MIREX automatic chord estimation tasks. This edition was the fourth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last three years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* This year the algorithms have been evaluated on the "Robbie Williams" dataset annotated at the [http://ispg.deib.polimi.it/ | Image and Sound Processing Group] of Politecnico di Milano, which is [http://ispg.deib.polimi.it/mir-software.html | publicly available]. A detailed description of this set can be found in [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=6623838 | DiGiorgi et al. (2013)].

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM1 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/CM1.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| DK1-DK4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/DK1.pdf PDF]
| Junqi Deng, Yu-Kwong Kwok
|-
| FK2, FK4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/FK2.pdf PDF]
| Filip Korzeniowski
|-
| KO1 (shineChords)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/KO1.pdf PDF]
| Maksim Khadkevich, Maurizio Omologo
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). 

=====Isophonics 2009=====
<csv>2016/ace/2016-Isophonics2009.csv</csv>
=====Billboard 2012=====
<csv>2016/ace/2016-Billboard2012.csv</csv>
=====Bllboard 2013=====
<csv>2016/ace/2016-Billboard2013.csv</csv>
=====JayChou 2015=====
<csv>2016/ace/2016-JayChou2015.csv</csv>
=====RobbieWilliams 2016=====
<csv>2016/ace/2016-RobbieWilliams2016.csv</csv>

===Comparative Statistics===

In progress

===Detailed Results===

In progress

===Algorithmic Output===

In progress

2016:MIREX2016 Results

2016-07-30T16:16:39Z

JohanPauwels: Added link to main ACE page

* Audio Key Detection Results
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/akd/mrx_05 MIREX 2005 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/akd/gsteps GiantSteps Dataset]  

* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/adc04/ ADC04 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/mrx05/ MIREX05 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/ind08/ INDIAN08 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/mrx09_0db/ MIREX09 0dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/orchset/ ORCHSET15 Dataset]  

* Query-by-Singing/Humming Results
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task1_hidden/ Hidden Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task1_jang/ Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task1_thinkit/ ThinkIt Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task1_ioacas/ IOACAS Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task2_jang/ Subtask2 Dataset]  

* [[2016:Audio Downbeat Estimation Results|Audio Downbeat Estimation Results]]
* [[2016:Audio Fingerprinting Results|Audio Fingerprinting Results]]

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/abt/smc/ SMC Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/abt/maz/ MAZ Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/abt/mck/ MCK Dataset]  

* [https://nema.lis.illinois.edu/nema_out/mirex2016/results/aod/ Audio Onset Detection Results]  
* [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ate/ Audio Tempo Estimation Results]  

* [[2016:Audio_Chord_Estimation_Results | Audio Chord Estimation]]
** [[2016:Audio_Chord_Estimation_Results#Isophonics_2009 | Isophonics 2009 Dataset]]  
** [[2016:Audio_Chord_Estimation_Results#Billboard_2012 | Billboard 2012 Dataset]]  
** [[2016:Audio_Chord_Estimation_Results#Billboard_2013 | Billboard 2013 Dataset]]  
** [[2016:Audio_Chord_Estimation_Results#JayChou_2015 | JayChou 2015 Dataset]]  
** [[2016:Audio_Chord_Estimation_Results#RobbieWilliams_2016 | RobbieWilliams 2016 Dataset]]  

2016:MIREX2016 Results

2016-07-30T16:14:47Z

JohanPauwels: Add ACE results

* Audio Key Detection Results
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/akd/mrx_05 MIREX 2005 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/akd/gsteps GiantSteps Dataset]  

* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/adc04/ ADC04 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/mrx05/ MIREX05 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/ind08/ INDIAN08 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/mrx09_0db/ MIREX09 0dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/orchset/ ORCHSET15 Dataset]  

* Query-by-Singing/Humming Results
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task1_hidden/ Hidden Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task1_jang/ Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task1_thinkit/ ThinkIt Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task1_ioacas/ IOACAS Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task2_jang/ Subtask2 Dataset]  

* [[2016:Audio Downbeat Estimation Results|Audio Downbeat Estimation Results]]
* [[2016:Audio Fingerprinting Results|Audio Fingerprinting Results]]

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/abt/smc/ SMC Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/abt/maz/ MAZ Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/abt/mck/ MCK Dataset]  

* [https://nema.lis.illinois.edu/nema_out/mirex2016/results/aod/ Audio Onset Detection Results]  
* [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ate/ Audio Tempo Estimation Results]  

* Audio Chord Estimation
** [[2016:Audio_Chord_Estimation_Results#Isophonics_2009 | Isophonics 2009 Dataset]]  
** [[2016:Audio_Chord_Estimation_Results#Billboard_2012 | Billboard 2012 Dataset]]  
** [[2016:Audio_Chord_Estimation_Results#Billboard_2013 | Billboard 2013 Dataset]]  
** [[2016:Audio_Chord_Estimation_Results#JayChou_2015 | JayChou 2015 Dataset]]  
** [[2016:Audio_Chord_Estimation_Results#RobbieWilliams_2016 | RobbieWilliams 2016 Dataset]]  

2010:MIREX 2010 Poster List

2010-08-10T09:25:53Z

JohanPauwels: /* Add your author names here, once for each poster along with "title of some sort" and (Task(s) covered) */

==MIREX 2010 Poster Session Planning List==
The MIREX 2010 Poster Session will be held Wednesday, 11 August: 16:00 - 17:45. We will be holding the MIREX plenary meeting 13:00-14:00 as a working lunch on the same day.

Our hosts in Utrecht need to know the number of posters so they can set up the room. Please add you name and the task(s) dealt with in your poster.

We had many groups/individuals submit across tasks. You can choose to create one ISMIR poster bringing all your data together or can split up your data across, say, two or three posters. If you have questions, please contact me at jdownie@illinois.edu or the MIREX mailing list about task poster options.

As a reminder, the MIREX posters need to follow the [http://ismir2010.ismir.net/information-for-authors/information-for-presenters/ ISMIR 2010 poster guidelines] (i.e., A0, portrait orientation).

==Add your author names here, once for each poster along with "title of some sort" and (Task(s) covered)==
# IMIRSEL: ''MIREX 2010 Overview, Part I'' (Train Test Tasks)
# IMIRSEL: ''MIREX 2010 Overview, Part II'' (All Other Tasks)
# Andreas Arzt and Gerhard Widmer: "Real-time Music Tracking using Tempo-aware On-line Dynamic Time Warping" (Real-time Audio to Score Alignment (a.k.a Score Following))
# Pasi Saari and Olivier Lartillot: "SubEnsemble - Classification framework based on the Ensemble Approach and Feature Selection" (Train Test Tasks)
# Gabriel Sargent, Frédéric Bimbot and Emmanuel Vincent: "Structural segmentation of songs using multi-criteria generalized likelihood ratio and regularity constraints" (Structural Segmentation Task)
# Emmanouil Benetos and Simon Dixon: "Multiple fundamental frequency estimation using spectral structure and temporal evolution rules" (Multiple Fundamental Frequency Estimation & Tracking Task)
# J. Urbano, J. Lloréns, J. Morato and S. Sánchez-Cuadrado: ''Local Alignment with Geometric Representations'' (Symbolic Melodic Similarity)
# F.J.Rodriguez-Serrano, P.Vera-Candeas, P.Cabanas-Molero, J.J.Carabias-Orti, N.Ruiz-Reyes: ''AM Sinusoidal Modeling for Onset Detection FOR ONSET DETECTION'' (Audio Onset Detection)
# R.Mata-Campos, F.J.Rodriguez-Serrano, P.Vera-Candeas, J.J.Carabias-Orti, F.J.Canadas-Quesada: ''Beat Tracking improved by AM Sinusoidal Modeled Onsets'' (Audio Beat Tracking)
# F.J.Rodriguez-Serrano, P.Vera-Candeas, J.J.Carabias-Orti,P.Cabanas-Molero, N.Ruiz-Reyes: ''Real time audio to score alignment based on NLS multipitch estimation'' (Real-time Audio to Score Alignment (a.k.a Score Following))
# F.J. Cañadas-Quesada, F. Rodríguez-Serrano, P. Vera-Candeas, N. Ruiz-Reyes and J. Carabias-Orti: ''Multiple Fundamental Frequency Estimation & Tracking in Polyphonic Music for MIREX 2010'' (Multiple Fundamental Frequency Estimation & Tracking)
# Zhiyao Duan and Bryan Pardo: "A Real-time Score Follower for MIREX 2010" (Real-time Audio to Score Alignment (a.k.a Score Following))
# Zhiyao Duan, Jinyu Han and Bryan Pardo: "A Multi-pitch Estimation and Tracking System" (Multiple Fundamental Frequency Estimation & Tracking Task)
# I.S.H.Suyoto and A.L.Uitdenbogerd: "Orthogonal Pitch with IOI Symbolic Music Matching" (Symbolic Melodic Similarity)
# J.-C. Wang, H.-Y. Lo, S.-K. Jeng and H.-M. Wang: "IISSLG Team: Audio Train/Test and Tag Classification for MIREX 2010" (Audio Train/Test Classification, Audio Tag Classification)
# E. Di Buccio, N. Montecchio and N. Orio: "Applying Text-Based IR Techniques to Cover Song Identification" (Audio Cover Song Identification)
# F. Eyben, B. Schuller: "MIREX 2010: Music Classification with the Munich openSMILE toolkit." (Audio Train/Test Tasks; Audio Tempo Estimation)
# A. Gkiokas, V. Katsouros, G. Carayannis : "MIREX 2010 : Tempo Induction Using Filterbank Analysis and Tonal Features" (Audio Tempo Estimation)
# Y. Zhu, H. Tan, L. Chaisorn: "Poster #1" on Audio Beat Tracking
# H. Tan, Y. Zhu, L. Chaisorn: "Poster #2" on Audio Onset Detection
# J. Salamon, E. Gómez: ''MIREX 2010: Melody Extraction from Polyphonic Audio Music'' (Audio Melody Extraction)
# Aylon E., Bogdanov D., Herrera P., Laurier C., Serrà J., Wack N.: ''MIREX 2010: Train/Test, Audio Music Similarity and Tempo/Beat estimation'' (Audio Train/Test, Audio Music Similarity, Tempo/Beat estimation)
# P. Hamel, D. Eck : "Learning features from music audio with Deep Belief Networks" (Audio genre and tag classification)
# R. Weiss, J. Bello : "Music Structure Segmentation Using Shift-Invariant Probabilistic Latent Component Analysis" (Audio Structural Segmentation)
# K. Seyerlehner, M. Schedl, T. Pohle, P. Knees, : "Using Block-Level Features for Genre Classification, Tag Classification and Music Similarity Estimation" (Train-Test Task Set, Audio Music Similarity and Retrieval, Audio Tag Classification)
# K. Suzuki, Y. Ueda, S. Raczynski, N. Ono, S. Sagayama : ''Real-time Audio to Score Alignment Using Locally-constrained Dynamic Time Warping of Chromagrams'' (Real-time Audio to Score Alignment (a.k.a Score Following))
# H. Rump, S. Miyabe, E. Tsunoo, N. Ono, S. Sagayama : ''Autoregressive MFCC Models for Genre Classification Improved by Harmonic-Percussion Separation'' (Audio Train/Test)
# Johan Pauwels and Jean-Pierre Martens: "Integrating musicological knowledge into a probabilistic system for chord and key extraction" (Audio Chord Estimation/Audio Key Extraction))

==Below are some examples from MIREX 2009==

# Matt Hoffman: ''Using CBA to Automatically Tag Songs'' (Audio tag classification/retrieval)
# Suman Ravuri, Dan Ellis: ''The Hydra System of Cover Song Classification'' (Cover Song Identification)
# Joan Serra, Massimiliano Zanin, Ralph G Andrzejak: ''Cover song retrieval by recurrence quantification and unsupervised set detection'' (Cover Song Identification)
# MTG Team: "Music Type Groupers (MTG): Generic Music Classification Algorithms" (Audio Genre Classification, Mood Classification, Artist Identification, Classical Composer Identification)
# R. Jang: "Poster #2" (placeholder to get the auto-counter to increment)
# R. Jang: "Poster #3" (placeholder to get the auto-counter to increment)