2019:Audio Chord Estimation Results - Revision history

JohanPauwels: Update year count

2020-10-14T10:27:04Z

Update year count

JohanPauwels: Undo overzealous correction

2020-10-14T10:23:20Z

Undo overzealous correction

JohanPauwels: Remove outdated reference

2020-10-14T10:20:48Z

Remove outdated reference

JohanPauwels: Add link to Github repositories

2020-10-14T10:11:45Z

Add link to Github repositories

JohanPauwels: Make CASD results visible

2020-10-14T10:03:00Z

Make CASD results visible

Yun Hao at 18:27, 4 November 2019

2019-11-04T18:27:48Z

Yun Hao: Created page with "==Introduction== This page contains the results of the 2018 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of..."

2019-11-04T17:50:56Z

Created page with "==Introduction== This page contains the results of the 2018 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of..."

New page

==Introduction==

This page contains the results of the 2018 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* All datasets and evaluation procedures are the same as last year's MIREX.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output will be provided later  More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/CM1.pdf PDF]
| [https://code.soundsoftware.ac.uk/users/3 Chris Cannam], [http://c4dm.eecs.qmul.ac.uk/ Matthias Mauch]
|-
| JLCX1, JLCX2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/JLCX1.pdf PDF]
| [https://github.com/instr3/ Junyan Jiang], [https://github.com/RetroCirce Ke Chen], [http://www.cs.fudan.edu.cn/ Wei Li], [http://www.cs.cmu.edu/~gxia Guangyu Xia]
|-
| SG1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/SG1.pdf PDF]
| [https://www.fsit.services Franz Strasser], [http://www.jku.at/ Stefan Gaser]
|-
| FK2
| style="text-align: center;" | [PDF]
| [http://www.cp.jku.at Florian Krebs], [http://www.cp.jku.at Filip Korzeniowski], [http://www.ofai.at Sebastian Böck]
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====Isophonics2009=====
<csv>2019/ace/2019-Isophonics2009.csv</csv>
=====Billboard2012=====
<csv>2019/ace/2019-Billboard2012.csv</csv>
=====Billboard2013=====
<csv>2019/ace/2019-Billboard2013.csv</csv>
=====JayChou29=====
<csv>2019/ace/2019-JayChou29.csv</csv>
=====RobbieWilliams=====
<csv>2019/ace/2019-RobbieWilliams.csv</csv>
=====RWC-Popular=====
<csv>2019/ace/2019-RWC-Popular.csv</csv>
=====USPOP2002Chords=====
<csv>2019/ace/2019-USPOP2002Chords.csv</csv>



===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from ?.

===Algorithmic Output===

The raw output of the algorithms are available in ?. They can be used to experiment with alternative evaluation measures and statistics.

@@ Line 1: / Line 1: @@
 ==Introduction==
-This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].
+This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the seventh one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last six years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].
 ==What’s new?==

@@ Line 1: / Line 1: @@
 ==Introduction==
-This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2013 | 2013 edition]].
+This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].
 ==What’s new?==

@@ Line 68: / Line 68: @@
 ===Detailed Results===
-More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from ?.
+More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from [https://github.com/ismir-mirex/ace-results/tree/master/2019 this repository].
 ===Algorithmic Output===
-The raw output of the algorithms are available in ?. They can be used to experiment with alternative evaluation measures and statistics.
+The raw output of the algorithms are available in [https://github.com/ismir-mirex/ace-output/tree/master/2019 this repository]. They can be used to experiment with alternative evaluation measures and statistics.

@@ Line 48: / Line 48: @@
 =====USPOP2002Chords=====
 <csv>2019/ace/2019-USPOP2002Chords.csv</csv>
 <!--===Comparative Statistics===

@@ Line 1: / Line 1: @@
 ==Introduction==
-This page contains the results of the 2018 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].
+This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].
 ==What’s new?==
@@ Line 17: / Line 17: @@
 ! Abstract
 ! Contributors
 |-
 | CM1
-| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/CM1.pdf PDF]
+| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/CM1.pdf PDF]
-| [https://code.soundsoftware.ac.uk/users/3 Chris Cannam], [http://c4dm.eecs.qmul.ac.uk/ Matthias Mauch]
+| [https://code.soundsoftware.ac.uk/users/3 Chris Cannam], [http://matthiasmauch.net Matthias Mauch]
 |-
 |}