2017:Audio Chord Estimation Results - Revision history

JohanPauwels: Remove outdated reference

2020-10-14T10:18:43Z

Remove outdated reference

JohanPauwels: Add links to repositories

2017-11-28T17:01:01Z

Add links to repositories

JohanPauwels: Creation of 2017 ACE results

2017-10-16T17:33:44Z

Creation of 2017 ACE results

New page

==Introduction==

This page contains the results of the 2017 edition of the MIREX automatic chord estimation tasks. This edition was the fifth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last four years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* This year the algorithms have additionally been evaluated on the "RWC-Popular" and "USPOP2002Chords" dataset annotated at the [http://steinhardt.nyu.edu/marl/ Music and Audio Research Lab] of NYU, whose annotations are [https://github.com/tmc323/Chord-Annotations publicly available]. The [https://staff.aist.go.jp/m.goto/RWC-MDB/rwc-mdb-p.html RWC-Popular dataset] contains 100 pop songs recorded specifically for music information retrieval research. The USPOP2002Chords set is the 195 file subset of the [https://labrosa.ee.columbia.edu/projects/musicsim/uspop2002.html USPOP2002 dataset] that have been annotated with chord sequences.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output will be provided later  More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/CM2.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JLW1, JLW2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/JLW1.pdf PDF]
| Junyan Jiang, Wei Li, Yiming Wu
|-
| KBK1, KBK2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/KBK1.pdf PDF]
| Filip Korzeniowski, Sebastian Böck, Florian Krebs
|-
| WL1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/WL1.pdf PDF]
| Yiming Wu, Wei Li
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====Isophonics2009=====
<csv>2017/ace/2017-Isophonics2009.csv</csv>
=====Billboard2012=====
<csv>2017/ace/2017-Billboard2012.csv</csv>
=====Billboard2013=====
<csv>2017/ace/2017-Billboard2013.csv</csv>
=====JayChou29=====
<csv>2017/ace/2017-JayChou29.csv</csv>
=====RobbieWilliams=====
<csv>2017/ace/2017-RobbieWilliams.csv</csv>
=====RWC-Popular=====
<csv>2017/ace/2017-RWC-Popular.csv</csv>
=====USPOP2002Chords=====
<csv>2017/ace/2017-USPOP2002Chords.csv</csv>



===Detailed Results===

Coming later


===Algorithmic Output===

Coming later

@@ Line 7: / Line 7: @@
 ==Software==
-All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output will be provided later <!--provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below.--> More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].
+All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].
 The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

@@ Line 67: / Line 67: @@
 ===Detailed Results===
-Coming later
+More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from [https://github.com/ismir-mirex/ace-results/tree/master/2017 this repository].
 ===Algorithmic Output===
-Coming later
+The raw output of the algorithms are available in [https://github.com/ismir-mirex/ace-output/tree/master/2017 this repository]. They can be used to experiment with alternative evaluation measures and statistics.