2016:Audio Chord Estimation Results - Revision history

JohanPauwels: /* Detailed Results */

2016-08-30T12:29:32Z

‎Detailed Results

JohanPauwels: Add algorithmic output link

2016-08-30T12:24:19Z

Add algorithmic output link

JohanPauwels: typo fix

2016-08-06T21:00:58Z

typo fix

JohanPauwels: Correct external link formatting

2016-07-30T16:42:20Z

Correct external link formatting

JohanPauwels: Create ACE results page

2016-07-30T16:36:59Z

Create ACE results page

New page

==Introduction==

This page contains the results of the 2016 edition of the MIREX automatic chord estimation tasks. This edition was the fourth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last three years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].

==What’s new?==
* This year the algorithms have been evaluated on the "Robbie Williams" dataset annotated at the [http://ispg.deib.polimi.it/ | Image and Sound Processing Group] of Politecnico di Milano, which is [http://ispg.deib.polimi.it/mir-software.html | publicly available]. A detailed description of this set can be found in [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=6623838 | DiGiorgi et al. (2013)].

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM1 (<em>Chordino</em>)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/CM1.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| DK1-DK4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/DK1.pdf PDF]
| Junqi Deng, Yu-Kwong Kwok
|-
| FK2, FK4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/FK2.pdf PDF]
| Filip Korzeniowski
|-
| KO1 (<em>shineChords</em>)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/KO1.pdf PDF]
| Maksim Khadkevich, Maurizio Omologo
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). 

=====Isophonics 2009=====
<csv>2016/ace/2016-Isophonics2009.csv</csv>
=====Billboard 2012=====
<csv>2016/ace/2016-Billboard2012.csv</csv>
=====Bllboard 2013=====
<csv>2016/ace/2016-Billboard2013.csv</csv>
=====JayChou 2015=====
<csv>2016/ace/2016-JayChou2015.csv</csv>
=====RobbieWilliams 2016=====
<csv>2016/ace/2016-RobbieWilliams2016.csv</csv>

===Comparative Statistics===

In progress

===Detailed Results===

In progress

===Algorithmic Output===

In progress

← Older revision		Revision as of 12:24, 30 August 2016
Line 72:		Line 72:
	===Algorithmic Output===		===Algorithmic Output===

−	~~In progress <!--~~The raw output of the algorithms are available ~~in the archives below. This can be used to experiment with alternative evaluation measures and statistics.~~	+	The raw output of the algorithms are available on [https://github.com/ismir-mirex/ace-output/tree/master/2016 GitHub]. They can be used to experiment with alternative evaluation measures and statistics.
−
−	* [https://~~music-ir~~.~~org~~/mirex~~/results/2015~~/ace~~/Isophonics2009Output.zip Isophonics2009Output.zip]~~
−	* [https://music-~~ir.org/mirex~~/~~results~~/~~2015~~/~~ace/Billboard2012Output.zip Billboard2012Output.zip~~]
−	* [https://music-ir.~~org/mirex/results/2015/ace/Billboard2013Output.zip Billboard2013Output~~.~~zip]-->~~

@@ Line 63: / Line 63: @@
 ===Detailed Results===
-In progress<!--More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from these archives:
+More details about the performance of the algorithms, including per-song performance, confusion matrices and supplementary statistics, are available in this [https://music-ir.org/mirex/results/2016/ace/detailled-results-2016.zip zip-file].
 ===Algorithmic Output===
 The raw output of the algorithms are available on [https://github.com/ismir-mirex/ace-output/tree/master/2016 GitHub]. They can be used to experiment with alternative evaluation measures and statistics.

@@ Line 45: / Line 45: @@
 =====Billboard 2012=====
 <csv>2016/ace/2016-Billboard2012.csv</csv>
-=====Bllboard 2013=====
+=====Billboard 2013=====
 <csv>2016/ace/2016-Billboard2013.csv</csv>
 =====JayChou 2015=====

@@ Line 4: / Line 4: @@
 ==What’s new?==
-* This year the algorithms have been evaluated on the "Robbie Williams" dataset annotated at the [http://ispg.deib.polimi.it/ | Image and Sound Processing Group] of Politecnico di Milano, which is [http://ispg.deib.polimi.it/mir-software.html | publicly available]. A detailed description of this set can be found in [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=6623838 | DiGiorgi et al. (2013)].
+* This year the algorithms have been evaluated on the "Robbie Williams" dataset annotated at the [http://ispg.deib.polimi.it/ Image and Sound Processing Group] of Politecnico di Milano, which is [http://ispg.deib.polimi.it/mir-software.html publicly available]. A detailed description of this set can be found in [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=6623838 DiGiorgi et al. (2013)].
 ==Software==
-All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].
+All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].
 The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.