2015:Audio Chord Estimation Results - Revision history

Johan Pauwels: Added note about CM3

2015-10-22T22:32:49Z

Added note about CM3

Johan Pauwels: Fix latest edit

2015-10-21T18:45:55Z

Fix latest edit

Johan Pauwels: External link syntax cleanup

2015-10-21T18:44:08Z

External link syntax cleanup

Johan Pauwels: Improved spelling consistency

2015-10-21T18:42:06Z

Improved spelling consistency

Johan Pauwels: Fixed internal link

2015-10-21T18:37:27Z

Fixed internal link

Johan Pauwels: Creation of the 2015 ace results page

2015-10-21T18:35:53Z

Creation of the 2015 ace results page

New page

==Introduction==

This page contains the results of the 2015 edition of the MIREX automatic chord estimation tasks. This edition was the third one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last two years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results | 2013 edition]].

==What’s new?==
* A new data set, called "Jay Chou 2015" has been donated by [http://tangkk.net | Junqi Deng] of the [http://www.hku.hk | University of Hong Kong]. It consists of 29 Mandopop songs taken from various albums by [https://en.wikipedia.org/wiki/Jay_Chou | Jay Chou]. Most of the songs are ballads and special attention has been paid to the annotation of extended chords and inversions. Because Junqi was kind enough to provide this set before official publication, the algorithmic output on these files and their ground-truth has been withheld for the time being. Also the file names in the per track results have been anonymized.
* The algorithmic output and per track results on the Isophonics set now display the unmasked song name, such that an evaluation per artist/album can be performed.

==Software==
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].

The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CM3 (<em>Chordino</em>)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CM3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| DK4-DK9
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/DK4.pdf PDF]
| Junqi Deng, Yu-Kwong Kwok
|-
| KO1 (<em>shineChords</em>)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/KO1.pdf PDF]
| Maksim Khadkevich, Maurizio Omologo
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary.

=====Isophonics 2009=====
<csv>2015/ace/Isophonics2009.csv</csv>
=====Billboard 2012=====
<csv>2015/ace/Billboard2012.csv</csv>
=====Billboard 2013=====
<csv>2015/ace/Billboard2013.csv</csv>
=====JayChou 2015=====
<csv>2015/ace/JayChou2015.csv</csv>

===Comparative Statistics===

An analysis of the statistical difference between all submissions can be found on the following pages:

* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Isophonics2009 | Isophonics 2009 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2012 | Billboard 2012 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2013 | Billboard 2013 Dataset]]
* [[2015:Audio_Chord_Estimation_Statistical_Analysis_JayChou2015 | Jay Chou 2015 Dataset]]

===Detailed Results===

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from these archives:

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Results.zip Isophonics2009Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Results.zip Billboard2012Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Results.zip Billboard2013Results.zip]
* [https://music-ir.org/mirex/results/2015/ace/JayChou2015Results.zip JayChou2015Results.zip]

===Algorithmic Output===

The raw output of the algorithms are available in the archives below. This can be used to experiment with alternative evaluation measures and statistics.

* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Output.zip Isophonics2009Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Output.zip Billboard2012Output.zip]
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Output.zip Billboard2013Output.zip]

@@ Line 1: / Line 1: @@
 ==Introduction==
-This page contains the results of the 2015 edition of the MIREX automatic chord estimation tasks. This edition was the third one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last two years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 2013 edition]].
+This page contains the results of the 2015 edition of the MIREX automatic chord estimation tasks. This edition was the third one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last two years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].
 ==What’s new?==
-* A new data set, called "JayChou 2015" has been donated by [http://tangkk.net Junqi Deng] of the [http://www.hku.hk University of Hong Kong]. It consists of 29 Mandopop songs taken from various albums by [https://en.wikipedia.org/wiki/Jay_Chou | Jay Chou]. Most of the songs are ballads and special attention has been paid to the annotation of extended chords and inversions. Because Junqi was kind enough to provide this set before official publication, the algorithmic output on these files and their ground-truth has been withheld for the time being. Also the file names in the per track results have been anonymized.
+* A new data set, called "JayChou 2015" has been donated by [http://tangkk.net Junqi Deng] of the [http://www.hku.hk University of Hong Kong]. It consists of 29 Mandopop songs taken from various albums by [https://en.wikipedia.org/wiki/Jay_Chou Jay Chou]. Most of the songs are ballads and special attention has been paid to the annotation of extended chords and inversions. Because Junqi was kind enough to provide this set before official publication, the algorithmic output on these files and their ground-truth has been withheld for the time being. Also the file names in the per track results have been anonymized.
 * The algorithmic output and per track results on the Isophonics set now display the unmasked song name, such that an evaluation per artist/album can be performed.

@@ Line 1: / Line 1: @@
 ==Introduction==
-This page contains the results of the 2015 edition of the MIREX automatic chord estimation tasks. This edition was the third one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last two years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]].
+This page contains the results of the 2015 edition of the MIREX automatic chord estimation tasks. This edition was the third one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last two years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 2013 edition]].
 ==What’s new?==
-* A new data set, called "JayChou 2015" has been donated by [http://tangkk.net | Junqi Deng] of the [http://www.hku.hk | University of Hong Kong]. It consists of 29 Mandopop songs taken from various albums by [https://en.wikipedia.org/wiki/Jay_Chou | Jay Chou]. Most of the songs are ballads and special attention has been paid to the annotation of extended chords and inversions. Because Junqi was kind enough to provide this set before official publication, the algorithmic output on these files and their ground-truth has been withheld for the time being. Also the file names in the per track results have been anonymized.
+* A new data set, called "JayChou 2015" has been donated by [http://tangkk.net Junqi Deng] of the [http://www.hku.hk University of Hong Kong]. It consists of 29 Mandopop songs taken from various albums by [https://en.wikipedia.org/wiki/Jay_Chou | Jay Chou]. Most of the songs are ballads and special attention has been paid to the annotation of extended chords and inversions. Because Junqi was kind enough to provide this set before official publication, the algorithmic output on these files and their ground-truth has been withheld for the time being. Also the file names in the per track results have been anonymized.
 * The algorithmic output and per track results on the Isophonics set now display the unmasked song name, such that an evaluation per artist/album can be performed.

@@ Line 46: / Line 46: @@
 =====JayChou 2015=====
 <csv>2015/ace/JayChou2015.csv</csv>
 ===Comparative Statistics===

@@ Line 4: / Line 4: @@
 ==What’s new?==
-* A new data set, called "Jay Chou 2015" has been donated by [http://tangkk.net | Junqi Deng] of the [http://www.hku.hk | University of Hong Kong]. It consists of 29 Mandopop songs taken from various albums by [https://en.wikipedia.org/wiki/Jay_Chou | Jay Chou]. Most of the songs are ballads and special attention has been paid to the annotation of extended chords and inversions. Because Junqi was kind enough to provide this set before official publication, the algorithmic output on these files and their ground-truth has been withheld for the time being. Also the file names in the per track results have been anonymized.
+* A new data set, called "JayChou 2015" has been donated by [http://tangkk.net | Junqi Deng] of the [http://www.hku.hk | University of Hong Kong]. It consists of 29 Mandopop songs taken from various albums by [https://en.wikipedia.org/wiki/Jay_Chou | Jay Chou]. Most of the songs are ballads and special attention has been paid to the annotation of extended chords and inversions. Because Junqi was kind enough to provide this set before official publication, the algorithmic output on these files and their ground-truth has been withheld for the time being. Also the file names in the per track results have been anonymized.
 * The algorithmic output and per track results on the Isophonics set now display the unmasked song name, such that an evaluation per artist/album can be performed.
@@ Line 54: / Line 54: @@
 * [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2012 | Billboard 2012 Dataset]]
 * [[2015:Audio_Chord_Estimation_Statistical_Analysis_Billboard2013 | Billboard 2013 Dataset]]
-* [[2015:Audio_Chord_Estimation_Statistical_Analysis_JayChou2015 | Jay Chou 2015 Dataset]]
+* [[2015:Audio_Chord_Estimation_Statistical_Analysis_JayChou2015 | JayChou 2015 Dataset]]
 ===Detailed Results===