Difference between revisions of "2019:Audio Chord Estimation Results"
From MIREX Wiki
JohanPauwels (talk | contribs) (Update year count) |
|||
| (4 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
==Introduction== | ==Introduction== | ||
| − | This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the | + | This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the seventh one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last six years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the [[2013:Audio_Chord_Estimation_Results_MIREX_2009 | 2013 edition]]. |
==What’s new?== | ==What’s new?== | ||
| Line 7: | Line 7: | ||
==Software== | ==Software== | ||
| − | All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output | + | All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme]. |
The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input. | The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input. | ||
| Line 48: | Line 48: | ||
=====USPOP2002Chords===== | =====USPOP2002Chords===== | ||
<csv>2019/ace/2019-USPOP2002Chords.csv</csv> | <csv>2019/ace/2019-USPOP2002Chords.csv</csv> | ||
| + | =====CASD-Annotator1===== | ||
| + | <csv>2019/ace/2019-CASD-Annotator1.csv</csv> | ||
| + | =====CASD-Annotator2===== | ||
| + | <csv>2019/ace/2019-CASD-Annotator2.csv</csv> | ||
| + | =====CASD-Annotator3===== | ||
| + | <csv>2019/ace/2019-CASD-Annotator3.csv</csv> | ||
| + | =====CASD-Annotator4===== | ||
| + | <csv>2019/ace/2019-CASD-Annotator4.csv</csv> | ||
<!--===Comparative Statistics=== | <!--===Comparative Statistics=== | ||
| Line 60: | Line 68: | ||
===Detailed Results=== | ===Detailed Results=== | ||
| − | More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from | + | More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from [https://github.com/ismir-mirex/ace-results/tree/master/2019 this repository]. |
| + | . | ||
===Algorithmic Output=== | ===Algorithmic Output=== | ||
| − | The raw output of the algorithms are available in | + | The raw output of the algorithms are available in [https://github.com/ismir-mirex/ace-output/tree/master/2019 this repository]. They can be used to experiment with alternative evaluation measures and statistics. |
Latest revision as of 04:27, 14 October 2020
Introduction
This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the seventh one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last six years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the 2013 edition.
What’s new?
- All datasets and evaluation procedures are the same as last year's MIREX.
Software
All software used for the evaluation has been made open-source. The evaluation framework is described by Pauwels and Peeters (2013). The corresponding binaries and code repository can be found on GitHub and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the readme.
The statistical comparison between the different submissions is explained in Burgoyne et al. (2014). The software is available at BitBucket. It uses the detailed results provided below as input.
Submissions
| Abstract | Contributors | |
|---|---|---|
| CLSYJ1 | i Chien, Song Rong Lee, Yeh Ssuhung, Tzu-Chun Yeh, Jyh-Shing Roger Jang | |
| CM1 | Chris Cannam, Matthias Mauch |
Results
Summary
All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).
Isophonics2009
| Algorithm | MirexRoot | MirexMajMin | MirexMajMinBass | MirexSevenths | MirexSeventhsBass | MeanSeg | UnderSeg | OverSeg |
|---|---|---|---|---|---|---|---|---|
| CLSYJ1 | 79.89 | 76.50 | 74.33 | 67.79 | 65.82 | 82.61 | 81.56 | 85.11 |
| CM1 | 78.66 | 75.51 | 72.58 | 54.78 | 52.36 | 85.87 | 87.22 | 85.98 |
Billboard2012
| Algorithm | MirexRoot | MirexMajMin | MirexMajMinBass | MirexSevenths | MirexSeventhsBass | MeanSeg | UnderSeg | OverSeg |
|---|---|---|---|---|---|---|---|---|
| CLSYJ1 | 76.32 | 73.38 | 71.60 | 63.59 | 61.92 | 82.05 | 81.74 | 83.77 |
| CM1 | 74.23 | 72.31 | 70.25 | 55.44 | 53.48 | 83.60 | 85.33 | 83.31 |
Billboard2013
| Algorithm | MirexRoot | MirexMajMin | MirexMajMinBass | MirexSevenths | MirexSeventhsBass | MeanSeg | UnderSeg | OverSeg |
|---|---|---|---|---|---|---|---|---|
| CLSYJ1 | 73.69 | 69.47 | 67.66 | 57.41 | 55.78 | 78.80 | 78.67 | 82.52 |
| CM1 | 71.23 | 67.36 | 65.28 | 49.07 | 47.25 | 81.50 | 83.13 | 82.53 |
JayChou29
| Algorithm | MirexRoot | MirexMajMin | MirexMajMinBass | MirexSevenths | MirexSeventhsBass | MeanSeg | UnderSeg | OverSeg |
|---|---|---|---|---|---|---|---|---|
| CLSYJ1 | 65.44 | 62.49 | 55.27 | 46.83 | 40.60 | 78.95 | 80.78 | 77.55 |
| CM1 | 72.82 | 72.15 | 65.55 | 54.46 | 49.05 | 86.58 | 86.94 | 86.84 |
RobbieWilliams
| Algorithm | MirexRoot | MirexMajMin | MirexMajMinBass | MirexSevenths | MirexSeventhsBass | MeanSeg | UnderSeg | OverSeg |
|---|---|---|---|---|---|---|---|---|
| CLSYJ1 | 79.89 | 75.45 | 73.73 | 68.57 | 66.91 | 83.60 | 83.72 | 84.32 |
| CM1 | 81.94 | 78.29 | 76.09 | 57.97 | 55.94 | 87.95 | 89.00 | 87.39 |
RWC-Popular
| Algorithm | MirexRoot | MirexMajMin | MirexMajMinBass | MirexSevenths | MirexSeventhsBass | MeanSeg | UnderSeg | OverSeg |
|---|---|---|---|---|---|---|---|---|
| CLSYJ1 | 70.33 | 66.77 | 63.42 | 56.09 | 52.97 | 79.26 | 79.17 | 80.14 |
| CM1 | 79.12 | 77.95 | 74.30 | 63.33 | 59.89 | 88.62 | 88.11 | 89.67 |
USPOP2002Chords
| Algorithm | MirexRoot | MirexMajMin | MirexMajMinBass | MirexSevenths | MirexSeventhsBass | MeanSeg | UnderSeg | OverSeg |
|---|---|---|---|---|---|---|---|---|
| CLSYJ1 | 76.23 | 72.30 | 68.75 | 62.27 | 58.92 | 82.13 | 82.12 | 83.24 |
| CM1 | 78.41 | 76.24 | 72.74 | 59.84 | 56.67 | 85.81 | 86.81 | 86.02 |
CASD-Annotator1
| Algorithm | MirexRoot | MirexMajMin | MirexMajMinBass | MirexSevenths | MirexSeventhsBass | MeanSeg | UnderSeg | OverSeg |
|---|---|---|---|---|---|---|---|---|
| CLSYJ1 | 69.95 | 65.12 | 63.49 | 57.03 | 55.56 | 79.68 | 80.43 | 81.67 |
| CM1 | 72.55 | 69.00 | 66.59 | 49.76 | 47.92 | 83.99 | 85.95 | 83.96 |
CASD-Annotator2
| Algorithm | MirexRoot | MirexMajMin | MirexMajMinBass | MirexSevenths | MirexSeventhsBass | MeanSeg | UnderSeg | OverSeg |
|---|---|---|---|---|---|---|---|---|
| CLSYJ1 | 71.17 | 65.38 | 63.93 | 54.83 | 53.41 | 76.74 | 75.15 | 82.82 |
| CM1 | 72.26 | 68.17 | 66.44 | 44.62 | 43.29 | 80.23 | 79.71 | 84.64 |
CASD-Annotator3
| Algorithm | MirexRoot | MirexMajMin | MirexMajMinBass | MirexSevenths | MirexSeventhsBass | MeanSeg | UnderSeg | OverSeg |
|---|---|---|---|---|---|---|---|---|
| CLSYJ1 | 69.53 | 65.07 | 62.72 | 54.46 | 52.30 | 76.78 | 74.19 | 83.03 |
| CM1 | 73.08 | 69.29 | 66.28 | 50.88 | 48.46 | 81.49 | 79.69 | 86.08 |
CASD-Annotator4
| Algorithm | MirexRoot | MirexMajMin | MirexMajMinBass | MirexSevenths | MirexSeventhsBass | MeanSeg | UnderSeg | OverSeg |
|---|---|---|---|---|---|---|---|---|
| CLSYJ1 | 62.26 | 57.15 | 48.90 | 48.74 | 41.38 | 77.19 | 79.02 | 77.69 |
| CM1 | 61.55 | 57.37 | 48.05 | 38.28 | 32.15 | 79.53 | 82.70 | 78.29 |
Detailed Results
More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from this repository. .
Algorithmic Output
The raw output of the algorithms are available in this repository. They can be used to experiment with alternative evaluation measures and statistics.
