Introduction
This page contains the results of the 2018 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the 2013 edition.
What’s new?
- This year the algorithms have additionally been evaluated on the "RWC-Popular" and "USPOP2002Chords" dataset annotated at the Music and Audio Research Lab of NYU, whose annotations are publicly available. The RWC-Popular dataset contains 100 pop songs recorded specifically for music information retrieval research. The USPOP2002Chords set is the 195 file subset of the USPOP2002 dataset that have been annotated with chord sequences.
Software
All software used for the evaluation has been made open-source. The evaluation framework is described by Pauwels and Peeters (2013). The corresponding binaries and code repository can be found on GitHub and the used measures are available as presets. The raw algorithmic output will be provided later More help can be found in the readme.
The statistical comparison between the different submissions is explained in Burgoyne et al. (2014). The software is available at BitBucket. It uses the detailed results provided below as input.
Submissions
Results
Summary
All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).
Isophonics2009
| Algorithm |
MirexRoot |
MirexMajMin |
MirexMajMinBass |
MirexSevenths |
MirexSeventhsBass |
MeanSeg |
UnderSeg |
OverSeg |
| CM1 |
78.66 |
75.51 |
72.58 |
54.78 |
52.36 |
85.87 |
87.22 |
85.98 |
| FK2 |
87.38 |
86.80 |
83.43 |
75.55 |
72.60 |
89.29 |
87.24 |
92.35 |
| JLCX1 |
86.75 |
86.25 |
84.44 |
75.87 |
74.39 |
90.33 |
88.36 |
93.38 |
| JLCX2 |
86.51 |
86.05 |
84.23 |
75.64 |
74.17 |
90.12 |
87.94 |
93.48 |
| SG1 |
82.03 |
78.67 |
76.84 |
69.20 |
67.55 |
82.34 |
89.03 |
77.87 |
download these results as csv
Billboard2012
| Algorithm |
MirexRoot |
MirexMajMin |
MirexMajMinBass |
MirexSevenths |
MirexSeventhsBass |
MeanSeg |
UnderSeg |
OverSeg |
| CM1 |
74.23 |
72.31 |
70.25 |
55.44 |
53.48 |
83.60 |
85.33 |
83.31 |
| FK2 |
86.30 |
86.01 |
83.12 |
61.12 |
58.75 |
88.38 |
86.78 |
90.96 |
| JLCX1 |
83.09 |
82.79 |
81.48 |
70.88 |
69.72 |
87.72 |
86.65 |
89.85 |
| JLCX2 |
83.60 |
83.31 |
82.01 |
71.26 |
70.11 |
88.35 |
86.94 |
90.90 |
| SG1 |
82.11 |
79.28 |
78.04 |
66.57 |
65.53 |
81.60 |
90.21 |
76.22 |
download these results as csv
Billboard2013
| Algorithm |
MirexRoot |
MirexMajMin |
MirexMajMinBass |
MirexSevenths |
MirexSeventhsBass |
MeanSeg |
UnderSeg |
OverSeg |
| CM1 |
71.23 |
67.36 |
65.28 |
49.07 |
47.25 |
81.50 |
83.13 |
82.53 |
| FK2 |
80.60 |
78.37 |
75.85 |
55.81 |
53.57 |
83.59 |
83.01 |
87.54 |
| JLCX1 |
78.76 |
77.30 |
76.14 |
63.69 |
62.71 |
84.86 |
83.12 |
89.21 |
| JLCX2 |
79.38 |
77.92 |
76.74 |
64.22 |
63.20 |
85.55 |
83.53 |
90.28 |
| SG1 |
77.92 |
72.41 |
71.04 |
58.26 |
57.10 |
77.95 |
87.70 |
73.46 |
download these results as csv
JayChou29
| Algorithm |
MirexRoot |
MirexMajMin |
MirexMajMinBass |
MirexSevenths |
MirexSeventhsBass |
MeanSeg |
UnderSeg |
OverSeg |
| CM1 |
72.82 |
72.15 |
65.55 |
54.46 |
49.05 |
86.58 |
86.94 |
86.84 |
| FK2 |
81.42 |
80.48 |
69.66 |
51.77 |
43.18 |
89.08 |
87.67 |
90.87 |
| JLCX1 |
84.41 |
84.12 |
81.42 |
63.14 |
61.08 |
92.59 |
91.82 |
93.46 |
| JLCX2 |
84.77 |
84.49 |
81.75 |
63.38 |
61.31 |
93.16 |
92.26 |
94.16 |
| SG1 |
80.59 |
75.65 |
72.28 |
56.43 |
53.80 |
82.51 |
91.49 |
75.30 |
download these results as csv
RobbieWilliams
| Algorithm |
MirexRoot |
MirexMajMin |
MirexMajMinBass |
MirexSevenths |
MirexSeventhsBass |
MeanSeg |
UnderSeg |
OverSeg |
| CM1 |
81.94 |
78.29 |
76.09 |
57.97 |
55.94 |
87.95 |
89.00 |
87.39 |
| FK2 |
89.38 |
88.12 |
85.06 |
83.38 |
80.68 |
90.96 |
89.52 |
92.84 |
| JLCX1 |
87.14 |
85.33 |
83.31 |
75.91 |
74.10 |
90.95 |
89.30 |
93.23 |
| JLCX2 |
87.25 |
85.45 |
83.40 |
76.01 |
74.17 |
91.03 |
89.04 |
93.73 |
| SG1 |
84.87 |
80.28 |
78.58 |
73.78 |
72.16 |
85.24 |
90.85 |
80.75 |
download these results as csv
RWC-Popular
| Algorithm |
MirexRoot |
MirexMajMin |
MirexMajMinBass |
MirexSevenths |
MirexSeventhsBass |
MeanSeg |
UnderSeg |
OverSeg |
| CM1 |
79.12 |
77.95 |
74.30 |
63.33 |
59.89 |
88.62 |
88.11 |
89.67 |
| FK2 |
87.20 |
86.58 |
80.92 |
58.04 |
52.78 |
89.94 |
87.94 |
92.37 |
| JLCX1 |
87.11 |
86.71 |
84.93 |
73.78 |
72.17 |
91.15 |
88.77 |
93.99 |
| JLCX2 |
86.90 |
86.53 |
84.76 |
73.51 |
71.88 |
91.07 |
88.48 |
94.16 |
| SG1 |
82.22 |
78.06 |
76.06 |
63.65 |
61.86 |
82.62 |
90.66 |
76.44 |
download these results as csv
USPOP2002Chords
| Algorithm |
MirexRoot |
MirexMajMin |
MirexMajMinBass |
MirexSevenths |
MirexSeventhsBass |
MeanSeg |
UnderSeg |
OverSeg |
| CM1 |
78.41 |
76.24 |
72.74 |
59.84 |
56.67 |
85.81 |
86.81 |
86.02 |
| FK2 |
82.34 |
80.72 |
76.04 |
64.50 |
60.30 |
86.59 |
85.49 |
89.17 |
| JLCX1 |
85.66 |
84.89 |
82.95 |
75.08 |
73.35 |
89.57 |
88.22 |
91.88 |
| JLCX2 |
85.34 |
84.67 |
82.72 |
74.86 |
73.12 |
89.35 |
87.72 |
92.00 |
| SG1 |
81.67 |
77.38 |
75.16 |
65.69 |
63.68 |
83.46 |
90.37 |
78.59 |
download these results as csv
Detailed Results
More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from ?.
Algorithmic Output
The raw output of the algorithms are available in ?. They can be used to experiment with alternative evaluation measures and statistics.
