2016:Audio Chord Estimation Results

Introduction

This page contains the results of the 2016 edition of the MIREX automatic chord estimation tasks. This edition was the fourth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last three years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the 2013 edition.

What’s new?

• This year the algorithms have been evaluated on the "Robbie Williams" dataset annotated at the Image and Sound Processing Group of Politecnico di Milano, which is publicly available. A detailed description of this set can be found in DiGiorgi et al. (2013).

Software

All software used for the evaluation has been made open-source. The evaluation framework is described by Pauwels and Peeters (2013). The corresponding binaries and code repository can be found on GitHub and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the readme.

The statistical comparison between the different submissions is explained in Burgoyne et al. (2014). The software is available at BitBucket. It uses the detailed results provided below as input.

Submissions

	Abstract	Contributors
CM1 (Chordino)	PDF	Chris Cannam, Matthias Mauch
DK1-DK4	PDF	Junqi Deng, Yu-Kwong Kwok
FK2, FK4	PDF	Filip Korzeniowski
KO1 (shineChords)	PDF	Maksim Khadkevich, Maurizio Omologo

Results

Summary

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

Isophonics 2009

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CM1	78.56	75.41	72.48	54.67	52.26	85.90	87.17	86.09
DK1	79.21	76.19	74.00	66.02	64.15	85.71	82.62	91.23
DK2	77.84	74.49	71.93	61.61	59.47	85.82	82.72	91.28
DK3	80.03	77.55	74.79	68.40	65.88	85.81	82.50	91.53
DK4	76.05	72.96	71.41	62.77	61.44	78.19	87.97	72.43
FK2	86.09	85.53	82.24	74.42	71.54	87.76	85.79	90.73
FK4	82.28	80.93	78.03	70.91	68.26	85.62	82.40	90.89
KO1	82.93	82.19	79.61	76.04	73.43	87.69	85.66	91.24

download these results as csv

Billboard 2012

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CM1	74.15	72.22	70.21	55.35	53.40	83.64	85.31	83.39
DK1	75.28	73.57	71.87	59.98	58.53	83.35	80.26	88.52
DK2	73.77	71.69	69.86	58.66	57.00	83.57	80.40	88.70
DK3	75.92	74.75	72.69	53.42	51.67	83.39	79.97	88.92
DK4	72.59	70.85	69.78	56.29	55.36	76.13	87.72	70.05
FK2	85.64	85.38	82.55	60.70	58.38	87.62	86.09	90.13
FK4	79.23	78.62	76.20	56.53	54.51	85.09	81.98	89.94
KO1	77.45	75.58	73.51	57.68	55.82	84.16	82.80	87.44

download these results as csv

Billboard 2013

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CM1	71.16	67.28	65.20	48.99	47.17	81.54	83.11	82.63
DK1	72.06	68.69	67.26	54.54	53.29	80.82	77.58	88.06
DK2	70.18	66.54	64.66	52.97	51.41	80.85	77.68	88.02
DK3	72.39	68.53	66.55	48.99	47.28	80.76	77.26	88.30
DK4	69.56	65.83	64.78	51.81	50.93	74.55	86.31	69.18
FK2	80.07	77.89	75.42	55.41	53.22	82.94	82.43	86.80
FK4	74.66	71.85	69.44	51.93	49.80	80.61	77.19	88.70
KO1	75.36	71.39	69.43	53.57	51.78	81.63	79.61	87.75

download these results as csv

JayChou 2015

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CM1	72.75	72.08	65.48	54.39	48.98	86.60	86.89	86.91
DK1	74.70	73.87	70.33	54.98	52.25	86.76	82.78	91.79
DK2	72.19	72.55	69.10	54.09	51.46	87.09	83.35	91.75
DK3	75.01	74.75	63.56	49.27	40.24	86.76	82.54	92.08
DK4	71.51	69.03	65.93	50.07	47.45	78.11	87.87	70.56
FK2	79.51	78.66	68.15	50.69	42.34	86.81	85.43	88.56
FK4	76.13	75.44	64.36	49.69	40.74	84.55	81.22	88.95
KO1	78.73	77.69	66.87	54.16	44.55	88.46	87.12	90.11

download these results as csv

RobbieWilliams 2016

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CM1	81.90	78.25	76.05	57.92	55.90	87.96	88.96	87.45
DK1	81.50	77.77	76.10	68.88	67.34	87.03	83.22	92.11
DK2	79.01	75.97	73.57	65.26	62.98	87.20	83.40	92.23
DK3	81.85	78.56	76.16	74.71	72.55	86.98	82.95	92.34
DK4	78.92	75.15	73.66	66.72	65.34	81.82	88.44	76.88
FK2	88.53	87.23	84.19	82.57	79.88	90.04	88.62	91.88
FK4	83.37	80.96	78.42	77.04	74.76	87.22	84.50	91.02
KO1	83.55	80.33	78.16	73.54	71.39	88.04	85.39	91.68

download these results as csv

Comparative Statistics

In progress

Detailed Results

More details about the performance of the algorithms, including per-song performance, confusion matrices and supplementary statistics, are available in this zip-file.

Algorithmic Output

The raw output of the algorithms are available on GitHub. They can be used to experiment with alternative evaluation measures and statistics.