2019:Audio Chord Estimation Results

Introduction

This page contains the results of the 2019 edition of the MIREX automatic chord estimation tasks. This edition was the seventh one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last six years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the 2013 edition.

What’s new?

• All datasets and evaluation procedures are the same as last year's MIREX.

Software

All software used for the evaluation has been made open-source. The evaluation framework is described by Pauwels and Peeters (2013). The corresponding binaries and code repository can be found on GitHub and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the readme.

The statistical comparison between the different submissions is explained in Burgoyne et al. (2014). The software is available at BitBucket. It uses the detailed results provided below as input.

Submissions

	Abstract	Contributors
CLSYJ1	PDF	i Chien, Song Rong Lee, Yeh Ssuhung, Tzu-Chun Yeh, Jyh-Shing Roger Jang
CM1	PDF	Chris Cannam, Matthias Mauch

Results

Summary

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

Isophonics2009

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CLSYJ1	79.89	76.50	74.33	67.79	65.82	82.61	81.56	85.11
CM1	78.66	75.51	72.58	54.78	52.36	85.87	87.22	85.98

download these results as csv

Billboard2012

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CLSYJ1	76.32	73.38	71.60	63.59	61.92	82.05	81.74	83.77
CM1	74.23	72.31	70.25	55.44	53.48	83.60	85.33	83.31

download these results as csv

Billboard2013

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CLSYJ1	73.69	69.47	67.66	57.41	55.78	78.80	78.67	82.52
CM1	71.23	67.36	65.28	49.07	47.25	81.50	83.13	82.53

download these results as csv

JayChou29

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CLSYJ1	65.44	62.49	55.27	46.83	40.60	78.95	80.78	77.55
CM1	72.82	72.15	65.55	54.46	49.05	86.58	86.94	86.84

download these results as csv

RobbieWilliams

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CLSYJ1	79.89	75.45	73.73	68.57	66.91	83.60	83.72	84.32
CM1	81.94	78.29	76.09	57.97	55.94	87.95	89.00	87.39

download these results as csv

RWC-Popular

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CLSYJ1	70.33	66.77	63.42	56.09	52.97	79.26	79.17	80.14
CM1	79.12	77.95	74.30	63.33	59.89	88.62	88.11	89.67

download these results as csv

USPOP2002Chords

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CLSYJ1	76.23	72.30	68.75	62.27	58.92	82.13	82.12	83.24
CM1	78.41	76.24	72.74	59.84	56.67	85.81	86.81	86.02

download these results as csv

CASD-Annotator1

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CLSYJ1	69.95	65.12	63.49	57.03	55.56	79.68	80.43	81.67
CM1	72.55	69.00	66.59	49.76	47.92	83.99	85.95	83.96

download these results as csv

CASD-Annotator2

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CLSYJ1	71.17	65.38	63.93	54.83	53.41	76.74	75.15	82.82
CM1	72.26	68.17	66.44	44.62	43.29	80.23	79.71	84.64

download these results as csv

CASD-Annotator3

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CLSYJ1	69.53	65.07	62.72	54.46	52.30	76.78	74.19	83.03
CM1	73.08	69.29	66.28	50.88	48.46	81.49	79.69	86.08

download these results as csv

CASD-Annotator4

Algorithm	MirexRoot	MirexMajMin	MirexMajMinBass	MirexSevenths	MirexSeventhsBass	MeanSeg	UnderSeg	OverSeg
CLSYJ1	62.26	57.15	48.90	48.74	41.38	77.19	79.02	77.69
CM1	61.55	57.37	48.05	38.28	32.15	79.53	82.70	78.29

download these results as csv

Detailed Results

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from this repository. .

Algorithmic Output

The raw output of the algorithms are available in this repository. They can be used to experiment with alternative evaluation measures and statistics.