2018:Audio Key Detection Results

Introduction

This page contains the results of the 2018 edition of the MIREX automatic key detection estimation task.

What’s new?

• The NEMA system was retired last year, since a bug has been found in the calculation of the results. Keys with tonics related by a fifth and the same mode (a.k.a. adjacent keys) are supposed to get a score of 0.5, but only ascending fifths (going from ground-truth to estimation) were counted, not descending ones. It has been brought to my attention that the description of the measure on the wiki has been ambiguous for years, and probably the NEMA implementer got confused by this. However, the intention has always been to count ascending and descending fifth (or fourth) relationships between the tonics (in my humble opinion).
• New datasets: "PresegmentedKeyIsophonics" and "PresegmentedKeyRobbieWilliams" use the local key annotations for the Isophonics set and the Robbie Williams set, but have been split into separate files according to the local key annotations. The segments annotated with major and minor modes have been retained and were presented to the submissions. Therefore their results are slightly optimistic in the sense that the segments are guaranteed to contain just a single key, which is not the case for real-world songs. Keep also in mind that some files are strongly correlated (different segments or even repeated chorusses of the same song). Any statistical analysis of the results (e.g. pairwise significance tests) that relies on independence between files is consequently invalid.
• New dataset: "Billboard2012Key" is the subset of the Billboard2012 chord dataset for which it was possible to derive the key automatically from the chord annotations (using the procedure outlined by Korzeniowski & Widmer in their 2017 EUSIPCO paper). The annotations are freely available

Submissions

	Abstract	Contributors
CG1	PDF	David Castells-Rufas, Adria Galin
GC1	PDF	Adria Galin, David Castells-Rufas
CN1	PDF	Chris Cannam, Katy Noland
NA1	PDF	Nestor Napoles Lopez , Claire Arthur
FK1, FK3	PDF	Filip Korzeniowski
OM1-OM3	PDF	James Owers, Andrew McLeod

Results

Summary

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

MIREX2005Key

Algorithm	Mirex	Correct	Adjacent	Relative	Parallel	Chromatic	Other
CG1	88.78	84.27	6.63	3.19	1.20	0.16	4.55
CN1	88.07	82.75	8.31	2.72	1.76	0.08	4.39
FK1	82.50	72.52	16.77	4.07	1.84	0.24	4.55
GC1	76.58	63.34	21.49	7.11	1.84	0.08	6.15
NA1	84.06	74.44	15.42	5.03	2.00	0.08	3.04
OM1	82.52	73.16	14.78	4.95	2.40	0.08	4.63
OM2	73.43	58.07	25.00	8.07	2.24	0.08	6.55
OM3	49.64	36.98	11.18	13.50	15.10	0.00	23.24

download these results as csv

GiantStepsKey

Algorithm	Mirex	Correct	Adjacent	Relative	Parallel	Chromatic	Other
CG1	28.21	16.72	17.05	5.13	7.12	3.64	50.33
CN1	50.53	39.74	11.92	13.24	4.30	2.65	28.15
FK1	74.62	67.88	6.95	8.11	4.14	2.98	9.93
GC1	30.03	19.04	12.91	8.28	10.26	1.99	47.52
NA1	59.42	48.34	11.59	13.08	6.79	1.82	18.38
OM1	58.11	49.50	11.59	5.96	5.13	2.48	25.33
OM2	49.72	39.07	17.55	3.48	4.14	3.97	31.79
OM3	48.58	40.89	10.76	3.48	6.29	3.81	34.77

download these results as csv

PresegmentedKeyIsophonics

Algorithm	Mirex	Correct	Adjacent	Relative	Parallel	Chromatic	Other
CG1	61.86	50.57	15.40	6.90	7.59	1.15	18.39
CN1	64.99	55.40	13.56	4.60	7.13	3.91	15.40
FK1	82.30	76.32	7.36	5.52	3.22	2.30	5.29
GC1	68.05	56.78	17.70	5.75	3.45	1.84	14.48
NA1	75.72	65.75	12.64	9.89	3.45	2.07	6.21
OM1	62.92	53.10	12.41	5.29	10.11	1.38	17.70
OM2	51.15	34.25	22.99	9.43	12.87	1.61	18.85
OM3	41.13	25.52	15.63	8.05	26.90	0.92	22.99

download these results as csv

PresegmentedKeyRobbieWilliams

Algorithm	Mirex	Correct	Adjacent	Relative	Parallel	Chromatic	Other
CG1	54.66	41.80	20.63	4.23	6.35	1.06	25.93
CN1	62.28	53.97	10.58	7.94	3.17	7.41	16.93
FK1	81.38	72.49	10.05	12.17	1.06	0.00	4.23
GC1	62.70	51.32	16.93	7.94	2.65	0.00	21.16
NA1	77.20	66.14	16.40	8.47	1.59	0.00	7.41
OM1	63.86	53.97	13.76	6.88	4.76	0.00	20.63
OM2	51.32	38.62	17.46	5.82	11.11	0.00	26.98
OM3	57.99	48.15	11.64	5.29	12.17	0.53	22.22

download these results as csv

Billboard2012Key

Algorithm	Mirex	Correct	Adjacent	Relative	Parallel	Chromatic	Other
CG1	63.01	52.74	15.07	2.74	9.59	2.05	17.81
CN1	67.40	58.22	12.33	5.48	6.85	6.85	10.27
FK1	88.22	84.25	2.74	5.48	4.79	0.00	2.74
GC1	69.66	58.22	20.55	2.05	2.74	0.68	15.75
NA1	79.59	70.55	13.70	5.48	2.74	0.68	6.85
OM1	71.58	60.27	13.70	6.16	13.01	0.00	6.85
OM2	55.41	41.10	19.18	3.42	18.49	0.68	17.12
OM3	39.04	25.34	8.90	7.53	34.93	0.00	23.29

download these results as csv