Difference between revisions of "2018:Audio Key Detection Results"

From MIREX Wiki
(Created page with "==Introduction== This page contains the results of the 2018 edition of the MIREX automatic key detection estimation task. ==What’s new?== * The NEMA system was retired last y...")
 
(Add note about comparison with last year)
 
Line 4: Line 4:
  
 
==What’s new?==
 
==What’s new?==
−
* The NEMA system was retired last year, since a bug has been found in the calculation of the results. Keys with tonics related by a fifth and the same mode (a.k.a. adjacent keys) are supposed to get a score of 0.5, but only ascending fifths (going from ground-truth to estimation) were counted, not descending ones. It has been brought to my attention that the description of the measure on the wiki has been ambiguous for years, and probably the NEMA implementer got confused by this. However, the intention has always been to count ascending and descending fifth (or fourth) relationships between the tonics (in my humble opinion).
+
* Nothing has changed, which means the results can be directly compared to last year's.
−
* New datasets: "PresegmentedKeyIsophonics" and "PresegmentedKeyRobbieWilliams" use the local key annotations for the [http://isophonics.net/content/reference-annotations Isophonics set] and the [http://ispg.deib.polimi.it/mir-software.html Robbie Williams set], but have been split into separate files according to the local key annotations. The segments annotated with major and minor modes have been retained and were presented to the submissions. Therefore their results are slightly optimistic in the sense that the segments are guaranteed to contain just a single key, which is not the case for real-world songs. Keep also in mind that some files are strongly correlated (different segments or even repeated chorusses of the same song). Any statistical analysis of the results (e.g. pairwise significance tests) that relies on independence between files is consequently invalid.
 
−
* New dataset: "Billboard2012Key" is the subset of the Billboard2012 chord dataset for which it was possible to derive the key automatically from the chord annotations (using the procedure outlined by Korzeniowski & Widmer in their [https://arxiv.org/abs/1706.02921 2017 EUSIPCO paper]). The annotations are [http://www.cp.jku.at/people/korzeniowski/bb.zip freely available]
 
  
 
==Submissions==
 
==Submissions==
Line 42: Line 40:
  
 
==Results==
 
==Results==
−
 
−
===Summary===
 
  
 
All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).
 
All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).
  
−
=====MIREX2005Key=====
+
====MIREX2005Key====
 
<csv>2018/akd/2018-MIREX2005Key.csv</csv>
 
<csv>2018/akd/2018-MIREX2005Key.csv</csv>
−
=====GiantStepsKey=====
+
====GiantStepsKey====
 
<csv>2018/akd/2018-GiantStepsKey.csv</csv>
 
<csv>2018/akd/2018-GiantStepsKey.csv</csv>
−
=====PresegmentedKeyIsophonics=====
+
====PresegmentedKeyIsophonics====
 
<csv>2018/akd/2018-PresegmentedKeyIsophonics.csv</csv>
 
<csv>2018/akd/2018-PresegmentedKeyIsophonics.csv</csv>
−
=====PresegmentedKeyRobbieWilliams=====
+
====PresegmentedKeyRobbieWilliams====
 
<csv>2018/akd/2018-PresegmentedKeyRobbieWilliams.csv</csv>
 
<csv>2018/akd/2018-PresegmentedKeyRobbieWilliams.csv</csv>
−
=====Billboard2012Key=====
+
====Billboard2012Key====
 
<csv>2018/akd/2018-Billboard2012Key.csv</csv>
 
<csv>2018/akd/2018-Billboard2012Key.csv</csv>

Latest revision as of 03:05, 22 September 2018

Introduction

This page contains the results of the 2018 edition of the MIREX automatic key detection estimation task.

What’s new?

  • Nothing has changed, which means the results can be directly compared to last year's.

Submissions

Abstract Contributors
CG1 PDF David Castells-Rufas, Adria Galin
GC1 PDF Adria Galin, David Castells-Rufas
CN1 PDF Chris Cannam, Katy Noland
NA1 PDF Nestor Napoles Lopez , Claire Arthur
FK1, FK3 PDF Filip Korzeniowski
OM1-OM3 PDF James Owers, Andrew McLeod

Results

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

MIREX2005Key

Algorithm Mirex Correct Adjacent Relative Parallel Chromatic Other
CG1 88.78 84.27 6.63 3.19 1.20 0.16 4.55
CN1 88.07 82.75 8.31 2.72 1.76 0.08 4.39
FK1 82.50 72.52 16.77 4.07 1.84 0.24 4.55
GC1 76.58 63.34 21.49 7.11 1.84 0.08 6.15
NA1 84.06 74.44 15.42 5.03 2.00 0.08 3.04
OM1 82.52 73.16 14.78 4.95 2.40 0.08 4.63
OM2 73.43 58.07 25.00 8.07 2.24 0.08 6.55
OM3 49.64 36.98 11.18 13.50 15.10 0.00 23.24

download these results as csv

GiantStepsKey

Algorithm Mirex Correct Adjacent Relative Parallel Chromatic Other
CG1 28.21 16.72 17.05 5.13 7.12 3.64 50.33
CN1 50.53 39.74 11.92 13.24 4.30 2.65 28.15
FK1 74.62 67.88 6.95 8.11 4.14 2.98 9.93
GC1 30.03 19.04 12.91 8.28 10.26 1.99 47.52
NA1 59.42 48.34 11.59 13.08 6.79 1.82 18.38
OM1 58.11 49.50 11.59 5.96 5.13 2.48 25.33
OM2 49.72 39.07 17.55 3.48 4.14 3.97 31.79
OM3 48.58 40.89 10.76 3.48 6.29 3.81 34.77

download these results as csv

PresegmentedKeyIsophonics

Algorithm Mirex Correct Adjacent Relative Parallel Chromatic Other
CG1 61.86 50.57 15.40 6.90 7.59 1.15 18.39
CN1 64.99 55.40 13.56 4.60 7.13 3.91 15.40
FK1 82.30 76.32 7.36 5.52 3.22 2.30 5.29
GC1 68.05 56.78 17.70 5.75 3.45 1.84 14.48
NA1 75.72 65.75 12.64 9.89 3.45 2.07 6.21
OM1 62.92 53.10 12.41 5.29 10.11 1.38 17.70
OM2 51.15 34.25 22.99 9.43 12.87 1.61 18.85
OM3 41.13 25.52 15.63 8.05 26.90 0.92 22.99

download these results as csv

PresegmentedKeyRobbieWilliams

Algorithm Mirex Correct Adjacent Relative Parallel Chromatic Other
CG1 54.66 41.80 20.63 4.23 6.35 1.06 25.93
CN1 62.28 53.97 10.58 7.94 3.17 7.41 16.93
FK1 81.38 72.49 10.05 12.17 1.06 0.00 4.23
GC1 62.70 51.32 16.93 7.94 2.65 0.00 21.16
NA1 77.20 66.14 16.40 8.47 1.59 0.00 7.41
OM1 63.86 53.97 13.76 6.88 4.76 0.00 20.63
OM2 51.32 38.62 17.46 5.82 11.11 0.00 26.98
OM3 57.99 48.15 11.64 5.29 12.17 0.53 22.22

download these results as csv

Billboard2012Key

Algorithm Mirex Correct Adjacent Relative Parallel Chromatic Other
CG1 63.01 52.74 15.07 2.74 9.59 2.05 17.81
CN1 67.40 58.22 12.33 5.48 6.85 6.85 10.27
FK1 88.22 84.25 2.74 5.48 4.79 0.00 2.74
GC1 69.66 58.22 20.55 2.05 2.74 0.68 15.75
NA1 79.59 70.55 13.70 5.48 2.74 0.68 6.85
OM1 71.58 60.27 13.70 6.16 13.01 0.00 6.85
OM2 55.41 41.10 19.18 3.42 18.49 0.68 17.12
OM3 39.04 25.34 8.90 7.53 34.93 0.00 23.29

download these results as csv

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2018:Audio_Key_Detection_Results&oldid=12829"