JohanPauwels: Add note about comparison with last year

2018-09-22T09:05:44Z

Add note about comparison with last year

Yun Hao: Created page with "==Introduction== This page contains the results of the 2018 edition of the MIREX automatic key detection estimation task. ==What’s new?== * The NEMA system was retired last y..."

2018-09-18T20:28:16Z

Created page with "==Introduction== This page contains the results of the 2018 edition of the MIREX automatic key detection estimation task. ==What’s new?== * The NEMA system was retired last y..."

New page

==Introduction==

This page contains the results of the 2018 edition of the MIREX automatic key detection estimation task.

==What’s new?==
* The NEMA system was retired last year, since a bug has been found in the calculation of the results. Keys with tonics related by a fifth and the same mode (a.k.a. adjacent keys) are supposed to get a score of 0.5, but only ascending fifths (going from ground-truth to estimation) were counted, not descending ones. It has been brought to my attention that the description of the measure on the wiki has been ambiguous for years, and probably the NEMA implementer got confused by this. However, the intention has always been to count ascending and descending fifth (or fourth) relationships between the tonics (in my humble opinion).
* New datasets: "PresegmentedKeyIsophonics" and "PresegmentedKeyRobbieWilliams" use the local key annotations for the [http://isophonics.net/content/reference-annotations Isophonics set] and the [http://ispg.deib.polimi.it/mir-software.html Robbie Williams set], but have been split into separate files according to the local key annotations. The segments annotated with major and minor modes have been retained and were presented to the submissions. Therefore their results are slightly optimistic in the sense that the segments are guaranteed to contain just a single key, which is not the case for real-world songs. Keep also in mind that some files are strongly correlated (different segments or even repeated chorusses of the same song). Any statistical analysis of the results (e.g. pairwise significance tests) that relies on independence between files is consequently invalid.
* New dataset: "Billboard2012Key" is the subset of the Billboard2012 chord dataset for which it was possible to derive the key automatically from the chord annotations (using the procedure outlined by Korzeniowski & Widmer in their [https://arxiv.org/abs/1706.02921 2017 EUSIPCO paper]). The annotations are [http://www.cp.jku.at/people/korzeniowski/bb.zip freely available]

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CG1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/CG1.pdf PDF]
| [http://www.uab.cat David Castells-Rufas], [http://www.uab.cat/enginyeria/ Adria Galin]
|-
| GC1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/GC1.pdf PDF]
| [http://www.uab.cat/enginyeria/ Adria Galin], [http://www.uab.cat David Castells-Rufas]
|-
| CN1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/CN1.pdf PDF]
| [https://code.soundsoftware.ac.uk/users/3 Chris Cannam], [http://c4dm.eecs.qmul.ac.uk/ Katy Noland]
|-
| NA1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/NA1.pdf PDF]
| [http://ddmal.music.mcgill.ca/ Nestor Napoles Lopez] , [https://music.gatech.edu/ Claire Arthur]
|-
| FK1, FK3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/FK1.pdf PDF]
| [http://www.cp.jku.at Filip Korzeniowski]
|-
| OM1-OM3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/OM1.pdf PDF]
| [http://JamesOwers.github.io James Owers], [http://homepages.inf.ed.ac.uk/amcleod8 Andrew McLeod]
|-
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

=====MIREX2005Key=====
<csv>2018/akd/2018-MIREX2005Key.csv</csv>
=====GiantStepsKey=====
<csv>2018/akd/2018-GiantStepsKey.csv</csv>
=====PresegmentedKeyIsophonics=====
<csv>2018/akd/2018-PresegmentedKeyIsophonics.csv</csv>
=====PresegmentedKeyRobbieWilliams=====
<csv>2018/akd/2018-PresegmentedKeyRobbieWilliams.csv</csv>
=====Billboard2012Key=====
<csv>2018/akd/2018-Billboard2012Key.csv</csv>

@@ Line 4: / Line 4: @@
 ==What’s new?==
-* The NEMA system was retired last year, since a bug has been found in the calculation of the results. Keys with tonics related by a fifth and the same mode (a.k.a. adjacent keys) are supposed to get a score of 0.5, but only ascending fifths (going from ground-truth to estimation) were counted, not descending ones. It has been brought to my attention that the description of the measure on the wiki has been ambiguous for years, and probably the NEMA implementer got confused by this. However, the intention has always been to count ascending and descending fifth (or fourth) relationships between the tonics (in my humble opinion).
+* Nothing has changed, which means the results can be directly compared to last year's.
 ==Submissions==
@@ Line 42: / Line 40: @@
 ==Results==
 All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).
-=====MIREX2005Key=====
+====MIREX2005Key====
 <csv>2018/akd/2018-MIREX2005Key.csv</csv>
-=====GiantStepsKey=====
+====GiantStepsKey====
 <csv>2018/akd/2018-GiantStepsKey.csv</csv>
-=====PresegmentedKeyIsophonics=====
+====PresegmentedKeyIsophonics====
 <csv>2018/akd/2018-PresegmentedKeyIsophonics.csv</csv>
-=====PresegmentedKeyRobbieWilliams=====
+====PresegmentedKeyRobbieWilliams====
 <csv>2018/akd/2018-PresegmentedKeyRobbieWilliams.csv</csv>
-=====Billboard2012Key=====
+====Billboard2012Key====
 <csv>2018/akd/2018-Billboard2012Key.csv</csv>

2018:Audio Key Detection Results - Revision history

JohanPauwels: Add note about comparison with last year

Yun Hao: Created page with "==Introduction== This page contains the results of the 2018 edition of the MIREX automatic key detection estimation task. ==What’s new?== * The NEMA system was retired last y..."