Difference between revisions of "2018:Audio Chord Estimation Results"

From MIREX Wiki
(→‎Submissions)
(Add link to Github repositories)
 
(3 intermediate revisions by 2 users not shown)
Line 4: Line 4:
  
 
==What’s new?==
 
==What’s new?==
−
* This year the algorithms have additionally been evaluated on the "RWC-Popular" and "USPOP2002Chords" dataset annotated at the [http://steinhardt.nyu.edu/marl/ Music and Audio Research Lab] of NYU, whose annotations are [https://github.com/tmc323/Chord-Annotations publicly available]. The [https://staff.aist.go.jp/m.goto/RWC-MDB/rwc-mdb-p.html RWC-Popular dataset] contains 100 pop songs recorded specifically for music information retrieval research. The USPOP2002Chords set is the 195 file subset of the [https://labrosa.ee.columbia.edu/projects/musicsim/uspop2002.html USPOP2002 dataset] that have been annotated with chord sequences.
+
* All datasets and evaluation procedures are the same as last year's MIREX.
  
 
==Software==
 
==Software==
−
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output will be provided later <!--provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below.--> More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].
+
All software used for the evaluation has been made open-source. The evaluation framework is described by [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6637748 Pauwels and Peeters (2013)]. The corresponding binaries and code repository can be found on [https://github.com/jpauwels/MusOOEvaluator GitHub] and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the [https://github.com/jpauwels/MusOOEvaluator/blob/master/README.md readme].
  
 
The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.
 
The statistical comparison between the different submissions is explained in [http://www.terasoft.com.tw/conf/ismir2014/proceedings/T095_250_Paper.pdf Burgoyne et al. (2014)]. The software is available at [https://bitbucket.org/jaburgoyne/mirexace BitBucket]. It uses the detailed results provided below as input.
Line 31: Line 31:
 
|-
 
|-
 
| FK2
 
| FK2
−
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/WL1.pdf PDF]
+
| style="text-align: center;" | [PDF]
 
| [http://www.cp.jku.at Florian Krebs], [http://www.cp.jku.at Filip Korzeniowski], [http://www.ofai.at Sebastian Böck]
 
| [http://www.cp.jku.at Florian Krebs], [http://www.cp.jku.at Filip Korzeniowski], [http://www.ofai.at Sebastian Böck]
 
|}
 
|}
Line 67: Line 67:
 
===Detailed Results===
 
===Detailed Results===
  
−
More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from ?.
+
More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from [https://github.com/ismir-mirex/ace-results/tree/master/2018 this repository].
  
 
===Algorithmic Output===
 
===Algorithmic Output===
  
−
The raw output of the algorithms are available in ?. They can be used to experiment with alternative evaluation measures and statistics.
+
The raw output of the algorithms are available in [https://github.com/ismir-mirex/ace-output/tree/master/2018 this repository]. They can be used to experiment with alternative evaluation measures and statistics.

Latest revision as of 04:26, 14 October 2020

Introduction

This page contains the results of the 2018 edition of the MIREX automatic chord estimation tasks. This edition was the sixth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last five years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the 2013 edition.

What’s new?

  • All datasets and evaluation procedures are the same as last year's MIREX.

Software

All software used for the evaluation has been made open-source. The evaluation framework is described by Pauwels and Peeters (2013). The corresponding binaries and code repository can be found on GitHub and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the readme.

The statistical comparison between the different submissions is explained in Burgoyne et al. (2014). The software is available at BitBucket. It uses the detailed results provided below as input.

Submissions

Abstract Contributors
CM1 PDF Chris Cannam, Matthias Mauch
JLCX1, JLCX2 PDF Junyan Jiang, Ke Chen, Wei Li, Guangyu Xia
SG1 PDF Franz Strasser, Stefan Gaser
FK2 [PDF] Florian Krebs, Filip Korzeniowski, Sebastian Böck

Results

Summary

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

Isophonics2009
Algorithm MirexRoot MirexMajMin MirexMajMinBass MirexSevenths MirexSeventhsBass MeanSeg UnderSeg OverSeg
CM1 78.66 75.51 72.58 54.78 52.36 85.87 87.22 85.98
FK2 87.38 86.80 83.43 75.55 72.60 89.29 87.24 92.35
JLCX1 86.75 86.25 84.44 75.87 74.39 90.33 88.36 93.38
JLCX2 86.51 86.05 84.23 75.64 74.17 90.12 87.94 93.48
SG1 82.03 78.67 76.84 69.20 67.55 82.34 89.03 77.87

download these results as csv

Billboard2012
Algorithm MirexRoot MirexMajMin MirexMajMinBass MirexSevenths MirexSeventhsBass MeanSeg UnderSeg OverSeg
CM1 74.23 72.31 70.25 55.44 53.48 83.60 85.33 83.31
FK2 86.30 86.01 83.12 61.12 58.75 88.38 86.78 90.96
JLCX1 83.09 82.79 81.48 70.88 69.72 87.72 86.65 89.85
JLCX2 83.60 83.31 82.01 71.26 70.11 88.35 86.94 90.90
SG1 82.11 79.28 78.04 66.57 65.53 81.60 90.21 76.22

download these results as csv

Billboard2013
Algorithm MirexRoot MirexMajMin MirexMajMinBass MirexSevenths MirexSeventhsBass MeanSeg UnderSeg OverSeg
CM1 71.23 67.36 65.28 49.07 47.25 81.50 83.13 82.53
FK2 80.60 78.37 75.85 55.81 53.57 83.59 83.01 87.54
JLCX1 78.76 77.30 76.14 63.69 62.71 84.86 83.12 89.21
JLCX2 79.38 77.92 76.74 64.22 63.20 85.55 83.53 90.28
SG1 77.92 72.41 71.04 58.26 57.10 77.95 87.70 73.46

download these results as csv

JayChou29
Algorithm MirexRoot MirexMajMin MirexMajMinBass MirexSevenths MirexSeventhsBass MeanSeg UnderSeg OverSeg
CM1 72.82 72.15 65.55 54.46 49.05 86.58 86.94 86.84
FK2 81.42 80.48 69.66 51.77 43.18 89.08 87.67 90.87
JLCX1 84.41 84.12 81.42 63.14 61.08 92.59 91.82 93.46
JLCX2 84.77 84.49 81.75 63.38 61.31 93.16 92.26 94.16
SG1 80.59 75.65 72.28 56.43 53.80 82.51 91.49 75.30

download these results as csv

RobbieWilliams
Algorithm MirexRoot MirexMajMin MirexMajMinBass MirexSevenths MirexSeventhsBass MeanSeg UnderSeg OverSeg
CM1 81.94 78.29 76.09 57.97 55.94 87.95 89.00 87.39
FK2 89.38 88.12 85.06 83.38 80.68 90.96 89.52 92.84
JLCX1 87.14 85.33 83.31 75.91 74.10 90.95 89.30 93.23
JLCX2 87.25 85.45 83.40 76.01 74.17 91.03 89.04 93.73
SG1 84.87 80.28 78.58 73.78 72.16 85.24 90.85 80.75

download these results as csv

RWC-Popular
Algorithm MirexRoot MirexMajMin MirexMajMinBass MirexSevenths MirexSeventhsBass MeanSeg UnderSeg OverSeg
CM1 79.12 77.95 74.30 63.33 59.89 88.62 88.11 89.67
FK2 87.20 86.58 80.92 58.04 52.78 89.94 87.94 92.37
JLCX1 87.11 86.71 84.93 73.78 72.17 91.15 88.77 93.99
JLCX2 86.90 86.53 84.76 73.51 71.88 91.07 88.48 94.16
SG1 82.22 78.06 76.06 63.65 61.86 82.62 90.66 76.44

download these results as csv

USPOP2002Chords
Algorithm MirexRoot MirexMajMin MirexMajMinBass MirexSevenths MirexSeventhsBass MeanSeg UnderSeg OverSeg
CM1 78.41 76.24 72.74 59.84 56.67 85.81 86.81 86.02
FK2 82.34 80.72 76.04 64.50 60.30 86.59 85.49 89.17
JLCX1 85.66 84.89 82.95 75.08 73.35 89.57 88.22 91.88
JLCX2 85.34 84.67 82.72 74.86 73.12 89.35 87.72 92.00
SG1 81.67 77.38 75.16 65.69 63.68 83.46 90.37 78.59

download these results as csv


Detailed Results

More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from this repository.

Algorithmic Output

The raw output of the algorithms are available in this repository. They can be used to experiment with alternative evaluation measures and statistics.

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2018:Audio_Chord_Estimation_Results&oldid=13284"