2009:Audio Chord Detection Results

From MIREX Wiki
Revision as of 19:10, 13 May 2010 by IMIRSELBot (talk | contribs) (Robot: Automated text replacement (-\[http://music-ir.org/mirex/2009/results/abs/ +[http://www.music-ir.org/mirex/abs/2009/))

Introduction

These are the results for the 2009 running of the Audio Chord Detection task set. For background information about this task set please refer to the 2009:Audio Chord Detection page.

Task Descriptions

Beatles data donated by Christopher Harte and extra dataset of around 37 songs donated by Matthias Mauch was used. 7 songs were excluded from the list where the average score for these songs across participants were less than 0.25

Task 1 (Pretrained Systems) Go to Task 1 Results:


Task 2 (Train-Test Systems) Go to Task 2 Results: System trained on ~2/3 of the dataset and tested on ~1/3. Folds were divided such that 2/3 of each album was represented in the train list and 1/3 in the test list.

Overlap score was calculated as the ratio between the overlap of the ground truth and detected chords and ground truth duration. Also a secondary overlap score was calculated by ignoring the major-minor variations of the detected chord (e.g., C major == C minor, etc.). The weighted average is computed across the songs by weighting each score by its the duration.

General Legend

Team ID

CH = Christopher Harte
DE = Dan Ellis
KO1 = Maksim Khadkevich, Maurizio Omologo
KO2 = Maksim Khadkevich, Maurizio Omologo
MD = Matthias Mauch, Katy Noland, Simon Dixon
OGF1 = Laurent Oudre, Yves Grenier,Cedric Fevotte (majmin)
OGF2 = Laurent Oudre, Yves Grenier,Cedric Fevotte (majmin7)
PP = Helene Papadopoulos, Geoffroy Peeters
PVM1 = Johan Pauwels, Matthias Varewyck,Jean-Pierre Martens
PVM2 = Johan Pauwels, Matthias Varewyck,Jean-Pierre Martens
RRHS1 = Thomas Rocher, Matthias Robine, Pierre Hanna,Robert Strandh
RSHS2 = Thomas Rocher, Matthias Robine, Pierre Hanna,Robert Strandh
RSHS3 = Thomas Rocher, Matthias Robine, Pierre Hanna,Robert Strandh
RUSUSL = J.T.Reed,Yushi Ueda,S.Siniscalchi,Yuki Uchiyama,Shigeki Sagayama,C.H.Lee
WEJ1 = Adrian Weller, Daniel Ellis, Tony Jebara
WEJ2 = Adrian Weller, Daniel Ellis, Tony Jebara
WEJ3 = Adrian Weller, Daniel Ellis, Tony Jebara
WEJ4 = Adrian Weller, Daniel Ellis, Tony Jebara

Overall Summary Results

Task 1 Results

Task 1 Overall Results

CH DE KO1 KO2 MD OGF1 OGF2 PP PVM1 PVM2 RRHS1 RRHS2 RRHS3
Wt. Ave. Overlap Score 0.576 0.697 0.697 0.708 0.712 0.706 0.711 0.673 0.682 0.654 0.629 0.545 0.538
Wt. Ave. Overlap Score (Merged maj/min) 0.625 0.731 0.734 0.741 0.748 0.770 0.777 0.715 0.710 0.698 0.688 0.600 0.688

download these results as csv

Task 1 Run Times

TBA file /nema-raid/www/mirex/results/2009/chord/task1/task1_runtimes.csv not found

Task 1 Results Per Artist/Album

2009:Audio Chord Detection Results Per Album#Task 1

Task 1 Friedman's Test for Significant Differences

The Tukey-Kramer HSD multi-comparison data below was generate using the following MATLAB instruction.

Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);

TeamID TeamID Lowerbound Mean Upperbound Significance
OGF2 MD -1.1304 0.1286 1.3876 FALSE
OGF2 OGF1 -1.0638 0.1952 1.4543 FALSE
OGF2 KO2 -0.7900 0.4690 1.7281 FALSE
OGF2 DE -0.2638 0.9952 2.2543 FALSE
OGF2 KO1 -0.0185 1.2405 2.4995 FALSE
OGF2 PP 0.3743 1.6333 2.8924 TRUE
OGF2 PVM1 0.1981 1.4571 2.7162 TRUE
OGF2 PVM2 1.4029 2.6619 3.9209 TRUE
OGF2 RRHS1 2.6981 3.9571 5.2162 TRUE
OGF2 CH 4.5767 5.8357 7.0947 TRUE
OGF2 RRHS2 5.8696 7.1286 8.3876 TRUE
OGF2 RRHS3 5.9719 7.2310 8.4900 TRUE
MD OGF1 -1.1924 0.0667 1.3257 FALSE
MD KO2 -0.9185 0.3405 1.5995 FALSE
MD DE -0.3924 0.8667 2.1257 FALSE
MD KO1 -0.1471 1.1119 2.3709 FALSE
MD PP 0.2457 1.5048 2.7638 TRUE
MD PVM1 0.0696 1.3286 2.5876 TRUE
MD PVM2 1.2743 2.5333 3.7924 TRUE
MD RRHS1 2.5696 3.8286 5.0876 TRUE
MD CH 4.4481 5.7071 6.9662 TRUE
MD RRHS2 5.7410 7.0000 8.2590 TRUE
MD RRHS3 5.8434 7.1024 8.3614 TRUE
OGF1 KO2 -0.9852 0.2738 1.5328 FALSE
OGF1 DE -0.4590 0.8000 2.0590 FALSE
OGF1 KO1 -0.2138 1.0452 2.3043 FALSE
OGF1 PP 0.1791 1.4381 2.6971 TRUE
OGF1 PVM1 0.0029 1.2619 2.5209 TRUE
OGF1 PVM2 1.2076 2.4667 3.7257 TRUE
OGF1 RRHS1 2.5029 3.7619 5.0209 TRUE
OGF1 CH 4.3815 5.6405 6.8995 TRUE
OGF1 RRHS2 5.6743 6.9333 8.1924 TRUE
OGF1 RRHS3 5.7767 7.0357 8.2947 TRUE
KO2 DE -0.7328 0.5262 1.7852 FALSE
KO2 KO1 -0.4876 0.7714 2.0304 FALSE
KO2 PP -0.0947 1.1643 2.4233 FALSE
KO2 PVM1 -0.2709 0.9881 2.2471 FALSE
KO2 PVM2 0.9338 2.1929 3.4519 TRUE
KO2 RRHS1 2.2291 3.4881 4.7471 TRUE
KO2 CH 4.1076 5.3667 6.6257 TRUE
KO2 RRHS2 5.4005 6.6595 7.9185 TRUE
KO2 RRHS3 5.5029 6.7619 8.0209 TRUE
DE KO1 -1.0138 0.2452 1.5043 FALSE
DE PP -0.6209 0.6381 1.8971 FALSE
DE PVM1 -0.7971 0.4619 1.7209 FALSE
DE PVM2 0.4076 1.6667 2.9257 TRUE
DE RRHS1 1.7029 2.9619 4.2209 TRUE
DE CH 3.5815 4.8405 6.0995 TRUE
DE RRHS2 4.8743 6.1333 7.3924 TRUE
DE RRHS3 4.9767 6.2357 7.4947 TRUE
KO1 PP -0.8662 0.3929 1.6519 FALSE
KO1 PVM1 -1.0424 0.2167 1.4757 FALSE
KO1 PVM2 0.1624 1.4214 2.6804 TRUE
KO1 RRHS1 1.4576 2.7167 3.9757 TRUE
KO1 CH 3.3362 4.5952 5.8543 TRUE
KO1 RRHS2 4.6291 5.8881 7.1471 TRUE
KO1 RRHS3 4.7315 5.9905 7.2495 TRUE
PP PVM1 -1.4352 -0.1762 1.0828 FALSE
PP PVM2 -0.2304 1.0286 2.2876 FALSE
PP RRHS1 1.0648 2.3238 3.5828 TRUE
PP CH 2.9434 4.2024 5.4614 TRUE
PP RRHS2 4.2362 5.4952 6.7543 TRUE
PP RRHS3 4.3386 5.5976 6.8566 TRUE
PVM1 PVM2 -0.0543 1.2048 2.4638 FALSE
PVM1 RRHS1 1.2410 2.5000 3.7590 TRUE
PVM1 CH 3.1196 4.3786 5.6376 TRUE
PVM1 RRHS2 4.4124 5.6714 6.9304 TRUE
PVM1 RRHS3 4.5148 5.7738 7.0328 TRUE
PVM2 RRHS1 0.0362 1.2952 2.5543 TRUE
PVM2 CH 1.9148 3.1738 4.4328 TRUE
PVM2 RRHS2 3.2076 4.4667 5.7257 TRUE
PVM2 RRHS3 3.3100 4.5690 5.8281 TRUE
RRHS1 CH 0.6196 1.8786 3.1376 TRUE
RRHS1 RRHS2 1.9124 3.1714 4.4304 TRUE
RRHS1 RRHS3 2.0148 3.2738 4.5328 TRUE
CH RRHS2 0.0338 1.2929 2.5519 TRUE
CH RRHS3 0.1362 1.3952 2.6543 TRUE
RRHS2 RRHS3 -1.1566 0.1024 1.3614 FALSE

download these results as csv

https://music-ir.org/mirex/2009/results/chord/task1/summary/small.Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png

Task 2 Results

Task 2 Overall Results

RUSUSL WEJ1 WEJ2 WEJ3 WEJ4
Wt. Ave. Overlap Score 0.701 0.704 0.723 0.723 0.742
Wt. Ave. Overlap Score (Merged maj/min) 0.760 0.743 0.762 0.760 0.777

download these results as csv

Task 2 Run Times

TBA file /nema-raid/www/mirex/results/2009/chord/task2/task2_runtimes.csv not found

Task 2 Results Per Artist/Album

2009:Audio Chord Detection Results Per Album#Task 2

Task 2 Friedman's Test for Significant Differences

The Tukey-Kramer HSD multi-comparison data below was generate using the following MATLAB instruction.

Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);

TeamID TeamID Lowerbound Mean Upperbound Significance
WEJ4 WEJ3 0.2936 0.7143 1.1349 TRUE
WEJ4 WEJ2 0.2341 0.6548 1.0754 TRUE
WEJ4 WEJ1 1.0055 1.4262 1.8468 TRUE
WEJ4 RUSUSL 0.7960 1.2167 1.6373 TRUE
WEJ3 WEJ2 -0.4802 -0.0595 0.3611 FALSE
WEJ3 WEJ1 0.2913 0.7119 1.1326 TRUE
WEJ3 RUSUSL 0.0817 0.5024 0.9230 TRUE
WEJ2 WEJ1 0.3508 0.7714 1.1921 TRUE
WEJ2 RUSUSL 0.1413 0.5619 0.9826 TRUE
WEJ1 RUSUSL -0.6302 -0.2095 0.2111 FALSE

download these results as csv

https://music-ir.org/mirex/2009/results/chord/task2/summary/small.Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_Mean_Ranks.png

Individual Results

Task1

CH = Christopher Harte
DE = Dan Ellis
KO1 = Maksim Khadkevich, Maurizio Omologo
KO2 = Maksim Khadkevich, Maurizio Omologo
MD = Matthias Mauch, Simon Dixon
OGF1 = Laurent Oudre,Yves Grenier,Cedric Fevotte
OGF2 = Laurent Oudre,Yves Grenier,Cedric Fevotte
PP = Helene Papadopoulos, Geoffroy Peeters
PVM1 = Johan Pauwels,Matthias Varewyck,Jean-Pierre Martens
PVM2 = Johan Pauwels,Matthias Varewyck,Jean-Pierre Martens
RRHS1 = Thomas Rocher, Matthias Robine,Pierre Hanna,Robert Strandh
RSHS2 = Thomas Rocher, Matthias Robine,Pierre Hanna,Robert Strandh
RSHS3 = Thomas Rocher, Matthias Robine,Pierre Hanna,Robert Strandh

Task2

RUSUSL = J.T.Reed,Yushi Ueda,S.Siniscalchi,Yuki Uchiyama,Shigeki Sagayama,C.H.Lee
WEJ1 = Adrian Weller, Daniel Ellis, Tony Jebara
WEJ2 = Adrian Weller, Daniel Ellis, Tony Jebara
WEJ3 = Adrian Weller, Daniel Ellis, Tony Jebara
WEJ4 = Adrian Weller, Daniel Ellis, Tony Jebara

Evaluations by Christopher Harte's Metrics

TBA

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2009:Audio_Chord_Detection_Results&oldid=6587"