2007:Audio Cover Song Identification Results

From MIREX Wiki

Introduction

These are the results for the 2007 running of the Audio Cover Song Identification task. For background information about this task set please refer to the 2007:Audio Cover Song Identification page.

Each system was given a collection of 1000 songs which included of 30 different classes (sets) of cover songs where each class/set was represented by 11 different versions of a particular song. Each of the 330 cover songs were used as queries and the systems were required to return 10 results for each query. Systems were evaluated on the number of the songs from the same class/set as the query that were returned in the list of 10 results for each query. Average precision, which looks at the entire per-query rank-ordered list of all songs in the collection, is a new metric explored this year.


General Legend

Team ID

EC = Daniel P. W. Ellis, Courtenay V. Cotton
IM = IMIRSEL M2K
JB = Juan Bello
JEC = Jesper Højvang Jensen, Daniel P. W. Ellis, Mads G. Christensen, Søren Holdt
KL1 = Kyogu Lee 1
KL2 = Kyogu Lee 2
KP = Youngmoo E. Kim, Daniel Perelstein
SG = Joan Serrà, Emilia Gómez

Overall Summary Results

Measure Range EC JB JEC KL1 KL2 KP IM SG
Total number of covers identified in top 10 [0-3300] 1207 869 762 425 291 190 34 1653
Mean number of covers identified in top 10 (average performance) [0-10] 3.658 2.633 2.309 1.288 0.882 0.576 0.103 5.009
Mean (arithmetic) of Avg. Precisions [0-1] 0.330 0.267 0.238 0.13 0.086 0.061 0.017 0.521
Mean rank of first correctly identified cover [0-1] 13.994 29.527 22.209 57.542 51.094 46.539 97.470 9.367

download these results as csv

Runtimes

Where algorithms have been multi-threaded, the longest runtime is reported.

Where runtimes were not properly reported, file timestamps have been used to approximate a runtime.

CLOCK TIME MEASURES (not cpu time)
Team Clock time (HH:MM) # Threads
EC 04:28 5
IM 02:04 1
JB 04:32 8
JEC 00:47 8
KL1 10:45 8
KL2 02:37 1
KP ~03:51 1
SG 01:37 1

download these results as csv

Number of Correct Covers at Rank X Returned in Top Ten

Rank EC JB JEC KL2 KL1 KP IM SG
1 164 151 143 83 4 19 6 242
2 182 127 124 78 8 26 3 222
3 169 121 91 62 6 24 6 209
4 153 103 98 45 9 21 2 191
5 147 88 74 35 8 18 2 180
6 112 70 67 37 9 22 3 166
7 108 77 53 15 11 14 2 147
8 84 59 50 21 51 17 1 131
9 58 44 39 27 86 13 4 102
10 30 29 23 22 99 16 5 63
SUM 1207 869 762 425 291 190 34 1653

download these results as csv

Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Friedman's ANOVA Table
Source SS df MS Chi-sq Prob>Chi-sq
Columns 871.3167 7 124.4738 145.3925 0
Error 387.1833 203 1.9073
Total 1.2585e+003 239

download these results as csv

TeamID TeamID Lowerbound Mean Upperbound Significance
EC JB -0.5491 1.3667 3.2824 FALSE
EC JEC -0.7158 1.2000 3.1158 FALSE
EC KL1 0.8676 2.7833 4.6991 TRUE
EC KL2 1.2676 3.1833 5.0991 TRUE
EC KP 1.6176 3.5333 5.4491 TRUE
EC IM 3.1842 5.1000 7.0158 TRUE
EC SG -3.0824 -1.1667 0.7491 FALSE
JB JEC -2.0824 -0.1667 1.7491 FALSE
JB KL1 -0.4991 1.4167 3.3324 FALSE
JB KL2 -0.0991 1.8167 3.7324 FALSE
JB KP 0.2509 2.1667 4.0824 TRUE
JB IM 1.8176 3.7333 5.6491 TRUE
JB SG -4.4491 -2.5333 -0.6176 TRUE
JEC KL1 -0.3324 1.5833 3.4991 FALSE
JEC KL2 0.0676 1.9833 3.8991 TRUE
JEC KP 0.4176 2.3333 4.2491 TRUE
JEC IM 1.9842 3.9000 5.8158 TRUE
JEC SG -4.2824 -2.3667 -0.4509 TRUE
KL1 KL2 -1.5158 0.4000 2.3158 FALSE
KL1 KP -1.1658 0.7500 2.6658 FALSE
KL1 IM 0.4009 2.3167 4.2324 TRUE
KL1 SG -5.8658 -3.9500 -2.0342 TRUE
KL2 KP -1.5658 0.3500 2.2658 FALSE
KL2 IM 0.0009 1.9167 3.8324 TRUE
KL2 SG -6.2658 -4.3500 -2.4342 TRUE
KP IM -0.3491 1.5667 3.4824 FALSE
KP SG -6.6158 -4.7000 -2.7842 TRUE
IM SG -8.1824 -6.2667 -4.3509 TRUE

download these results as csv

2007 cover friedmans 1.png

Average Performance per Query Group

These are the arithmetic means of the average precisions within each of the 30 query groups.

Cover Group EC JB JEC KL1 KL2 KP IM SG
1 0.384 0.627 0.122 0.139 0.202 0.048 0.020 0.861
2 0.331 0.106 0.149 0.159 0.092 0.057 0.013 0.400
3 0.206 0.035 0.057 0.046 0.026 0.022 0.025 0.201
4 0.838 0.566 0.791 0.293 0.326 0.302 0.020 0.998
5 0.373 0.705 0.468 0.141 0.235 0.055 0.023 0.825
6 0.078 0.032 0.049 0.034 0.033 0.038 0.020 0.121
7 0.551 0.644 0.416 0.430 0.194 0.049 0.020 0.961
8 0.392 0.205 0.176 0.037 0.043 0.017 0.010 0.439
9 0.354 0.084 0.120 0.043 0.047 0.043 0.019 0.342
10 0.343 0.220 0.273 0.181 0.142 0.080 0.011 0.696
11 0.072 0.398 0.206 0.217 0.099 0.089 0.016 0.859
12 0.217 0.038 0.226 0.020 0.041 0.068 0.015 0.194
13 0.158 0.132 0.266 0.190 0.059 0.080 0.015 0.405
14 0.377 0.949 0.705 0.220 0.071 0.046 0.013 0.999
15 0.588 0.770 0.174 0.218 0.065 0.040 0.016 0.766
16 0.512 0.186 0.092 0.022 0.029 0.028 0.014 0.525
17 0.398 0.411 0.238 0.433 0.208 0.141 0.018 0.905
18 0.473 0.451 0.302 0.167 0.095 0.043 0.016 0.697
19 0.055 0.045 0.122 0.029 0.022 0.064 0.014 0.051
20 0.168 0.060 0.018 0.100 0.070 0.043 0.012 0.173
21 0.131 0.072 0.117 0.021 0.033 0.019 0.023 0.389
22 0.060 0.014 0.021 0.026 0.017 0.043 0.018 0.052
23 0.521 0.280 0.486 0.038 0.027 0.044 0.016 0.761
24 0.203 0.063 0.045 0.009 0.016 0.054 0.013 0.088
25 0.420 0.284 0.327 0.018 0.020 0.040 0.023 0.471
26 0.774 0.233 0.608 0.169 0.094 0.065 0.012 0.838
27 0.152 0.104 0.021 0.019 0.038 0.033 0.033 0.225
28 0.114 0.153 0.116 0.101 0.018 0.014 0.018 0.450
29 0.208 0.035 0.054 0.018 0.020 0.022 0.015 0.241
30 0.453 0.102 0.380 0.352 0.197 0.134 0.010 0.703

download these results as csv

Individual Results Files

Average Precision Scores for Each Query

EC = Daniel P. W. Ellis, Courtenay V. Cotton
IM = IMIRSEL M2K
JB = Juan Bello
JEC = Jesper Højvang Jensen, Daniel P. W. Ellis, Mads G. Christensen, Søren Holdt
KL1 = Kyogu Lee 1
KL2 = Kyogu Lee 2
KP = Youngmoo E. Kim, Daniel Perelstein
SG = Joan Serrà, Emilia Gómez

Ranks of the Ten Cover Songs Returned for Each Query

EC = Daniel P. W. Ellis, Courtenay V. Cotton
IM = IMIRSEL M2K
JB = Juan Bello
JEC = Jesper Højvang Jensen, Daniel P. W. Ellis, Mads G. Christensen, Søren Holdt
KL1 = Kyogu Lee 1
KL2 = Kyogu Lee 2
KP = Youngmoo E. Kim, Daniel Perelstein
SG = Joan Serrà, Emilia Gómez

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2007:Audio_Cover_Song_Identification_Results&oldid=7398"