Difference between revisions of "2008:Audio Cover Song Identification Results"

From MIREX Wiki
(→‎Average Precision Scores for Each Query)
(→‎Ranks of the Ten Cover Songs Returned for Each Query)
 
Line 62: Line 62:
 
'''EL3''' = [https://www.music-ir.org/mirex/results/2008/cover.el3.eval.debug.csv A. Egorov, G. Linetsky] <br />
 
'''EL3''' = [https://www.music-ir.org/mirex/results/2008/cover.el3.eval.debug.csv A. Egorov, G. Linetsky] <br />
 
'''JCJ''' = [https://www.music-ir.org/mirex/results/2008/cover.jcj.eval.debug.csv J. H. Jensen, M. G. Christensen, S. H. Jensen] <br />
 
'''JCJ''' = [https://www.music-ir.org/mirex/results/2008/cover.jcj.eval.debug.csv J. H. Jensen, M. G. Christensen, S. H. Jensen] <br />
−
'''SGH1''' = [https://www.music-ir.org/mirex/results/2008/cover.sgh1.eval.debug.csv J. Serrà, E. Gόmez, P. Herrera] <br />
+
'''SGH1''' = [https://www.music-ir.org/mirex/results/2008/cover.sgh1.eval.debug.csv J. Serrà, E. Gómez, P. Herrera] <br />
−
'''SGH2''' = [https://www.music-ir.org/mirex/results/2008/cover.sgh2.eval.csv J. Serrà, E. Gόmez, P. Herrera] <br />
+
'''SGH2''' = [https://www.music-ir.org/mirex/results/2008/cover.sgh2.eval.csv J. Serrà, E. Gómez, P. Herrera] <br />
  
 
===Runtimes===
 
===Runtimes===

Latest revision as of 16:10, 23 July 2010

Still missing runtimes JSD Sept. 11 2008.

Introduction

These are the results for the 2008 running of the Audio Cover Song Identification task. For background information about this task set please refer to the 2008:Audio Cover Song Identification page.

Each system was given a collection of 1000 songs which included of 30 different classes (sets) of cover songs where each class/set was represented by 11 different versions of a particular song. Each of the 330 cover songs were used as queries and the systems were required to return 10 results for each query. Systems were evaluated on the number of the songs from the same class/set as the query that were returned in the list of 10 results for each query. Average precision, which looks at the entire per-query rank-ordered list of all songs in the collection, was the new metric introduced last year.


General Legend

Team ID

CL1 = C. Cao, M. Li
CL2 = C. Cao, M. Li
EL1 = A. Egorov, G. Linetsky
EL2 = A. Egorov, G. Linetsky
EL3 = A. Egorov, G. Linetsky
JCJ = J. H. Jensen, M. G. Christensen, S. H. Jensen
SGH1 = J. Serrà, E. Gómez, P. Herrera
SGH2 = J. Serrà, E. Gómez, P. Herrera

Overall Summary Results

Measure Range CL1 CL2 EL1 EL2 EL3 JCJ SGH1 SGH2
Total number of covers identified in top 10 [0-3300] 1056 1073 1762 1781 1778 763 2422 2116
Mean number of covers identified in top 10 (average performance) [0-10] 3.20 3.25 5.34 5.40 5.39 2.31 7.34 6.41
Mean (arithmetic) of Avg. Precisions [0-1] 0.34 0.34 0.55 0.56 0.56 0.23 0.75 0.66
Mean rank of first correctly identified cover [1-1000] 18.83 18.78 11.19 11.17 11.55 23.98 4.80 4.55

download these results as csv


Number of Correct Covers at Rank X Returned in Top Ten

RankX CL1 CL2 EL1 EL2 EL3 JCJ SGH1 SGH2
1 192 191 259 259 258 151 281 281
2 164 159 250 251 253 122 286 277
3 143 150 236 235 231 95 278 261
4 131 132 201 213 214 80 272 250
5 114 118 177 176 178 76 265 231
6 96 90 176 177 179 67 265 222
7 80 87 154 158 155 52 247 200
8 62 63 136 141 136 45 230 170
9 47 53 121 120 119 39 183 134
10 27 30 52 51 55 36 115 90
Total 1056 1073 1762 1781 1778 763 2422 2116

download these results as csv

Run Times

CL1 CL2 EL1 EL2 EL3 JCJ SGH1 SGH2
Run Time (sec) 281487 281352 136469 435240 434712 3648 373370 373371

download these results as csv

CL1,CL2 ran on FAST2,FAST3. All others ran on ALE Nodes.

Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Source SS df MS Chi-sq Prob>Chi-sq
Columns 854.92 7 122.131 144.03 0
Error 391.58 203 1.929
Total 1246.5 239

download these results as csv

TeamID TeamID Lowerbound Mean Upperbound Significance
CL1 CL2 -2.2066 -0.3000 1.6066 FALSE
CL1 EL1 -4.4233 -2.5167 -0.6101 TRUE
CL1 EL2 -4.4399 -2.5333 -0.6267 TRUE
CL1 EL3 -4.3399 -2.4333 -0.5267 TRUE
CL1 JCJ -1.2899 0.6167 2.5233 FALSE
CL1 SGH1 -6.9399 -5.0333 -3.1267 TRUE
CL1 SGH2 -6.1066 -4.2000 -2.2934 TRUE
CL2 EL1 -4.1233 -2.2167 -0.3101 TRUE
CL2 EL2 -4.1399 -2.2333 -0.3267 TRUE
CL2 EL3 -4.0399 -2.1333 -0.2267 TRUE
CL2 JCJ -0.9899 0.9167 2.8233 FALSE
CL2 SGH1 -6.6399 -4.7333 -2.8267 TRUE
CL2 SGH2 -5.8066 -3.9000 -1.9934 TRUE
EL1 EL2 -1.9233 -0.0167 1.8899 FALSE
EL1 EL3 -1.8233 0.0833 1.9899 FALSE
EL1 JCJ 1.2267 3.1333 5.0399 TRUE
EL1 SGH1 -4.4233 -2.5167 -0.6101 TRUE
EL1 SGH2 -3.5899 -1.6833 0.2233 FALSE
EL2 EL3 -1.8066 0.1000 2.0066 FALSE
EL2 JCJ 1.2434 3.1500 5.0566 TRUE
EL2 SGH1 -4.4066 -2.5000 -0.5934 TRUE
EL2 SGH2 -3.5733 -1.6667 0.2399 FALSE
EL3 JCJ 1.1434 3.0500 4.9566 TRUE
EL3 SGH1 -4.5066 -2.6000 -0.6934 TRUE
EL3 SGH2 -3.6733 -1.7667 0.1399 FALSE
JCJ SGH1 -7.5566 -5.6500 -3.7434 TRUE
JCJ SGH2 -6.7233 -4.8167 -2.9101 TRUE
SGH1 SGH2 -1.0733 0.8333 2.7399 FALSE

download these results as csv

File:Coversong.friedman.png

Average Performance per Query Group

These are the arithmetic means of the average precisions within each of the 30 query groups.

QueryGroupID CL1 CL2 EL1 EL2 EL3 JCJ SGH1 SGH2
1 0.411 0.412 0.792 0.819 0.819 0.128 1.000 0.770
2 0.163 0.152 0.383 0.381 0.375 0.165 0.690 0.562
3 0.024 0.023 0.338 0.320 0.332 0.038 0.696 0.534
4 0.730 0.752 0.991 0.991 0.991 0.501 0.995 0.995
5 0.819 0.838 0.844 0.843 0.843 0.475 0.879 0.847
6 0.033 0.030 0.136 0.128 0.130 0.025 0.211 0.212
7 0.406 0.346 0.923 0.920 0.929 0.268 0.998 0.981
8 0.054 0.032 0.503 0.562 0.578 0.203 0.671 0.536
9 0.306 0.314 0.302 0.377 0.371 0.259 0.894 0.734
10 0.507 0.506 0.720 0.713 0.717 0.285 1.000 0.916
11 0.585 0.595 0.749 0.749 0.751 0.207 0.847 0.851
12 0.355 0.388 0.398 0.463 0.386 0.174 0.985 0.529
13 0.316 0.323 0.426 0.427 0.423 0.300 0.474 0.482
14 0.416 0.379 0.998 0.998 0.996 0.589 1.000 1.000
15 0.511 0.498 0.816 0.814 0.814 0.160 0.832 0.832
16 0.041 0.037 0.567 0.492 0.547 0.132 0.899 0.766
17 0.849 0.852 0.936 0.935 0.936 0.280 0.977 0.977
18 0.472 0.511 0.739 0.743 0.725 0.298 0.951 0.930
19 0.093 0.104 0.093 0.104 0.103 0.079 0.118 0.100
20 0.427 0.454 0.357 0.352 0.353 0.021 0.608 0.487
21 0.037 0.034 0.460 0.459 0.460 0.084 0.717 0.516
22 0.054 0.058 0.078 0.078 0.078 0.056 0.027 0.035
23 0.333 0.338 0.704 0.734 0.737 0.663 0.820 0.800
24 0.019 0.020 0.148 0.130 0.135 0.034 0.567 0.444
25 0.025 0.038 0.553 0.556 0.545 0.121 0.843 0.683
26 0.832 0.831 0.821 0.821 0.770 0.672 1.000 1.000
27 0.221 0.200 0.336 0.362 0.372 0.119 0.722 0.474
28 0.163 0.161 0.489 0.510 0.476 0.329 0.474 0.457
29 0.085 0.091 0.244 0.297 0.272 0.102 0.648 0.520
30 0.831 0.835 0.787 0.787 0.787 0.281 0.966 0.950
Mean 0.337 0.338 0.554 0.562 0.558 0.235 0.750 0.664

download these results as csv

Individual Results Files

Average Precision Scores for Each Query

CL1 = C. Cao, M. Li
CL2 = C. Cao, M. Li
EL1 = A. Egorov, G. Linetsky
EL2 = A. Egorov, G. Linetsky
EL3 = A. Egorov, G. Linetsky
JCJ = J. H. Jensen, M. G. Christensen, S. H. Jensen
SGH1 = J. Serrà, E. Gómez, P. Herrera
SGH2 = J. Serrà, E. Gómez, P. Herrera

Ranks of the Ten Cover Songs Returned for Each Query

CL1 = C. Cao, M. Li
CL2 = C. Cao, M. Li
EL1 = A. Egorov, G. Linetsky
EL2 = A. Egorov, G. Linetsky
EL3 = A. Egorov, G. Linetsky
JCJ = J. H. Jensen, M. G. Christensen, S. H. Jensen
SGH1 = J. Serrà, E. Gómez, P. Herrera
SGH2 = J. Serrà, E. Gómez, P. Herrera

Runtimes

Where algorithms have been multi-threaded, the longest runtime is reported.

Where runtimes were not properly reported, file timestamps have been used to approximate a runtime.

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2008:Audio_Cover_Song_Identification_Results&oldid=7382"