Difference between revisions of "2007:Audio Cover Song Identification Results"
From MIREX Wiki
(→Number of Correct Covers at Rank X Returned in Top Ten) |
(→Ranks of the Ten Cover Songs Returned for Each Query) |
||
| (18 intermediate revisions by 5 users not shown) | |||
| Line 1: | Line 1: | ||
| − | |||
| − | |||
==Introduction== | ==Introduction== | ||
| − | These are the results for the 2007 running of the Audio Cover Song Identification task. For background information about this task set please refer to the [[Audio Cover Song Identification]] page. | + | These are the results for the 2007 running of the Audio Cover Song Identification task. For background information about this task set please refer to the [[2007:Audio Cover Song Identification]] page. |
| − | Each system was given a collection of 1000 songs which included of 30 different classes (sets) of cover songs where each class/set was represented by 11 different versions of a particular song. Each of the 330 cover songs were used as queries and the systems were required to return 10 results for each query. Systems were evaluated on the number of the songs from the same class/set as the query that were returned in the list of 10 results for each query. | + | Each system was given a collection of 1000 songs which included of 30 different classes (sets) of cover songs where each class/set was represented by 11 different versions of a particular song. Each of the 330 cover songs were used as queries and the systems were required to return 10 results for each query. Systems were evaluated on the number of the songs from the same class/set as the query that were returned in the list of 10 results for each query. Average precision, which looks at the entire per-query rank-ordered list of all songs in the collection, is a new metric explored this year. |
===General Legend=== | ===General Legend=== | ||
====Team ID==== | ====Team ID==== | ||
| − | + | '''EC''' = [https://www.music-ir.org/mirex/abstracts/2007/CS_ellis.pdf Daniel P. W. Ellis, Courtenay V. Cotton]<br /> | |
| − | '''EC''' = [https://www.music-ir.org/ | ||
'''IM''' = IMIRSEL M2K<br/> | '''IM''' = IMIRSEL M2K<br/> | ||
| − | '''JB''' = [https://www.music-ir.org/ | + | '''JB''' = [https://www.music-ir.org/mirex/abstracts/2007/CS_bello.pdf Juan Bello] <br/> |
| − | '''JEC''' = [https://www.music-ir.org/ | + | '''JEC''' = [https://www.music-ir.org/mirex/abstracts/2007/CS_jensen.pdf Jesper Højvang Jensen, Daniel P. W. Ellis, Mads G. Christensen, Søren Holdt]<br /> |
'''KL1''' = Kyogu Lee 1<br /> | '''KL1''' = Kyogu Lee 1<br /> | ||
'''KL2''' = Kyogu Lee 2<br /> | '''KL2''' = Kyogu Lee 2<br /> | ||
| − | '''KP''' = [https://www.music-ir.org/ | + | '''KP''' = [https://www.music-ir.org/mirex/abstracts/2007/CS_kim.pdf Youngmoo E. Kim, Daniel Perelstein] <br /> |
| − | '''SG''' = [https://www.music-ir.org/ | + | '''SG''' = [https://www.music-ir.org/mirex/abstracts/2007/CS_serra.pdf Joan Serrà, Emilia Gómez]<br /> |
==Overall Summary Results== | ==Overall Summary Results== | ||
| − | <csv>cover.overall.stats.csv</csv> | + | <csv>2007/cover.overall.stats.csv</csv> |
| + | |||
| + | ===Runtimes=== | ||
| + | Where algorithms have been multi-threaded, the longest runtime is reported. | ||
| + | |||
| + | Where runtimes were not properly reported, file timestamps have been used to approximate a runtime. | ||
| + | |||
| + | <csv>2007/cover.runtimes.csv</csv> | ||
===Number of Correct Covers at Rank X Returned in Top Ten=== | ===Number of Correct Covers at Rank X Returned in Top Ten=== | ||
| − | <csv>cover.rank.distributions.csv</csv> | + | <csv>2007/cover.rank.distributions.csv</csv> |
===Friedman's Test for Significant Differences=== | ===Friedman's Test for Significant Differences=== | ||
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups.<br /> Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); | The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups.<br /> Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); | ||
| − | <csv>cover.friedmans.summary.csv</csv> | + | <csv>2007/cover.friedmans.summary.csv</csv> |
| − | <csv>cover.friedmans.details.csv</csv> | + | <csv>2007/cover.friedmans.details.csv</csv> |
[[Image:2007 cover friedmans 1.png]] | [[Image:2007 cover friedmans 1.png]] | ||
| + | |||
| + | ===Average Performance per Query Group=== | ||
| + | These are the arithmetic means of the average precisions within each of the 30 query groups. | ||
| + | |||
| + | <csv>2007/cover.avg.prec.by.groups.csv</csv> | ||
==Individual Results Files== | ==Individual Results Files== | ||
===Average Precision Scores for Each Query=== | ===Average Precision Scores for Each Query=== | ||
| − | '''EC''' = [https://www.music-ir.org/ | + | '''EC''' = [https://www.music-ir.org/mirex/results/2007/cover.labrosa.eval.csv Daniel P. W. Ellis, Courtenay V. Cotton]<br /> |
| − | '''IM''' = [https://www.music-ir.org/ | + | '''IM''' = [https://www.music-ir.org/mirex/results/2007/cover.imirsel.eval.csv IMIRSEL M2K]<br/> |
| − | '''JB''' = [https://www.music-ir.org/ | + | '''JB''' = [https://www.music-ir.org/mirex/results/2007/cover.bello.eval.csv Juan Bello] <br/> |
| − | '''JEC''' = [https://www.music-ir.org/ | + | '''JEC''' = [https://www.music-ir.org/mirex/results/2007/cover.jensen.eval.csv Jesper Højvang Jensen, Daniel P. W. Ellis, Mads G. Christensen, Søren Holdt]<br /> |
| − | '''KL1''' = [https://www.music-ir.org/ | + | '''KL1''' = [https://www.music-ir.org/mirex/results/2007/cover.klee1.eval.csv Kyogu Lee 1]<br /> |
| − | '''KL2''' = [https://www.music-ir.org/ | + | '''KL2''' = [https://www.music-ir.org/mirex/results/2007/cover.klee2.eval.csv Kyogu Lee 2]<br /> |
| − | '''KP''' = [https://www.music-ir.org/ | + | '''KP''' = [https://www.music-ir.org/mirex/results/2007/cover.metlab.eval.csv Youngmoo E. Kim, Daniel Perelstein] <br /> |
| − | '''SG''' = [https://www.music-ir.org/ | + | '''SG''' = [https://www.music-ir.org/mirex/results/2007/cover.serra.eval.csv Joan Serrà, Emilia Gómez]<br /> |
===Ranks of the Ten Cover Songs Returned for Each Query=== | ===Ranks of the Ten Cover Songs Returned for Each Query=== | ||
| − | '''EC''' = [https://www.music-ir.org/ | + | '''EC''' = [https://www.music-ir.org/mirex/results/2007/cover.labrosa.eval.debug.csv Daniel P. W. Ellis, Courtenay V. Cotton]<br /> |
| − | '''IM''' = [https://www.music-ir.org/ | + | '''IM''' = [https://www.music-ir.org/mirex/results/2007/cover.imirsel.eval.debug.csv IMIRSEL M2K]<br/> |
| − | '''JB''' = [https://www.music-ir.org/ | + | '''JB''' = [https://www.music-ir.org/mirex/results/2007/cover.bello.eval.debug.csv Juan Bello] <br/> |
| − | '''JEC''' = [https://www.music-ir.org/ | + | '''JEC''' = [https://www.music-ir.org/mirex/results/2007/cover.jensen.eval.debug.csv Jesper Højvang Jensen, Daniel P. W. Ellis, Mads G. Christensen, Søren Holdt]<br /> |
| − | '''KL1''' = [https://www.music-ir.org/ | + | '''KL1''' = [https://www.music-ir.org/mirex/results/2007/cover.klee1.eval.debug.csv Kyogu Lee 1]<br /> |
| − | '''KL2''' = [https://www.music-ir.org/ | + | '''KL2''' = [https://www.music-ir.org/mirex/results/2007/cover.klee2.eval.debug.csv Kyogu Lee 2]<br /> |
| − | '''KP''' = [https://www.music-ir.org/ | + | '''KP''' = [https://www.music-ir.org/mirex/results/2007/cover.metlab.eval.debug.csv Youngmoo E. Kim, Daniel Perelstein] <br /> |
| − | '''SG''' = [https://www.music-ir.org/ | + | '''SG''' = [https://www.music-ir.org/mirex/results/2007/cover.serra.eval.debug.csv Joan Serrà, Emilia Gómez]<br /> |
[[Category: Results]] | [[Category: Results]] | ||
Latest revision as of 16:35, 23 July 2010
Introduction
These are the results for the 2007 running of the Audio Cover Song Identification task. For background information about this task set please refer to the 2007:Audio Cover Song Identification page.
Each system was given a collection of 1000 songs which included of 30 different classes (sets) of cover songs where each class/set was represented by 11 different versions of a particular song. Each of the 330 cover songs were used as queries and the systems were required to return 10 results for each query. Systems were evaluated on the number of the songs from the same class/set as the query that were returned in the list of 10 results for each query. Average precision, which looks at the entire per-query rank-ordered list of all songs in the collection, is a new metric explored this year.
General Legend
Team ID
EC = Daniel P. W. Ellis, Courtenay V. Cotton
IM = IMIRSEL M2K
JB = Juan Bello
JEC = Jesper Højvang Jensen, Daniel P. W. Ellis, Mads G. Christensen, Søren Holdt
KL1 = Kyogu Lee 1
KL2 = Kyogu Lee 2
KP = Youngmoo E. Kim, Daniel Perelstein
SG = Joan Serrà, Emilia Gómez
Overall Summary Results
| Measure | Range | EC | JB | JEC | KL1 | KL2 | KP | IM | SG |
|---|---|---|---|---|---|---|---|---|---|
| Total number of covers identified in top 10 | [0-3300] | 1207 | 869 | 762 | 425 | 291 | 190 | 34 | 1653 |
| Mean number of covers identified in top 10 (average performance) | [0-10] | 3.658 | 2.633 | 2.309 | 1.288 | 0.882 | 0.576 | 0.103 | 5.009 |
| Mean (arithmetic) of Avg. Precisions | [0-1] | 0.330 | 0.267 | 0.238 | 0.13 | 0.086 | 0.061 | 0.017 | 0.521 |
| Mean rank of first correctly identified cover | [0-1] | 13.994 | 29.527 | 22.209 | 57.542 | 51.094 | 46.539 | 97.470 | 9.367 |
Runtimes
Where algorithms have been multi-threaded, the longest runtime is reported.
Where runtimes were not properly reported, file timestamps have been used to approximate a runtime.
| CLOCK TIME MEASURES (not cpu time) | ||
|---|---|---|
| Team | Clock time (HH:MM) | # Threads |
| EC | 04:28 | 5 |
| IM | 02:04 | 1 |
| JB | 04:32 | 8 |
| JEC | 00:47 | 8 |
| KL1 | 10:45 | 8 |
| KL2 | 02:37 | 1 |
| KP | ~03:51 | 1 |
| SG | 01:37 | 1 |
Number of Correct Covers at Rank X Returned in Top Ten
| Rank | EC | JB | JEC | KL2 | KL1 | KP | IM | SG |
|---|---|---|---|---|---|---|---|---|
| 1 | 164 | 151 | 143 | 83 | 4 | 19 | 6 | 242 |
| 2 | 182 | 127 | 124 | 78 | 8 | 26 | 3 | 222 |
| 3 | 169 | 121 | 91 | 62 | 6 | 24 | 6 | 209 |
| 4 | 153 | 103 | 98 | 45 | 9 | 21 | 2 | 191 |
| 5 | 147 | 88 | 74 | 35 | 8 | 18 | 2 | 180 |
| 6 | 112 | 70 | 67 | 37 | 9 | 22 | 3 | 166 |
| 7 | 108 | 77 | 53 | 15 | 11 | 14 | 2 | 147 |
| 8 | 84 | 59 | 50 | 21 | 51 | 17 | 1 | 131 |
| 9 | 58 | 44 | 39 | 27 | 86 | 13 | 4 | 102 |
| 10 | 30 | 29 | 23 | 22 | 99 | 16 | 5 | 63 |
| SUM | 1207 | 869 | 762 | 425 | 291 | 190 | 34 | 1653 |
Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
| Friedman's ANOVA Table | |||||
|---|---|---|---|---|---|
| Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
| Columns | 871.3167 | 7 | 124.4738 | 145.3925 | 0 |
| Error | 387.1833 | 203 | 1.9073 | ||
| Total | 1.2585e+003 | 239 |
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| EC | JB | -0.5491 | 1.3667 | 3.2824 | FALSE |
| EC | JEC | -0.7158 | 1.2000 | 3.1158 | FALSE |
| EC | KL1 | 0.8676 | 2.7833 | 4.6991 | TRUE |
| EC | KL2 | 1.2676 | 3.1833 | 5.0991 | TRUE |
| EC | KP | 1.6176 | 3.5333 | 5.4491 | TRUE |
| EC | IM | 3.1842 | 5.1000 | 7.0158 | TRUE |
| EC | SG | -3.0824 | -1.1667 | 0.7491 | FALSE |
| JB | JEC | -2.0824 | -0.1667 | 1.7491 | FALSE |
| JB | KL1 | -0.4991 | 1.4167 | 3.3324 | FALSE |
| JB | KL2 | -0.0991 | 1.8167 | 3.7324 | FALSE |
| JB | KP | 0.2509 | 2.1667 | 4.0824 | TRUE |
| JB | IM | 1.8176 | 3.7333 | 5.6491 | TRUE |
| JB | SG | -4.4491 | -2.5333 | -0.6176 | TRUE |
| JEC | KL1 | -0.3324 | 1.5833 | 3.4991 | FALSE |
| JEC | KL2 | 0.0676 | 1.9833 | 3.8991 | TRUE |
| JEC | KP | 0.4176 | 2.3333 | 4.2491 | TRUE |
| JEC | IM | 1.9842 | 3.9000 | 5.8158 | TRUE |
| JEC | SG | -4.2824 | -2.3667 | -0.4509 | TRUE |
| KL1 | KL2 | -1.5158 | 0.4000 | 2.3158 | FALSE |
| KL1 | KP | -1.1658 | 0.7500 | 2.6658 | FALSE |
| KL1 | IM | 0.4009 | 2.3167 | 4.2324 | TRUE |
| KL1 | SG | -5.8658 | -3.9500 | -2.0342 | TRUE |
| KL2 | KP | -1.5658 | 0.3500 | 2.2658 | FALSE |
| KL2 | IM | 0.0009 | 1.9167 | 3.8324 | TRUE |
| KL2 | SG | -6.2658 | -4.3500 | -2.4342 | TRUE |
| KP | IM | -0.3491 | 1.5667 | 3.4824 | FALSE |
| KP | SG | -6.6158 | -4.7000 | -2.7842 | TRUE |
| IM | SG | -8.1824 | -6.2667 | -4.3509 | TRUE |
Average Performance per Query Group
These are the arithmetic means of the average precisions within each of the 30 query groups.
| Cover Group | EC | JB | JEC | KL1 | KL2 | KP | IM | SG |
|---|---|---|---|---|---|---|---|---|
| 1 | 0.384 | 0.627 | 0.122 | 0.139 | 0.202 | 0.048 | 0.020 | 0.861 |
| 2 | 0.331 | 0.106 | 0.149 | 0.159 | 0.092 | 0.057 | 0.013 | 0.400 |
| 3 | 0.206 | 0.035 | 0.057 | 0.046 | 0.026 | 0.022 | 0.025 | 0.201 |
| 4 | 0.838 | 0.566 | 0.791 | 0.293 | 0.326 | 0.302 | 0.020 | 0.998 |
| 5 | 0.373 | 0.705 | 0.468 | 0.141 | 0.235 | 0.055 | 0.023 | 0.825 |
| 6 | 0.078 | 0.032 | 0.049 | 0.034 | 0.033 | 0.038 | 0.020 | 0.121 |
| 7 | 0.551 | 0.644 | 0.416 | 0.430 | 0.194 | 0.049 | 0.020 | 0.961 |
| 8 | 0.392 | 0.205 | 0.176 | 0.037 | 0.043 | 0.017 | 0.010 | 0.439 |
| 9 | 0.354 | 0.084 | 0.120 | 0.043 | 0.047 | 0.043 | 0.019 | 0.342 |
| 10 | 0.343 | 0.220 | 0.273 | 0.181 | 0.142 | 0.080 | 0.011 | 0.696 |
| 11 | 0.072 | 0.398 | 0.206 | 0.217 | 0.099 | 0.089 | 0.016 | 0.859 |
| 12 | 0.217 | 0.038 | 0.226 | 0.020 | 0.041 | 0.068 | 0.015 | 0.194 |
| 13 | 0.158 | 0.132 | 0.266 | 0.190 | 0.059 | 0.080 | 0.015 | 0.405 |
| 14 | 0.377 | 0.949 | 0.705 | 0.220 | 0.071 | 0.046 | 0.013 | 0.999 |
| 15 | 0.588 | 0.770 | 0.174 | 0.218 | 0.065 | 0.040 | 0.016 | 0.766 |
| 16 | 0.512 | 0.186 | 0.092 | 0.022 | 0.029 | 0.028 | 0.014 | 0.525 |
| 17 | 0.398 | 0.411 | 0.238 | 0.433 | 0.208 | 0.141 | 0.018 | 0.905 |
| 18 | 0.473 | 0.451 | 0.302 | 0.167 | 0.095 | 0.043 | 0.016 | 0.697 |
| 19 | 0.055 | 0.045 | 0.122 | 0.029 | 0.022 | 0.064 | 0.014 | 0.051 |
| 20 | 0.168 | 0.060 | 0.018 | 0.100 | 0.070 | 0.043 | 0.012 | 0.173 |
| 21 | 0.131 | 0.072 | 0.117 | 0.021 | 0.033 | 0.019 | 0.023 | 0.389 |
| 22 | 0.060 | 0.014 | 0.021 | 0.026 | 0.017 | 0.043 | 0.018 | 0.052 |
| 23 | 0.521 | 0.280 | 0.486 | 0.038 | 0.027 | 0.044 | 0.016 | 0.761 |
| 24 | 0.203 | 0.063 | 0.045 | 0.009 | 0.016 | 0.054 | 0.013 | 0.088 |
| 25 | 0.420 | 0.284 | 0.327 | 0.018 | 0.020 | 0.040 | 0.023 | 0.471 |
| 26 | 0.774 | 0.233 | 0.608 | 0.169 | 0.094 | 0.065 | 0.012 | 0.838 |
| 27 | 0.152 | 0.104 | 0.021 | 0.019 | 0.038 | 0.033 | 0.033 | 0.225 |
| 28 | 0.114 | 0.153 | 0.116 | 0.101 | 0.018 | 0.014 | 0.018 | 0.450 |
| 29 | 0.208 | 0.035 | 0.054 | 0.018 | 0.020 | 0.022 | 0.015 | 0.241 |
| 30 | 0.453 | 0.102 | 0.380 | 0.352 | 0.197 | 0.134 | 0.010 | 0.703 |
Individual Results Files
Average Precision Scores for Each Query
EC = Daniel P. W. Ellis, Courtenay V. Cotton
IM = IMIRSEL M2K
JB = Juan Bello
JEC = Jesper Højvang Jensen, Daniel P. W. Ellis, Mads G. Christensen, Søren Holdt
KL1 = Kyogu Lee 1
KL2 = Kyogu Lee 2
KP = Youngmoo E. Kim, Daniel Perelstein
SG = Joan Serrà, Emilia Gómez
Ranks of the Ten Cover Songs Returned for Each Query
EC = Daniel P. W. Ellis, Courtenay V. Cotton
IM = IMIRSEL M2K
JB = Juan Bello
JEC = Jesper Højvang Jensen, Daniel P. W. Ellis, Mads G. Christensen, Søren Holdt
KL1 = Kyogu Lee 1
KL2 = Kyogu Lee 2
KP = Youngmoo E. Kim, Daniel Perelstein
SG = Joan Serrà, Emilia Gómez
