2008:Audio Cover Song Identification Results

Still missing runtimes JSD Sept. 11 2008.

Introduction

These are the results for the 2008 running of the Audio Cover Song Identification task. For background information about this task set please refer to the 2008:Audio Cover Song Identification page.

Each system was given a collection of 1000 songs which included of 30 different classes (sets) of cover songs where each class/set was represented by 11 different versions of a particular song. Each of the 330 cover songs were used as queries and the systems were required to return 10 results for each query. Systems were evaluated on the number of the songs from the same class/set as the query that were returned in the list of 10 results for each query. Average precision, which looks at the entire per-query rank-ordered list of all songs in the collection, was the new metric introduced last year.

General Legend

Team ID

CL1 = C. Cao, M. Li
CL2 = C. Cao, M. Li
EL1 = A. Egorov, G. Linetsky
EL2 = A. Egorov, G. Linetsky
EL3 = A. Egorov, G. Linetsky
JCJ = J. H. Jensen, M. G. Christensen, S. H. Jensen
SGH1 = J. Serrà, E. Gómez, P. Herrera
SGH2 = J. Serrà, E. Gómez, P. Herrera

Overall Summary Results

Measure	Range	CL1	CL2	EL1	EL2	EL3	JCJ	SGH1	SGH2
Total number of covers identified in top 10	[0-3300]	1056	1073	1762	1781	1778	763	2422	2116
Mean number of covers identified in top 10 (average performance)	[0-10]	3.20	3.25	5.34	5.40	5.39	2.31	7.34	6.41
Mean (arithmetic) of Avg. Precisions	[0-1]	0.34	0.34	0.55	0.56	0.56	0.23	0.75	0.66
Mean rank of first correctly identified cover	[1-1000]	18.83	18.78	11.19	11.17	11.55	23.98	4.80	4.55

download these results as csv

Number of Correct Covers at Rank X Returned in Top Ten

RankX	CL1	CL2	EL1	EL2	EL3	JCJ	SGH1	SGH2
1	192	191	259	259	258	151	281	281
2	164	159	250	251	253	122	286	277
3	143	150	236	235	231	95	278	261
4	131	132	201	213	214	80	272	250
5	114	118	177	176	178	76	265	231
6	96	90	176	177	179	67	265	222
7	80	87	154	158	155	52	247	200
8	62	63	136	141	136	45	230	170
9	47	53	121	120	119	39	183	134
10	27	30	52	51	55	36	115	90
Total	1056	1073	1762	1781	1778	763	2422	2116

download these results as csv

Run Times

	CL1	CL2	EL1	EL2	EL3	JCJ	SGH1	SGH2
Run Time (sec)	281487	281352	136469	435240	434712	3648	373370	373371

download these results as csv

CL1,CL2 ran on FAST2,FAST3. All others ran on ALE Nodes.

Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Source	SS	df	MS	Chi-sq	Prob>Chi-sq
Columns	854.92	7	122.131	144.03	0
Error	391.58	203	1.929
Total	1246.5	239

download these results as csv

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
CL1	CL2	-2.2066	-0.3000	1.6066	FALSE
CL1	EL1	-4.4233	-2.5167	-0.6101	TRUE
CL1	EL2	-4.4399	-2.5333	-0.6267	TRUE
CL1	EL3	-4.3399	-2.4333	-0.5267	TRUE
CL1	JCJ	-1.2899	0.6167	2.5233	FALSE
CL1	SGH1	-6.9399	-5.0333	-3.1267	TRUE
CL1	SGH2	-6.1066	-4.2000	-2.2934	TRUE
CL2	EL1	-4.1233	-2.2167	-0.3101	TRUE
CL2	EL2	-4.1399	-2.2333	-0.3267	TRUE
CL2	EL3	-4.0399	-2.1333	-0.2267	TRUE
CL2	JCJ	-0.9899	0.9167	2.8233	FALSE
CL2	SGH1	-6.6399	-4.7333	-2.8267	TRUE
CL2	SGH2	-5.8066	-3.9000	-1.9934	TRUE
EL1	EL2	-1.9233	-0.0167	1.8899	FALSE
EL1	EL3	-1.8233	0.0833	1.9899	FALSE
EL1	JCJ	1.2267	3.1333	5.0399	TRUE
EL1	SGH1	-4.4233	-2.5167	-0.6101	TRUE
EL1	SGH2	-3.5899	-1.6833	0.2233	FALSE
EL2	EL3	-1.8066	0.1000	2.0066	FALSE
EL2	JCJ	1.2434	3.1500	5.0566	TRUE
EL2	SGH1	-4.4066	-2.5000	-0.5934	TRUE
EL2	SGH2	-3.5733	-1.6667	0.2399	FALSE
EL3	JCJ	1.1434	3.0500	4.9566	TRUE
EL3	SGH1	-4.5066	-2.6000	-0.6934	TRUE
EL3	SGH2	-3.6733	-1.7667	0.1399	FALSE
JCJ	SGH1	-7.5566	-5.6500	-3.7434	TRUE
JCJ	SGH2	-6.7233	-4.8167	-2.9101	TRUE
SGH1	SGH2	-1.0733	0.8333	2.7399	FALSE

download these results as csv

Average Performance per Query Group

These are the arithmetic means of the average precisions within each of the 30 query groups.

QueryGroupID	CL1	CL2	EL1	EL2	EL3	JCJ	SGH1	SGH2
1	0.411	0.412	0.792	0.819	0.819	0.128	1.000	0.770
2	0.163	0.152	0.383	0.381	0.375	0.165	0.690	0.562
3	0.024	0.023	0.338	0.320	0.332	0.038	0.696	0.534
4	0.730	0.752	0.991	0.991	0.991	0.501	0.995	0.995
5	0.819	0.838	0.844	0.843	0.843	0.475	0.879	0.847
6	0.033	0.030	0.136	0.128	0.130	0.025	0.211	0.212
7	0.406	0.346	0.923	0.920	0.929	0.268	0.998	0.981
8	0.054	0.032	0.503	0.562	0.578	0.203	0.671	0.536
9	0.306	0.314	0.302	0.377	0.371	0.259	0.894	0.734
10	0.507	0.506	0.720	0.713	0.717	0.285	1.000	0.916
11	0.585	0.595	0.749	0.749	0.751	0.207	0.847	0.851
12	0.355	0.388	0.398	0.463	0.386	0.174	0.985	0.529
13	0.316	0.323	0.426	0.427	0.423	0.300	0.474	0.482
14	0.416	0.379	0.998	0.998	0.996	0.589	1.000	1.000
15	0.511	0.498	0.816	0.814	0.814	0.160	0.832	0.832
16	0.041	0.037	0.567	0.492	0.547	0.132	0.899	0.766
17	0.849	0.852	0.936	0.935	0.936	0.280	0.977	0.977
18	0.472	0.511	0.739	0.743	0.725	0.298	0.951	0.930
19	0.093	0.104	0.093	0.104	0.103	0.079	0.118	0.100
20	0.427	0.454	0.357	0.352	0.353	0.021	0.608	0.487
21	0.037	0.034	0.460	0.459	0.460	0.084	0.717	0.516
22	0.054	0.058	0.078	0.078	0.078	0.056	0.027	0.035
23	0.333	0.338	0.704	0.734	0.737	0.663	0.820	0.800
24	0.019	0.020	0.148	0.130	0.135	0.034	0.567	0.444
25	0.025	0.038	0.553	0.556	0.545	0.121	0.843	0.683
26	0.832	0.831	0.821	0.821	0.770	0.672	1.000	1.000
27	0.221	0.200	0.336	0.362	0.372	0.119	0.722	0.474
28	0.163	0.161	0.489	0.510	0.476	0.329	0.474	0.457
29	0.085	0.091	0.244	0.297	0.272	0.102	0.648	0.520
30	0.831	0.835	0.787	0.787	0.787	0.281	0.966	0.950
Mean	0.337	0.338	0.554	0.562	0.558	0.235	0.750	0.664

download these results as csv

Individual Results Files

Average Precision Scores for Each Query

CL1 = C. Cao, M. Li
CL2 = C. Cao, M. Li
EL1 = A. Egorov, G. Linetsky
EL2 = A. Egorov, G. Linetsky
EL3 = A. Egorov, G. Linetsky
JCJ = J. H. Jensen, M. G. Christensen, S. H. Jensen
SGH1 = J. Serrà, E. Gómez, P. Herrera
SGH2 = J. Serrà, E. Gómez, P. Herrera

Ranks of the Ten Cover Songs Returned for Each Query

CL1 = C. Cao, M. Li
CL2 = C. Cao, M. Li
EL1 = A. Egorov, G. Linetsky
EL2 = A. Egorov, G. Linetsky
EL3 = A. Egorov, G. Linetsky
JCJ = J. H. Jensen, M. G. Christensen, S. H. Jensen
SGH1 = J. Serrà, E. Gómez, P. Herrera
SGH2 = J. Serrà, E. Gómez, P. Herrera

Runtimes

Where algorithms have been multi-threaded, the longest runtime is reported.

Where runtimes were not properly reported, file timestamps have been used to approximate a runtime.