Difference between revisions of "2006:Audio Cover Song Identification Results"

Revision as of 18:32, 13 May 2010

Introduction

These are the results for the 2006 running of the Audio Cover Song Identification task. For background information about this task set please refer to the 2006:Audio Cover Song page.

Each system was given a collection of 1000 songs which included of 30 different classes (sets) of cover songs where each class/set was represented by 11 different versions of a particular song. Each of the 330 cover songs were used as queries and the systems were required to return 10 results for each query. Systems were evaluated on the number of the songs from the same class/set as the query that were returned in the list of 10 results for each query. Special note: Some systems (e.g., KWL, KWT, LR and TP) participated as a by-product of the 2006:Audio Music Similarity and Retrieval task, that is, these systems were not specifically written to detect cover song variants. Other systems (e.g., CS, DE, KL1 and KL2) were specifically written to detect cover song variants.

General Legend

Team ID

CS = Christian Sailer and Karin Dressler⁽¹⁾
DE = Daniel P. W. Ellis ⁽¹⁾
KL1 = Kyogu Lee 1 ⁽¹⁾
KL2 = Kyogu Lee 2 ⁽¹⁾
KWL = Kris West (Likely) ⁽²⁾
KWT = Kris West (Trans) ⁽²⁾
LR = Thomas Lidy and Andreas Rauber ⁽²⁾
TP = Tim Pohle ⁽²⁾

⁽¹⁾ Denotes submissions specifically designed to detect cover song variants.
⁽²⁾ Denotes submissions not-specifically designed to detect cover song variants (See 2006:Audio Music Similarity and Retrieval Results).

Calculating Summary Measures

MRR = Mean Reciprocal Rank. Reciprocal rank is the reciprocal of the rank of the first correctly identified cover for each query (1/rank). These values are averaged for each cover song group as well as overall.

Overall Summary Results

Measure	Range	CS	DE	KL1	KL2	KWL	KWT	LR	TP
Total number of covers identified	[0-3300]	211	761	365	314	117	102	149	116
Mean number of covers identified (average performance)	[0-10]	0.64	2.31	1.11	0.95	0.35	0.31	0.45	0.35
Mean of maxima (average of best-case performance)	[0-10]	2.13	4.53	2.50	2.27	1.30	1.23	1.57	1.50
Mean reciprocal rank of first correctly identified cover	[0-1]	0.21	0.49	0.22	0.22	0.10	0.10	0.12	0.09

download these results as csv

Audio Cover Song Identification Runtime Data

Team ID		Machine	Run-time(seconds)
CS	feature	beer 8	28219
CS	distmat	beer 8	1707
DE	feature	beer 6	26814
DE	distmat	beer 6	60511
KL1	feature	beer 1	15600
KL1	distmat	beer 6	140360
KL2	feature	beer 0	15623
KL2	distmat	beer 0	136379

download these results as csv

Friedman Test with Multiple Comparisons Results (p=0.05)

The Friedman test was run in MATLAB against the MRR summary data over the 30 song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Friedman's ANOVA Table
Source	SS	df	MS	Chi-sq	Prob>Chi-sq
Columns	313.1	7	44.7286	52.5798	4.49E-09
Error	937.4	203	4.6177
Total	1250.5	239

download these results as csv

TeamID	TeamID	Lowerbound	Mean	Upperbound	Signficance
CS	DE	-3.9763	-2.0667	-0.157	TRUE
CS	KL1	-1.043	0.8667	2.7763	FALSE
CS	KL2	-1.193	0.7167	2.6263	FALSE
CS	KWL	-0.3597	1.55	3.4597	FALSE
CS	KWT	-0.2597	1.65	3.5597	FALSE
CS	LR	-1.2263	0.6833	2.593	FALSE
CS	TP	-0.3763	1.5333	3.443	FALSE
DE	KL1	1.0237	2.9333	4.843	TRUE
DE	KL2	0.8737	2.7833	4.693	TRUE
DE	KWL	1.707	3.6167	5.5263	TRUE
DE	KWT	1.807	3.7167	5.6263	TRUE
DE	LR	0.8403	2.75	4.6597	TRUE
DE	TP	1.6903	3.6	5.5097	TRUE
KL1	KL2	-2.0597	-0.15	1.7597	FALSE
KL1	KWL	-1.2263	0.6833	2.593	FALSE
KL1	KWT	-1.1263	0.7833	2.693	FALSE
KL1	LR	-2.093	-0.1833	1.7263	FALSE
KL1	TP	-1.243	0.6667	2.5763	FALSE
KL2	KWL	-1.0763	0.8333	2.743	FALSE
KL2	KWT	-0.9763	0.9333	2.843	FALSE
KL2	LR	-1.943	-0.0333	1.8763	FALSE
KL2	TP	-1.093	0.8167	2.7263	FALSE
KWL	KWT	-1.8097	0.1	2.0097	FALSE
KWL	LR	-2.7763	-0.8667	1.043	FALSE
KWL	TP	-1.9263	-0.0167	1.893	FALSE
KWT	LR	-2.8763	-0.9667	0.943	FALSE
KWT	TP	-2.0263	-0.1167	1.793	FALSE
LR	TP	-1.0597	0.85	2.7597	FALSE

download these results as csv

Mean Reciprocal Rank (MRR) Results

Song ID	Query	CS	DE	KL1	KL2	KWL	KWT	LR	TP
1	c000095	0	0	0	0	0	0	0	0
1	c000142	0	0	1/7	0	0	0	0	0
1	c000219	0	0	0	0	0	0	0	0
1	c000275	1/6	0	0	0	1/7	0	0	0
1	c000337	0	0	0	0	0	0	0	1
1	c000353	0	0	0	0	0	0	1/6	0
1	c000457	1/8	0	0	0	0	0	1/9	0
1	c000561	0	0	0	0	0	0	0	0
1	c000604	0	0	0	0	0	0	1/2	0
1	c000606	0	0	0	0	0	0	0	1
1	c000707	0	0	0	0	0	0	0	0
1	MRR	0.03	0.00	0.01	0.00	0.01	0.00	0.07	0.18
2	c000047	1/2	1	0	0	0	0	1/7	0
2	c000088	1	1	0	0	0	0	0	0
2	c000144	0	1	0	0	0	0	0	0
2	c000172	0	1	0	0	0	0	0	0
2	c000186	1/6	1/3	0	0	0	1/5	1/5	1/5
2	c000323	0	1/5	0	0	0	0	1/4	0
2	c000538	0	0	0	0	0	0	0	0
2	c000748	0	1/3	0	1/9	1/9	1/2	0	1/7
2	c000862	0	1/2	1/2	1/2	0	0	1	1/5
2	c000956	0	1	0	0	0	0	0	0
2	c000981	0	0	0	0	0	0	0	0
2	MRR	0.16	0.58	0.05	0.06	0.01	0.07	0.14	0.05
3	c000001	0	1	0	0	0	1/9	1/9	0
3	c000198	1/4	1/3	1/2	1/4	0	0	1/6	1
3	c000212	0	0	0	0	0	1	1	1/3
3	c000288	0	1	1/3	1/2	0	0	0	0
3	c000366	0	0	0	0	0	0	0	0
3	c000395	1	1/2	1/5	1/4	0	0	0	0
3	c000483	1/3	1/4	1	1	0	1/7	0	0
3	c000491	0	1/5	1/5	1/3	0	0	0	0
3	c000551	0	1/5	0	0	0	1/6	1/5	1/6
3	c000570	1/6	1/3	0	0	1/5	0	0	0
3	c000599	1/2	1	0	0	1/8	0	1/6	1/4
3	MRR	0.20	0.44	0.21	0.21	0.03	0.15	0.15	0.16
4	c000055	1/3	1	0	0	0	0	0	0
4	c000057	0	1	0	0	0	0	0	0
4	c000228	0	0	0	0	1/2	0	1/4	1/5
4	c000333	0	1	0	0	0	0	0	0
4	c000347	0	1	0	0	1/2	0	0	0
4	c000461	0	1/2	0	0	0	0	0	0
4	c000542	0	1	0	0	0	0	0	0
4	c000548	0	0	0	0	0	0	1/3	0
4	c000639	0	1/6	0	0	0	0	0	0
4	c000689	0	0	0	0	0	0	0	0
4	c000948	0	0	0	1/6	1/7	0	0	0
4	MRR	0.03	0.52	0.00	0.02	0.10	0.00	0.05	0.02
5	c000004	1/2	1	1	1	1/4	1/7	0	1/3
5	c000101	0	1	1	0	0	1/6	1/2	1/6
5	c000189	0	1	1/2	1/2	0	1/7	1/4	1/2
5	c000224	0	1	1	1	1/7	1/3	1/3	1/9
5	c000245	1/8	1	1	1	0	0	0	0
5	c000316	0	1	1	1	1/7	1/8	0	0
5	c000385	0	1	0	1	0	1/7	0	0
5	c000603	0	1	1	1	1/7	1/7	1	1/2
5	c000734	0	0	1	0	1	1/6	0	1/8
5	c000765	0	1	1	1	0	1/3	0	1
5	c000832	0	1	1	1	1/4	1/5	1/2	1
5	MRR	0.06	0.91	0.86	0.77	0.18	0.17	0.23	0.35
6	c000005	1	1/2	0	0	1/5	0	1	0
6	c000043	1/2	0	0	1/5	1/2	0	1/4	0
6	c000082	1	1/6	0	0	0	0	0	0
6	c000126	1/3	1/2	0	0	0	0	0	0
6	c000204	0	1/9	0	0	1/5	0	0	0
6	c000256	0	1/7	0	0	0	0	0	0
6	c000289	1/3	0	1/4	1	0	0	0	1/6
6	c000377	1/4	0	0	0	0	0	0	0
6	c000399	1/5	1/4	0	0	0	0	0	0
6	c000768	0	1/8	1/7	0	0	0	0	0
6	c000928	1	0	1/5	0	0	0	0	1/8
6	MRR	0.42	0.16	0.05	0.12	0.08	0.00	0.12	0.03
7	c000112	0	0	1/7	0	0	0	1/8	0
7	c000145	0	0	0	0	0	0	0	0
7	c000314	1	1	0	0	0	0	0	1/6
7	c000345	0	1/5	0	0	0	0	0	0
7	c000348	0	1	0	0	0	0	0	0
7	c000396	0	1/4	0	0	0	0	0	1/7
7	c000539	1	1	0	0	0	0	0	1/5
7	c000578	1/2	1	0	0	0	0	0	0
7	c000654	0	0	0	0	1/8	0	0	1/5
7	c000890	1	1/5	0	0	0	0	0	0
7	c000952	0	1	0	0	0	1	1/2	0
7	MRR	0.32	0.51	0.01	0.00	0.01	0.10	0.07	0.06
8	c000015	1	0	0	0	1	0	0	0
8	c000155	0	0	0	0	0	0	0	0
8	c000182	1	0	0	0	0	1/9	1/6	1/6
8	c000610	1/4	1/9	0	0	1/3	1/9	0	0
8	c000690	1/3	1/5	0	0	1/2	1/2	0	0
8	c000706	1	0	0	0	0	0	0	1/5
8	c000842	0	0	0	0	0	1/6	0	0
8	c000846	1	0	0	0	0	0	0	0
8	c000897	1	0	0	0	0	0	1/4	0
8	c000975	0	1/8	0	0	0	0	0	0
8	c000998	0	0	0	0	1/3	1/7	1/3	1/9
8	MRR	0.51	0.04	0.00	0.00	0.20	0.09	0.07	0.06
9	c000042	0	1/3	0	0	0	0	0	0
9	c000136	0	0	0	0	0	0	0	0
9	c000157	0	1/8	1/2	0	0	0	0	0
9	c000210	0	1/2	0	0	0	1/3	1/5	1
9	c000221	1/9	1	1	1	0	1/3	1/2	0
9	c000418	0	1	0	0	0	1/6	0	0
9	c000801	1/5	1/2	0	0	0	1/8	0	0
9	c000834	0	1	0	0	1/8	1/6	1/6	1
9	c000885	0	1	0	0	0	0	0	0
9	c000908	0	1	0	0	0	0	0	0
9	c000919	0	1	1	1	1/8	0	0	0
9	MRR	0.03	0.68	0.23	0.18	0.03	0.11	0.08	0.18
10	c000007	0	0	0	0	0	0	0	1/9
10	c000021	0	0	0	0	0	1/2	0	0
10	c000050	0	0	0	0	0	0	0	0
10	c000073	1/2	0	0	0	1/6	0	0	0
10	c000218	0	0	0	0	0	0	1/7	0
10	c000241	0	0	0	0	0	1/2	1/6	1/2
10	c000679	0	0	0	0	0	0	0	0
10	c000730	0	0	0	0	0	1/8	1/6	1/9
10	c000804	0	0	0	0	0	0	0	0
10	c000867	0	0	0	0	1/6	1	0	0
10	c000922	0	0	0	1/9	0	1	1	1/2
10	MRR	0.05	0.00	0.00	0.01	0.03	0.28	0.15	0.11
11	c000059	1/5	1	1	1	0	1/4	1	1/3
11	c000197	0	1	1	1	1/4	0	1/8	0
11	c000220	0	0	1	1	0	0	0	0
11	c000263	1/8	1	1/2	1/3	1/3	1/4	1/9	1
11	c000375	1	1/5	1/4	0	0	1/8	0	0
11	c000429	1	0	1	1	1/4	0	0	0
11	c000431	1	1/4	1/2	0	1/2	0	1	1/3
11	c000529	1	1	0	1	1/3	1/6	1	1/3
11	c000568	1	0	0	1	1/3	1/2	1/4	1/3
11	c000685	1/2	1	1	1	0	0	0	1
11	c000692	0	0	0	0	0	0	0	0
11	MRR	0.53	0.50	0.57	0.67	0.18	0.12	0.32	0.30
12	c000179	0	1	1	1	0	0	0	0
12	c000181	1/8	1	1	1	1	0	0	0
12	c000260	0	1	1	1	1/7	1/5	0	1/6
12	c000284	0	1	1	1	1/2	0	1/3	1/5
12	c000329	0	1	0	0	0	0	0	0
12	c000367	1/5	1	0	1/2	1/5	0	1/6	0
12	c000521	0	1	1	1	0	0	0	0
12	c000800	0	1	0	1/5	0	0	0	0
12	c000865	1/2	1	1	1	0	0	0	0
12	c000962	0	1	1	1	1/2	0	0	0
12	c000971	0	1/2	1/3	1	0	0	0	0
12	MRR	0.08	0.95	0.67	0.79	0.21	0.02	0.05	0.03
13	c000013	1/9	0	0	0	0	0	0	0
13	c000303	0	0	0	0	0	0	0	0
13	c000364	0	0	0	0	0	0	0	0
13	c000412	0	0	0	0	1/3	0	1/7	0
13	c000567	0	0	0	1/3	1	0	0	0
13	c000713	0	0	1/7	0	0	0	0	0
13	c000761	0	1/8	0	0	1/4	0	1/3	0
13	c000781	0	1/6	0	0	0	1	0	0
13	c000868	0	1	0	0	1	1	0	0
13	c000909	0	1/3	0	0	0	1/6	0	1
13	c000994	1/2	0	0	0	0	0	1/2	0
13	MRR	0.06	0.15	0.01	0.03	0.23	0.20	0.09	0.09
14	c000063	1	1	1	1	0	0	0	0
14	c000110	0	1/5	0	0	0	0	0	0
14	c000116	0	1	1/6	0	0	0	0	0
14	c000146	0	1	1	1	0	0	0	0
14	c000336	0	1	1	1/5	0	1/5	0	0
14	c000378	1/4	1	1	1	0	0	0	0
14	c000579	1/4	0	1	1	0	1/4	0	0
14	c000823	0	1	1/8	0	0	0	0	1/8
14	c000869	1	1	1	1/7	0	1/2	0	0
14	c000972	0	1	0	0	0	0	1/9	0
14	c000997	0	0	0	0	0	0	0	0
14	MRR	0.23	0.75	0.57	0.39	0.00	0.10	0.01	0.01
15	c000048	1	1	0	0	0	0	0	0
15	c000125	0	1	0	0	0	0	0	0
15	c000191	0	1	0	0	0	0	1/7	0
15	c000451	1	1	0	0	0	0	0	0
15	c000475	0	1	0	0	0	0	0	0
15	c000585	0	1	0	0	1	0	0	0
15	c000614	1/4	0	0	0	0	0	0	0
15	c000658	1/3	1	0	0	0	0	0	1/8
15	c000659	0	0	0	0	0	0	0	0
15	c000724	1/5	1	0	0	1	0	0	0
15	c000915	0	1	0	0	1/2	0	0	0
15	MRR	0.26	0.83	0.00	0.01	0.23	0.01	0.01	0.01
16	c000038	0	1/4	0	1/2	0	1/3	1/6	1/2
16	c000080	0	0	0	0	0	0	0	0
16	c000085	0	0	0	0	0	0	1/3	1
16	c000153	1/4	1/6	0	0	0	0	0	0
16	c000162	0	1/3	0	0	0	1	1/5	0
16	c000257	0	1	0	0	0	1/3	1/3	1/4
16	c000324	0	0	0	0	0	0	0	1/6
16	c000416	0	1/2	0	0	0	0	0	0
16	c000512	0	0	0	0	0	0	0	0
16	c000793	1/8	0	0	1/3	0	0	0	0
16	c000859	0	0	0	0	0	0	0	0
16	MRR	0.03	0.20	0.00	0.08	0.00	0.16	0.09	0.17
17	c000006	0	1	0	0	0	0	1/2	0
17	c000019	1/3	1	0	0	0	0	0	0
17	c000149	0	1	1	0	0	0	0	1/2
17	c000195	0	1	0	0	0	0	0	0
17	c000266	1	1	1/5	1	0	1/4	1/5	0
17	c000372	0	1	0	0	0	0	0	0
17	c000425	0	1	0	0	1/5	1/3	0	1/2
17	c000649	1	1	0	0	0	0	0	0
17	c000775	0	0	0	0	1/5	0	1/2	1/3
17	c000888	0	1	1/4	0	0	0	0	0
17	c000955	0	1	0	1	0	0	0	0
17	MRR	0.21	0.91	0.13	0.19	0.04	0.05	0.11	0.12
18	c000074	0	1	1	1	0	0	0	0
18	c000081	1	1	1	1	0	0	1/8	1/7
18	c000128	0	1	1/4	0	0	0	0	0
18	c000167	1/2	1	1	1	0	0	0	0
18	c000386	1/2	1/2	0	1/6	1/7	0	1/5	0
18	c000400	0	1	1	1	0	0	1/9	0
18	c000525	0	1	0	0	0	0	0	0
18	c000637	0	1	0	1/8	0	0	0	0
18	c000684	1/2	1/5	0	0	0	0	0	0
18	c000731	0	1/3	0	1/9	0	0	0	0
18	c000773	0	1	1	1	1/4	0	1/6	0
18	MRR	0.24	0.82	0.49	0.49	0.04	0.00	0.05	0.01
19	c000016	0	0	0	0	0	0	0	0
19	c000199	0	1/2	0	0	0	0	1/4	0
19	c000265	0	0	0	0	1/2	0	1/6	0
19	c000279	0	0	0	0	0	0	1/6	0
19	c000326	0	1/5	0	0	0	0	0	0
19	c000423	0	1/8	0	0	0	0	0	0
19	c000437	0	1/2	0	0	1/7	1	1/6	0
19	c000788	0	1/2	0	0	0	0	0	1/6
19	c000851	1/4	0	0	0	0	0	0	0
19	c000924	0	0	0	0	0	0	0	0
19	c000934	0	1/3	0	0	0	0	1/5	0
19	MRR	0.02	0.21	0.00	0.00	0.06	0.09	0.09	0.02
20	c000304	0	0	0	0	0	0	0	0
20	c000330	0	0	0	0	1/9	0	1/3	1/3
20	c000447	0	0	0	0	0	0	0	0
20	c000508	0	0	0	0	0	0	0	0
20	c000625	0	0	0	0	0	0	1/6	0
20	c000674	0	1	0	0	0	0	1	1/9
20	c000716	0	0	0	0	0	0	0	0
20	c000854	0	1/2	0	0	0	0	0	0
20	c000870	0	0	0	0	0	1/7	1/3	0
20	c000882	0	1/8	0	0	0	0	0	0
20	c000925	0	1/2	0	0	0	0	0	0
20	MRR	0.00	0.19	0.00	0.00	0.01	0.01	0.18	0.04
21	c000023	0	0	0	1/8	0	0	0	0
21	c000120	1/5	0	1/2	1	1/7	1/6	1/9	0
21	c000230	0	0	0	0	0	0	0	0
21	c000276	0	0	0	0	0	0	0	0
21	c000281	1/6	1	0	0	0	0	1/2	1/6
21	c000342	0	1/7	0	0	0	0	0	0
21	c000359	0	1/3	1/4	1/3	0	0	0	0
21	c000467	0	1/2	0	0	1/2	1/8	0	1/6
21	c000666	1/3	0	0	0	1/6	0	1/5	0
21	c000688	0	0	0	0	0	0	1	0
21	c000769	1	0	1/5	1/6	0	1/8	1	0
21	MRR	0.15	0.18	0.09	0.15	0.07	0.04	0.26	0.03
22	c000097	1/9	1/2	0	0	0	1/4	0	0
22	c000129	0	1	0	0	1/9	1	0	1/6
22	c000234	0	1/5	0	0	0	1/9	0	1/4
22	c000319	1/8	1	0	1/8	0	0	0	0
22	c000446	1/4	0	1	1/4	0	0	0	0
22	c000473	0	0	0	0	0	0	0	0
22	c000635	1	0	0	0	0	0	0	0
22	c000737	0	1/2	0	0	0	0	0	0
22	c000778	0	1	0	0	0	0	0	0
22	c000876	1	0	0	0	0	0	0	0
22	c000929	1	1	1/6	0	1/7	0	0	0
22	MRR	0.32	0.47	0.11	0.03	0.02	0.12	0.00	0.05
23	c000058	1/4	1	1/8	1/4	0	0	0	0
23	c000096	1/8	1/5	1/2	1	0	1/4	0	1/2
23	c000203	1/4	0	0	0	1/5	0	0	0
23	c000229	1	1	1	1	1/9	1/2	1	1
23	c000274	0	1	1	0	0	1/8	0	0
23	c000322	1/3	0	1/2	1	1/5	1/3	1/2	0
23	c000495	1/3	1	1/4	1/2	0	1	1	0
23	c000555	1	0	0	0	0	0	0	0
23	c000586	1/4	1	1	1	1	1/9	1	1
23	c000602	1/2	0	0	0	0	0	0	0
23	c000873	1/3	1	0	0	0	1	1	1/7
23	MRR	0.40	0.56	0.40	0.43	0.14	0.30	0.43	0.24
24	c000207	0	0	0	0	0	0	1/2	0
24	c000302	0	0	0	0	0	0	1/4	0
24	c000341	1/5	0	0	0	1/7	0	0	0
24	c000371	1/3	1	0	0	0	1/7	0	1/2
24	c000430	0	1	0	0	0	0	1/3	0
24	c000574	1	1/3	0	0	0	0	0	0
24	c000708	1/8	0	0	0	0	0	0	0
24	c000739	0	0	0	0	0	1/4	0	1/3
24	c000762	1/2	0	0	0	0	0	0	0
24	c000875	1	1	0	0	0	1/7	1/6	0
24	c000903	1/6	0	0	0	0	0	0	0
24	MRR	0.30	0.30	0.00	0.00	0.01	0.05	0.11	0.08
25	c000022	0	0	0	0	0	0	0	0
25	c000067	1/3	1	0	0	1	0	1/4	0
25	c000098	0	0	1/2	0	0	0	1/7	0
25	c000174	1/6	1	1/4	1/5	1/8	0	0	0
25	c000223	0	1	1/2	1/6	0	0	0	1/4
25	c000268	0	1/2	1/5	0	0	0	0	1/2
25	c000273	0	1/3	0	0	0	0	0	0
25	c000296	0	0	0	0	0	0	0	0
25	c000519	0	0	1/4	0	0	0	0	0
25	c000580	1	1/2	1/2	1/2	1/9	1/5	0	0
25	c000794	0	0	0	0	1	0	0	0
25	MRR	0.14	0.39	0.22	0.09	0.20	0.02	0.04	0.07
26	c000099	0	1	0	0	0	0	1/6	0
26	c000271	1/3	0	0	0	0	0	0	1/3
26	c000338	1/3	1	0	0	0	1/6	0	0
26	c000349	1	1	0	0	0	0	1/9	0
26	c000370	1/8	1/2	1	1/3	1/4	1/2	1/2	1/3
26	c000424	0	0	0	1	0	0	0	0
26	c000573	0	1	1/5	1/2	0	1/8	0	0
26	c000663	0	1/6	0	0	0	0	0	0
26	c000701	0	0	0	0	0	0	0	0
26	c000771	0	1	0	0	0	0	0	0
26	c000959	1	0	0	0	0	1/6	0	1/8
26	MRR	0.25	0.52	0.11	0.17	0.02	0.09	0.07	0.07
27	c000454	0	0	1	1	0	0	0	0
27	c000571	1/4	1	0	0	0	1/5	0	0
27	c000627	0	0	0	1/6	0	0	0	0
27	c000633	0	1/2	0	0	0	0	0	0
27	c000643	0	1	1/5	1/8	0	1	0	0
27	c000749	0	0	1	1	0	0	0	0
27	c000780	0	1	1	1	1/8	1/5	0	1/4
27	c000815	0	1	1	1	0	0	1/5	1/4
27	c000824	0	0	0	0	0	0	0	0
27	c000840	0	0	1/8	1	1/2	0	1/2	0
27	c000855	0	1	0	0	1	1	0	0
27	MRR	0.02	0.50	0.39	0.48	0.15	0.22	0.06	0.05
28	c000027	1	1	0	0	1/8	1/2	0	1/7
28	c000041	1/5	1	1	1	0	0	1/2	0
28	c000106	0	1/4	0	1/7	1/4	1/5	1/2	0
28	c000298	0	1	0	0	0	0	0	0
28	c000358	1/3	1	0	0	0	0	1/9	0
28	c000362	1/2	1	1	0	1/2	0	1/8	0
28	c000449	1/3	1	0	1/5	0	1/7	0	1/4
28	c000478	1/2	1	1	1/4	1	1/7	0	0
28	c000511	1/2	1	1	0	0	0	0	1/5
28	c000770	1/2	1	0	0	0	0	0	1/6
28	c000816	1/2	1	0	1/2	1	1/2	1/2	0
28	MRR	0.40	0.93	0.36	0.19	0.26	0.14	0.16	0.08
29	c000009	0	1	0	1/4	0	0	0	0
29	c000046	1	1	1	1	1/2	0	1/6	0
29	c000049	0	1	1	1	0	0	1/8	0
29	c000202	0	1	1	1	0	1/2	1/4	1/7
29	c000238	1	0	1	1	0	0	1/6	1/5
29	c000293	0	1	0	0	0	0	1/7	0
29	c000409	0	1	1/4	1	0	0	0	0
29	c000484	0	1	1	1	1/2	1/6	1/2	1/3
29	c000618	0	1	0	1/3	0	0	1/6	1/2
29	c000792	0	1	0	0	1/4	0	0	1/4
29	c000983	0	1	1	1	1	1/2	1/2	0
29	MRR	0.18	0.91	0.57	0.69	0.20	0.11	0.18	0.14
30	c000090	1	1	1	0	0	1/7	0	1/8
30	c000240	1	1	1	1	1/2	0	1/7	0
30	c000513	0	0	0	0	1/7	0	0	0
30	c000572	1	1	1/5	1/5	1/4	0	0	0
30	c000652	1	1/3	1	1	0	1/7	0	1/4
30	c000757	0	1	0	0	1/2	0	0	0
30	c000817	0	1	1/8	1/9	1/9	0	1/7	0
30	c000848	1/3	1	1/2	0	1	0	0	0
30	c000967	1	1/4	1/6	1/5	1/3	0	0	0
30	c000989	1	0	0	0	0	1/6	0	0
30	c000996	1/6	1/5	0	0	1/7	0	1/5	0
30	MRR	0.59	0.62	0.36	0.23	0.27	0.04	0.04	0.03
	Grand MRR	0.21	0.49	0.22	0.22	0.10	0.10	0.12	0.09

download these results as csv

Raw Results

The total number of correctly identified covers for each query, and the maximum number of covers identified for each cover song group.

Song ID	Query	CS	DE	KL1	KL2	KWT	KWL	LR	TP
1	c000095	0	0	0	0	0	0	0	0
1	c000142	0	0	1	0	0	0	0	0
1	c000219	0	0	0	0	0	0	0	0
1	c000275	1	0	0	0	0	1	0	0
1	c000337	0	0	0	0	0	0	0	1
1	c000353	0	0	0	0	0	0	1	0
1	c000457	1	0	0	0	0	0	1	0
1	c000561	0	0	0	0	0	0	0	0
1	c000604	0	0	0	0	0	0	1	0
1	c000606	0	0	0	0	0	0	0	1
1	c000707	0	0	0	0	0	0	0	0
1	MAX	1	0	1	0	0	1	1	1
2	c000047	1	2	0	0	1	0	1	0
2	c000088	1	2	0	0	0	0	0	0
2	c000144	0	2	0	0	0	0	0	0
2	c000172	0	2	0	0	0	0	0	0
2	c000186	1	1	0	0	1	0	1	1
2	c000323	0	1	0	0	0	0	1	0
2	c000538	0	0	0	0	0	0	0	0
2	c000748	0	2	0	1	1	1	0	1
2	c000862	0	2	1	1	0	0	1	1
2	c000956	1	3	0	0	0	0	0	0
2	c000981	0	0	1	0	0	0	0	0
2	MAX	1	3	1	1	1	1	1	1
3	c000001	0	3	1	0	1	0	1	0
3	c000198	1	1	4	2	0	0	1	1
3	c000212	0	0	0	0	3	0	2	2
3	c000288	0	3	3	2	0	0	0	0
3	c000366	0	0	0	0	0	0	0	0
3	c000395	2	4	1	1	0	0	0	0
3	c000483	1	2	4	3	1	0	0	0
3	c000491	0	3	2	2	0	0	0	0
3	c000551	0	1	0	0	1	0	1	1
3	c000570	3	1	0	0	1	1	0	0
3	c000599	1	3	0	0	1	1	2	1
3	MAX	3	4	4	3	3	1	2	2
4	c000055	1	5	0	0	0	0	0	0
4	c000057	0	2	0	0	0	0	0	0
4	c000228	0	0	0	0	0	2	1	1
4	c000333	0	4	0	0	0	0	0	0
4	c000347	0	2	0	0	0	1	0	0
4	c000461	0	3	0	0	0	0	0	0
4	c000542	0	4	0	0	0	0	0	0
4	c000548	0	0	0	0	0	0	1	0
4	c000639	0	2	0	0	0	0	0	0
4	c000689	0	0	0	0	0	0	0	0
4	c000948	0	0	0	1	0	1	0	0
4	MAX	1	5	0	1	0	2	1	1
5	c000004	1	4	5	6	1	1	0	1
5	c000101	0	2	7	0	1	0	2	1
5	c000189	0	2	5	6	1	0	3	3
5	c000224	0	5	7	6	2	1	3	1
5	c000245	1	5	5	6	0	0	0	0
5	c000316	0	4	4	5	1	2	0	0
5	c000385	0	5	0	8	2	0	0	1
5	c000603	0	2	6	5	1	1	5	3
5	c000734	0	0	6	0	1	1	0	1
5	c000765	0	1	8	7	1	1	0	1
5	c000832	0	4	5	6	2	1	2	3
5	MAX	1	5	8	8	2	2	5	3
6	c000005	1	1	0	0	0	1	1	0
6	c000043	1	0	0	1	0	1	1	0
6	c000082	1	1	0	0	0	0	0	0
6	c000126	3	1	0	0	0	0	0	0
6	c000204	0	1	0	0	0	1	1	0
6	c000256	0	1	0	1	0	0	0	0
6	c000289	3	0	1	1	0	0	0	1
6	c000377	1	0	0	0	0	0	0	0
6	c000399	1	2	0	0	0	0	0	0
6	c000768	0	1	1	0	0	0	0	0
6	c000928	1	0	1	0	0	0	0	1
6	MAX	3	2	1	1	0	1	1	1
7	c000112	0	0	1	0	1	0	1	0
7	c000145	0	0	0	0	0	0	0	0
7	c000314	1	5	0	0	0	0	0	1
7	c000345	0	2	0	0	0	0	0	0
7	c000348	0	5	0	0	0	0	0	0
7	c000396	0	1	0	0	0	0	0	1
7	c000539	1	5	0	0	0	0	1	2
7	c000578	3	2	0	0	0	0	0	0
7	c000654	0	0	0	0	0	1	0	1
7	c000890	2	1	0	0	0	0	0	0
7	c000952	0	3	0	0	1	0	1	0
7	MAX	3	5	1	0	1	1	1	2
8	c000015	5	0	0	0	0	1	0	0
8	c000155	0	0	0	0	0	0	0	0
8	c000182	3	0	0	0	1	0	1	1
8	c000610	2	1	0	0	1	1	0	1
8	c000690	1	1	0	0	1	1	0	0
8	c000706	4	0	0	0	0	0	0	1
8	c000842	0	0	0	0	1	0	0	0
8	c000846	4	0	0	0	0	0	0	0
8	c000897	1	0	0	0	0	0	1	1
8	c000975	0	1	0	0	0	0	0	0
8	c000998	0	0	0	0	1	1	1	1
8	MAX	5	1	0	0	1	1	1	1
9	c000042	0	1	0	0	0	1	0	0
9	c000136	0	0	0	0	0	0	0	0
9	c000157	0	1	2	0	0	0	0	0
9	c000210	0	2	0	0	3	0	1	1
9	c000221	1	2	1	1	2	0	2	0
9	c000418	0	4	0	0	1	0	0	0
9	c000801	1	2	0	0	1	0	0	0
9	c000834	0	3	0	0	1	1	1	1
9	c000885	0	5	0	0	0	0	0	0
9	c000908	0	6	0	0	1	0	0	0
9	c000919	0	3	2	1	0	1	0	0
9	MAX	1	6	2	1	3	1	2	1
10	c000007	0	0	0	0	0	0	0	1
10	c000021	0	0	0	0	2	0	0	0
10	c000050	0	0	0	0	0	0	0	0
10	c000073	1	0	0	0	0	1	1	0
10	c000218	0	0	0	0	0	0	1	0
10	c000241	0	0	0	0	1	0	1	1
10	c000679	0	0	0	0	0	0	0	0
10	c000730	0	0	0	0	2	0	1	1
10	c000804	0	0	0	0	0	0	0	0
10	c000867	0	0	0	0	1	1	1	0
10	c000922	0	0	0	1	1	0	2	1
10	MAX	1	0	0	1	2	1	2	1
11	c000059	2	2	4	5	2	0	3	3
11	c000197	0	1	3	5	0	1	1	0
11	c000220	0	0	3	6	0	0	0	0
11	c000263	1	2	3	4	2	1	1	2
11	c000375	2	3	4	0	1	0	0	0
11	c000429	1	0	3	4	0	1	0	0
11	c000431	1	3	3	0	0	2	2	1
11	c000529	1	1	0	3	1	2	2	1
11	c000568	3	0	0	3	2	2	3	2
11	c000685	1	1	5	6	0	0	0	1
11	c000692	0	0	0	0	0	0	0	0
11	MAX	3	3	5	6	2	2	3	3
12	c000179	0	7	5	2	0	0	0	0
12	c000181	1	7	6	2	0	1	0	0
12	c000260	0	1	5	2	1	1	0	2
12	c000284	0	7	6	4	0	1	1	1
12	c000329	0	7	0	0	0	0	0	0
12	c000367	1	4	0	3	0	1	1	0
12	c000521	0	7	6	5	0	0	0	0
12	c000800	0	7	0	1	0	0	0	0
12	c000865	1	7	6	4	0	0	0	0
12	c000962	0	7	5	4	0	1	0	0
12	c000971	0	1	2	5	0	0	0	0
12	MAX	1	7	6	5	1	1	1	2
13	c000013	1	0	0	0	0	0	0	0
13	c000303	0	0	0	0	0	0	0	0
13	c000364	0	0	0	0	0	0	0	0
13	c000412	0	0	0	0	0	1	1	0
13	c000567	0	0	0	1	0	1	0	0
13	c000713	0	0	1	0	0	0	0	0
13	c000761	0	1	0	0	0	1	1	0
13	c000781	0	1	0	0	1	0	0	0
13	c000868	0	3	0	0	1	1	0	0
13	c000909	0	3	0	0	1	0	0	1
13	c000994	2	0	0	0	0	0	1	0
13	MAX	2	3	1	1	1	1	1	1
14	c000063	1	5	6	3	0	0	0	0
14	c000110	0	1	0	0	0	0	0	0
14	c000116	0	5	1	0	0	0	0	0
14	c000146	0	2	6	3	0	0	0	0
14	c000336	0	7	6	1	1	0	0	0
14	c000378	1	6	7	3	1	0	0	0
14	c000579	1	0	7	3	1	0	0	0
14	c000823	0	5	1	0	0	0	0	1
14	c000869	3	5	7	1	1	0	0	0
14	c000972	0	5	0	0	0	0	1	0
14	c000997	0	0	0	0	0	0	0	0
14	MAX	3	7	7	3	1	0	1	1
15	c000048	3	2	0	0	0	0	0	0
15	c000125	0	6	0	0	0	0	0	0
15	c000191	0	4	0	1	0	0	1	0
15	c000451	2	8	0	0	0	0	0	0
15	c000475	1	3	0	0	0	0	0	0
15	c000585	0	7	0	0	0	1	0	0
15	c000614	1	1	0	0	1	0	0	0
15	c000658	2	5	0	0	0	0	0	1
15	c000659	0	0	0	0	0	0	0	0
15	c000724	1	4	0	0	0	1	0	0
15	c000915	0	5	0	0	0	1	0	0
15	MAX	3	8	0	1	1	1	1	1
16	c000038	0	4	0	1	2	0	2	2
16	c000080	0	0	0	0	1	0	0	0
16	c000085	0	0	0	0	0	0	2	3
16	c000153	1	2	0	0	0	0	0	0
16	c000162	0	3	0	0	1	0	1	0
16	c000257	0	4	0	0	1	0	2	2
16	c000324	0	0	0	0	0	0	0	1
16	c000416	0	3	0	0	0	0	0	0
16	c000512	0	0	0	0	0	0	0	0
16	c000793	1	0	0	1	0	0	0	0
16	c000859	0	0	0	0	0	0	0	0
16	MAX	1	4	0	1	2	0	2	3
17	c000006	0	9	0	0	0	0	1	0
17	c000019	2	7	0	0	0	0	0	0
17	c000149	0	5	2	0	0	0	0	1
17	c000195	0	8	0	1	0	0	0	0
17	c000266	2	8	2	1	1	0	1	0
17	c000372	0	5	0	0	0	0	0	0
17	c000425	0	8	0	0	1	1	0	1
17	c000649	2	6	0	0	0	0	0	0
17	c000775	0	0	0	0	0	1	1	1
17	c000888	0	7	1	0	0	0	0	0
17	c000955	0	9	0	1	0	0	0	0
17	MAX	2	9	2	1	1	1	1	1
18	c000074	1	5	5	5	0	0	0	0
18	c000081	1	1	5	5	0	0	1	1
18	c000128	0	3	4	0	0	0	0	0
18	c000167	1	6	5	4	0	0	0	0
18	c000386	1	3	0	2	0	1	1	0
18	c000400	0	4	5	5	0	0	1	0
18	c000525	0	5	0	0	0	0	0	0
18	c000637	0	1	0	1	0	0	0	0
18	c000684	1	1	1	0	0	0	0	0
18	c000731	0	4	0	2	0	0	0	0
18	c000773	0	3	4	3	0	1	1	0
18	MAX	1	6	5	5	0	1	1	1
19	c000016	0	0	0	0	0	0	0	0
19	c000199	0	3	0	0	0	0	1	0
19	c000265	0	1	0	0	0	1	1	0
19	c000279	0	0	0	0	0	0	1	0
19	c000326	0	1	0	0	0	0	0	0
19	c000423	0	1	0	0	0	0	0	0
19	c000437	0	5	0	0	1	1	1	0
19	c000788	0	4	0	0	0	0	0	2
19	c000851	1	0	0	0	0	0	0	0
19	c000924	0	0	0	0	0	0	0	0
19	c000934	0	1	0	0	0	0	1	0
19	MAX	1	5	0	0	1	1	1	2
20	c000304	0	0	0	0	0	0	0	0
20	c000330	0	0	0	0	0	1	2	1
20	c000447	0	0	0	0	0	0	1	0
20	c000508	0	0	0	0	0	0	0	0
20	c000625	0	0	0	0	0	0	1	0
20	c000674	0	5	0	0	0	0	2	1
20	c000716	0	0	0	0	0	0	0	0
20	c000854	0	1	0	0	0	0	0	0
20	c000870	0	0	0	0	1	0	1	0
20	c000882	0	1	0	0	0	0	0	0
20	c000925	0	4	0	0	0	0	0	0
20	MAX	0	5	0	0	1	1	2	1
21	c000023	0	0	0	1	0	0	0	0
21	c000120	2	0	1	1	1	1	1	0
21	c000230	0	0	0	0	0	0	0	0
21	c000276	0	0	0	0	0	0	0	0
21	c000281	1	1	0	0	0	0	1	1
21	c000342	0	2	0	0	0	0	0	0
21	c000359	0	2	1	1	0	0	0	0
21	c000467	0	2	0	0	1	1	0	1
21	c000666	2	0	0	0	0	2	1	0
21	c000688	0	0	0	0	0	0	2	0
21	c000769	4	0	1	1	1	0	3	0
21	MAX	4	2	1	1	1	2	3	1
22	c000097	1	3	0	0	1	0	0	0
22	c000129	0	4	0	0	1	1	0	2
22	c000234	0	2	0	0	1	0	0	1
22	c000319	1	3	0	1	0	0	0	0
22	c000446	2	0	1	1	0	0	0	0
22	c000473	0	0	0	0	0	0	0	0
22	c000635	2	0	0	0	0	0	0	1
22	c000737	0	1	0	0	0	0	0	0
22	c000778	0	4	0	0	0	0	0	0
22	c000876	4	0	0	0	0	0	0	0
22	c000929	4	5	1	0	0	1	0	0
22	MAX	4	5	1	1	1	1	0	2
23	c000058	3	4	2	3	0	0	1	0
23	c000096	1	1	4	4	2	0	0	1
23	c000203	1	0	0	0	0	1	0	0
23	c000229	1	5	5	5	2	1	2	2
23	c000274	0	5	4	0	1	0	0	0
23	c000322	2	0	3	4	2	1	2	0
23	c000495	1	5	3	4	2	0	3	0
23	c000555	3	0	0	0	0	0	0	0
23	c000586	1	5	5	4	1	2	2	2
23	c000602	1	0	0	0	0	0	1	0
23	c000873	2	5	0	0	1	0	2	2
23	MAX	3	5	5	5	2	2	3	2
24	c000207	0	0	0	0	0	0	1	0
24	c000302	0	0	0	0	0	0	1	0
24	c000341	2	0	0	0	0	1	0	0
24	c000371	1	3	0	0	1	0	0	1
24	c000430	0	3	0	0	0	0	1	0
24	c000574	3	1	0	0	0	0	0	0
24	c000708	2	0	0	0	0	0	0	0
24	c000739	0	0	0	0	1	0	0	1
24	c000762	3	0	0	0	0	0	0	0
24	c000875	2	3	0	0	1	0	1	0
24	c000903	3	0	0	0	0	0	0	0
24	MAX	3	3	0	0	1	1	1	1
25	c000022	0	0	0	1	0	0	0	0
25	c000067	1	2	1	0	0	1	1	0
25	c000098	0	0	1	0	0	0	1	0
25	c000174	1	3	2	2	0	1	0	0
25	c000223	0	3	1	1	0	0	0	1
25	c000268	0	2	1	0	0	0	0	1
25	c000273	0	1	1	0	0	0	0	0
25	c000296	0	0	0	0	0	0	0	0
25	c000519	0	0	2	0	0	0	0	0
25	c000580	2	3	1	1	1	1	0	0
25	c000794	0	0	0	0	0	1	0	0
25	MAX	2	3	2	2	1	1	1	1
26	c000099	0	3	0	0	0	0	1	0
26	c000271	1	0	0	0	0	0	0	1
26	c000338	1	3	0	0	1	0	0	0
26	c000349	1	1	0	0	0	0	1	0
26	c000370	1	2	1	2	2	1	1	1
26	c000424	0	0	0	3	0	0	0	0
26	c000573	0	4	2	2	1	0	0	0
26	c000663	0	2	0	0	0	0	0	0
26	c000701	0	0	0	0	0	0	0	0
26	c000771	0	1	0	0	0	0	0	0
26	c000959	2	0	0	0	1	0	0	1
26	MAX	2	4	2	3	2	1	1	1
27	c000454	0	0	1	1	0	0	0	0
27	c000571	1	5	0	0	1	0	0	0
27	c000627	0	0	0	1	0	0	0	0
27	c000633	0	5	0	0	0	0	0	0
27	c000643	0	3	1	1	1	0	0	0
27	c000749	0	0	2	1	0	0	0	0
27	c000780	0	5	4	3	2	1	0	1
27	c000815	0	5	4	4	0	0	1	1
27	c000824	0	0	0	0	0	0	0	0
27	c000840	0	0	1	3	0	2	1	0
27	c000855	0	4	0	0	1	2	0	0
27	MAX	1	5	4	4	2	2	1	1
28	c000027	3	3	0	0	2	2	0	1
28	c000041	1	6	4	1	0	0	2	0
28	c000106	0	2	0	1	1	2	2	0
28	c000298	0	6	0	0	0	0	0	0
28	c000358	2	5	0	0	0	0	1	1
28	c000362	2	3	2	0	0	1	1	0
28	c000449	1	3	0	1	1	0	0	2
28	c000478	2	6	5	3	1	3	0	0
28	c000511	3	3	3	0	0	0	0	2
28	c000770	1	1	0	0	0	0	0	3
28	c000816	2	4	0	1	1	2	2	0
28	MAX	3	6	5	3	2	3	2	3
29	c000009	0	8	0	1	0	0	0	0
29	c000046	1	9	7	7	0	1	1	1
29	c000049	0	8	6	7	0	0	1	0
29	c000202	0	8	6	7	2	0	1	1
29	c000238	1	0	6	7	0	0	1	1
29	c000293	0	5	0	0	0	0	1	0
29	c000409	0	8	5	5	0	0	0	0
29	c000484	0	9	6	7	1	1	2	1
29	c000618	0	8	0	2	0	0	1	1
29	c000792	0	9	0	0	0	1	0	1
29	c000983	0	9	6	7	1	1	1	0
29	MAX	1	9	7	7	2	1	2	1
30	c000090	3	5	4	0	1	0	0	1
30	c000240	4	6	3	3	0	1	1	0
30	c000513	0	0	0	0	0	1	0	0
30	c000572	3	5	3	2	0	1	0	0
30	c000652	4	1	4	3	1	0	0	2
30	c000757	0	5	0	0	0	1	0	0
30	c000817	0	5	1	1	0	1	1	0
30	c000848	1	5	3	0	0	1	0	0
30	c000967	2	1	4	2	0	1	0	0
30	c000989	1	0	0	0	1	0	0	0
30	c000996	1	1	0	0	0	2	2	0
30	MAX	4	6	4	3	1	2	2	2
	Sum of maxima	64	136	75	68	39	37	47	45
	Grand Total	211	761	365	314	117	102	149	116

download these results as csv

@@ Line 24: / Line 24: @@
 ==Overall Summary Results==
-<csv>coversong_overall.csv</csv>
+<csv>2006/coversong_overall.csv</csv>
 ===Audio Cover Song Identification Runtime Data===
-<csv>ac06_runtime.csv</csv>
+<csv>2006/ac06_runtime.csv</csv>
 ==Friedman Test with Multiple Comparisons Results (p=0.05)==
 The Friedman test was run in MATLAB against the MRR summary data over the 30 song groups.<br /> Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
-<csv>cover_song_sum_friedman.csv</csv>
+<csv>2006/cover_song_sum_friedman.csv</csv>
-<csv>audiocover_friedman_mrr.csv</csv>
+<csv>2006/audiocover_friedman_mrr.csv</csv>
 ==Mean Reciprocal Rank (MRR) Results==
-<csv>coversong_mrr.csv</csv>
+<csv>2006/coversong_mrr.csv</csv>
 ==Raw Results==
 The total number of correctly identified covers for each query, and the maximum number of covers identified for each cover song group.
-<csv>coversong_results.csv</csv>
+<csv>2006/coversong_results.csv</csv>

Difference between revisions of "2006:Audio Cover Song Identification Results"

Revision as of 18:32, 13 May 2010

Contents

Introduction

General Legend

Team ID

Calculating Summary Measures

Overall Summary Results

Audio Cover Song Identification Runtime Data

Friedman Test with Multiple Comparisons Results (p=0.05)

Mean Reciprocal Rank (MRR) Results

Raw Results

Navigation menu

Views

Personal tools

MIREX by Year

Results by Year

Account Request

Search

Navigation

Tools