Difference between revisions of "2009:Audio Music Similarity and Retrieval Results"

Latest revision as of 15:31, 23 July 2010

Introduction

These are the results for the 2009 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.

Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 100 songs were randomly selected from the 10 genre groups (10 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical BROAD score with 3 categories: NS,SS,VS as explained below, and one FINE score (in the range from 0 to 10). A description and analysis is provided below.

The systems read in 30 second audio clips as their raw data. The same 30 second clips were used in the grading stage.

General Legend

Team ID

ANO = Anonymous
BF1 = Benjamin Fields (chr12)
BF2 = Benjamin Fields (mfcc10)
BSWH1 = Dmitry Bogdanov, Joan Serrà, Nicolas Wack, Perfecto Herrera (clas)
BSWH2 = Dmitry Bogdanov, Joan Serrà, Nicolas Wack, Perfecto Herrera (hybrid)
CL1 = Chuan Cao, Ming Li
CL2 = Chuan Cao, Ming Li
GT = George Tzanetakis
LR = Thomas Lidy, Andreas Rauber
ME1 = François Maillet, Douglas Eck (mlp)
ME2 = François Maillet, Douglas Eck (sda)
PS1 = Tim Pohle, Dominik Schnitzer (2007)
PS2 = Tim Pohle, Dominik Schnitzer (2009)
SH1 = Stephan Hübler
SH2 = Stephan Hübler

Broad Categories

NS = Not Similar
SS = Somewhat Similar
VS = Very Similar

Understanding Summary Measures

Fine = Has a range from 0 (failure) to 10 (perfection).
Broad = Has a range from 0 (failure) to 2 (perfection) as each query/candidate pair is scored with either NS=0, SS=1 or VS=2.

Human Evaluation

Overall Summary Results

Measure	ANO	BF1	BF2	BSWH1	BSWH2	CL1	CL2	GT	LR	ME1	ME2	PS1	PS2	SH1	SH2
Average FINE Score	5.391	2.401	2.587	5.137	5.734	2.525	5.392	5.343	5.470	2.331	2.585	5.751	6.458	5.042	4.932
Average BROAD Score	1.126	0.416	0.410	1.094	1.232	0.476	1.164	1.126	1.148	0.356	0.418	1.262	1.448	1.012	1.040

download these results as csv

Friedman's Tests

Friedman's Test (FINE Scores)

The Friedman test was run in MATLAB against the Fine summary data over the 100 queries.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
PS2	PS1	-0.298	1.845	3.988	FALSE
PS2	BSWH2	-0.278	1.865	4.008	FALSE
PS2	LR	0.502	2.645	4.788	TRUE
PS2	CL2	0.917	3.060	5.203	TRUE
PS2	ANO	0.767	2.910	5.053	TRUE
PS2	GT	0.817	2.960	5.103	TRUE
PS2	BSWH1	1.642	3.785	5.928	TRUE
PS2	SH1	1.617	3.760	5.903	TRUE
PS2	SH2	1.837	3.980	6.123	TRUE
PS2	BF2	6.507	8.650	10.793	TRUE
PS2	ME2	6.557	8.700	10.843	TRUE
PS2	CL1	6.627	8.770	10.913	TRUE
PS2	BF1	6.857	9.000	11.143	TRUE
PS2	ME1	6.952	9.095	11.238	TRUE
PS1	BSWH2	-2.123	0.020	2.163	FALSE
PS1	LR	-1.343	0.800	2.943	FALSE
PS1	CL2	-0.928	1.215	3.358	FALSE
PS1	ANO	-1.078	1.065	3.208	FALSE
PS1	GT	-1.028	1.115	3.258	FALSE
PS1	BSWH1	-0.203	1.940	4.083	FALSE
PS1	SH1	-0.228	1.915	4.058	FALSE
PS1	SH2	-0.008	2.135	4.278	FALSE
PS1	BF2	4.662	6.805	8.948	TRUE
PS1	ME2	4.712	6.855	8.998	TRUE
PS1	CL1	4.782	6.925	9.068	TRUE
PS1	BF1	5.012	7.155	9.298	TRUE
PS1	ME1	5.107	7.250	9.393	TRUE
BSWH2	LR	-1.363	0.780	2.923	FALSE
BSWH2	CL2	-0.948	1.195	3.338	FALSE
BSWH2	ANO	-1.098	1.045	3.188	FALSE
BSWH2	GT	-1.048	1.095	3.238	FALSE
BSWH2	BSWH1	-0.223	1.920	4.063	FALSE
BSWH2	SH1	-0.248	1.895	4.038	FALSE
BSWH2	SH2	-0.028	2.115	4.258	FALSE
BSWH2	BF2	4.642	6.785	8.928	TRUE
BSWH2	ME2	4.692	6.835	8.978	TRUE
BSWH2	CL1	4.762	6.905	9.048	TRUE
BSWH2	BF1	4.992	7.135	9.278	TRUE
BSWH2	ME1	5.087	7.230	9.373	TRUE
LR	CL2	-1.728	0.415	2.558	FALSE
LR	ANO	-1.878	0.265	2.408	FALSE
LR	GT	-1.828	0.315	2.458	FALSE
LR	BSWH1	-1.003	1.140	3.283	FALSE
LR	SH1	-1.028	1.115	3.258	FALSE
LR	SH2	-0.808	1.335	3.478	FALSE
LR	BF2	3.862	6.005	8.148	TRUE
LR	ME2	3.912	6.055	8.198	TRUE
LR	CL1	3.982	6.125	8.268	TRUE
LR	BF1	4.212	6.355	8.498	TRUE
LR	ME1	4.307	6.450	8.593	TRUE
CL2	ANO	-2.293	-0.150	1.993	FALSE
CL2	GT	-2.243	-0.100	2.043	FALSE
CL2	BSWH1	-1.418	0.725	2.868	FALSE
CL2	SH1	-1.443	0.700	2.843	FALSE
CL2	SH2	-1.223	0.920	3.063	FALSE
CL2	BF2	3.447	5.590	7.733	TRUE
CL2	ME2	3.497	5.640	7.783	TRUE
CL2	CL1	3.567	5.710	7.853	TRUE
CL2	BF1	3.797	5.940	8.083	TRUE
CL2	ME1	3.892	6.035	8.178	TRUE
ANO	GT	-2.093	0.050	2.193	FALSE
ANO	BSWH1	-1.268	0.875	3.018	FALSE
ANO	SH1	-1.293	0.850	2.993	FALSE
ANO	SH2	-1.073	1.070	3.213	FALSE
ANO	BF2	3.597	5.740	7.883	TRUE
ANO	ME2	3.647	5.790	7.933	TRUE
ANO	CL1	3.717	5.860	8.003	TRUE
ANO	BF1	3.947	6.090	8.233	TRUE
ANO	ME1	4.042	6.185	8.328	TRUE
GT	BSWH1	-1.318	0.825	2.968	FALSE
GT	SH1	-1.343	0.800	2.943	FALSE
GT	SH2	-1.123	1.020	3.163	FALSE
GT	BF2	3.547	5.690	7.833	TRUE
GT	ME2	3.597	5.740	7.883	TRUE
GT	CL1	3.667	5.810	7.953	TRUE
GT	BF1	3.897	6.040	8.183	TRUE
GT	ME1	3.992	6.135	8.278	TRUE
BSWH1	SH1	-2.168	-0.025	2.118	FALSE
BSWH1	SH2	-1.948	0.195	2.338	FALSE
BSWH1	BF2	2.722	4.865	7.008	TRUE
BSWH1	ME2	2.772	4.915	7.058	TRUE
BSWH1	CL1	2.842	4.985	7.128	TRUE
BSWH1	BF1	3.072	5.215	7.358	TRUE
BSWH1	ME1	3.167	5.310	7.453	TRUE
SH1	SH2	-1.923	0.220	2.363	FALSE
SH1	BF2	2.747	4.890	7.033	TRUE
SH1	ME2	2.797	4.940	7.083	TRUE
SH1	CL1	2.867	5.010	7.153	TRUE
SH1	BF1	3.097	5.240	7.383	TRUE
SH1	ME1	3.192	5.335	7.478	TRUE
SH2	BF2	2.527	4.670	6.813	TRUE
SH2	ME2	2.577	4.720	6.863	TRUE
SH2	CL1	2.647	4.790	6.933	TRUE
SH2	BF1	2.877	5.020	7.163	TRUE
SH2	ME1	2.972	5.115	7.258	TRUE
BF2	ME2	-2.093	0.050	2.193	FALSE
BF2	CL1	-2.023	0.120	2.263	FALSE
BF2	BF1	-1.793	0.350	2.493	FALSE
BF2	ME1	-1.698	0.445	2.588	FALSE
ME2	CL1	-2.073	0.070	2.213	FALSE
ME2	BF1	-1.843	0.300	2.443	FALSE
ME2	ME1	-1.748	0.395	2.538	FALSE
CL1	BF1	-1.913	0.230	2.373	FALSE
CL1	ME1	-1.818	0.325	2.468	FALSE
BF1	ME1	-2.048	0.095	2.238	FALSE

download these results as csv

https://music-ir.org/mirex/results/2009/ams/evalutron/small.evalutron.fine.friedman.tukeyKramerHSD.png

Friedman's Test (BROAD Scores)

The Friedman test was run in MATLAB against the BROAD summary data over the 100 queries.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
PS2	PS1	-0.352	1.730	3.812	FALSE
PS2	BSWH2	-0.052	2.030	4.112	FALSE
PS2	CL2	0.657	2.740	4.822	TRUE
PS2	LR	0.682	2.765	4.848	TRUE
PS2	ANO	1.157	3.240	5.322	TRUE
PS2	GT	0.802	2.885	4.968	TRUE
PS2	BSWH1	1.387	3.470	5.553	TRUE
PS2	SH2	1.742	3.825	5.907	TRUE
PS2	SH1	2.252	4.335	6.418	TRUE
PS2	CL1	6.277	8.360	10.443	TRUE
PS2	ME2	6.478	8.560	10.643	TRUE
PS2	BF1	6.402	8.485	10.568	TRUE
PS2	BF2	6.463	8.545	10.627	TRUE
PS2	ME1	6.772	8.855	10.938	TRUE
PS1	BSWH2	-1.782	0.300	2.382	FALSE
PS1	CL2	-1.073	1.010	3.092	FALSE
PS1	LR	-1.048	1.035	3.118	FALSE
PS1	ANO	-0.573	1.510	3.592	FALSE
PS1	GT	-0.927	1.155	3.237	FALSE
PS1	BSWH1	-0.343	1.740	3.822	FALSE
PS1	SH2	0.013	2.095	4.178	TRUE
PS1	SH1	0.522	2.605	4.688	TRUE
PS1	CL1	4.548	6.630	8.713	TRUE
PS1	ME2	4.747	6.830	8.912	TRUE
PS1	BF1	4.673	6.755	8.838	TRUE
PS1	BF2	4.732	6.815	8.898	TRUE
PS1	ME1	5.043	7.125	9.207	TRUE
BSWH2	CL2	-1.373	0.710	2.792	FALSE
BSWH2	LR	-1.347	0.735	2.817	FALSE
BSWH2	ANO	-0.873	1.210	3.292	FALSE
BSWH2	GT	-1.228	0.855	2.938	FALSE
BSWH2	BSWH1	-0.642	1.440	3.522	FALSE
BSWH2	SH2	-0.287	1.795	3.877	FALSE
BSWH2	SH1	0.223	2.305	4.388	TRUE
BSWH2	CL1	4.247	6.330	8.412	TRUE
BSWH2	ME2	4.447	6.530	8.613	TRUE
BSWH2	BF1	4.372	6.455	8.537	TRUE
BSWH2	BF2	4.433	6.515	8.598	TRUE
BSWH2	ME1	4.742	6.825	8.908	TRUE
CL2	LR	-2.058	0.025	2.107	FALSE
CL2	ANO	-1.583	0.500	2.583	FALSE
CL2	GT	-1.938	0.145	2.228	FALSE
CL2	BSWH1	-1.353	0.730	2.812	FALSE
CL2	SH2	-0.998	1.085	3.167	FALSE
CL2	SH1	-0.487	1.595	3.678	FALSE
CL2	CL1	3.538	5.620	7.702	TRUE
CL2	ME2	3.737	5.820	7.902	TRUE
CL2	BF1	3.663	5.745	7.827	TRUE
CL2	BF2	3.723	5.805	7.888	TRUE
CL2	ME1	4.032	6.115	8.197	TRUE
LR	ANO	-1.607	0.475	2.558	FALSE
LR	GT	-1.962	0.120	2.203	FALSE
LR	BSWH1	-1.377	0.705	2.788	FALSE
LR	SH2	-1.022	1.060	3.143	FALSE
LR	SH1	-0.512	1.570	3.652	FALSE
LR	CL1	3.513	5.595	7.678	TRUE
LR	ME2	3.712	5.795	7.878	TRUE
LR	BF1	3.638	5.720	7.803	TRUE
LR	BF2	3.697	5.780	7.862	TRUE
LR	ME1	4.008	6.090	8.172	TRUE
ANO	GT	-2.438	-0.355	1.728	FALSE
ANO	BSWH1	-1.853	0.230	2.312	FALSE
ANO	SH2	-1.498	0.585	2.667	FALSE
ANO	SH1	-0.988	1.095	3.178	FALSE
ANO	CL1	3.038	5.120	7.202	TRUE
ANO	ME2	3.237	5.320	7.402	TRUE
ANO	BF1	3.163	5.245	7.327	TRUE
ANO	BF2	3.223	5.305	7.388	TRUE
ANO	ME1	3.533	5.615	7.697	TRUE
GT	BSWH1	-1.498	0.585	2.667	FALSE
GT	SH2	-1.143	0.940	3.022	FALSE
GT	SH1	-0.632	1.450	3.533	FALSE
GT	CL1	3.393	5.475	7.558	TRUE
GT	ME2	3.592	5.675	7.758	TRUE
GT	BF1	3.518	5.600	7.683	TRUE
GT	BF2	3.578	5.660	7.742	TRUE
GT	ME1	3.888	5.970	8.053	TRUE
BSWH1	SH2	-1.728	0.355	2.438	FALSE
BSWH1	SH1	-1.218	0.865	2.947	FALSE
BSWH1	CL1	2.808	4.890	6.973	TRUE
BSWH1	ME2	3.007	5.090	7.173	TRUE
BSWH1	BF1	2.933	5.015	7.098	TRUE
BSWH1	BF2	2.993	5.075	7.157	TRUE
BSWH1	ME1	3.303	5.385	7.468	TRUE
SH2	SH1	-1.573	0.510	2.592	FALSE
SH2	CL1	2.453	4.535	6.617	TRUE
SH2	ME2	2.652	4.735	6.817	TRUE
SH2	BF1	2.578	4.660	6.742	TRUE
SH2	BF2	2.638	4.720	6.803	TRUE
SH2	ME1	2.947	5.030	7.112	TRUE
SH1	CL1	1.942	4.025	6.107	TRUE
SH1	ME2	2.143	4.225	6.308	TRUE
SH1	BF1	2.067	4.150	6.232	TRUE
SH1	BF2	2.127	4.210	6.293	TRUE
SH1	ME1	2.438	4.520	6.603	TRUE
CL1	ME2	-1.883	0.200	2.283	FALSE
CL1	BF1	-1.958	0.125	2.208	FALSE
CL1	BF2	-1.897	0.185	2.268	FALSE
CL1	ME1	-1.587	0.495	2.578	FALSE
ME2	BF1	-2.158	-0.075	2.007	FALSE
ME2	BF2	-2.098	-0.015	2.067	FALSE
ME2	ME1	-1.788	0.295	2.377	FALSE
BF1	BF2	-2.022	0.060	2.143	FALSE
BF1	ME1	-1.712	0.370	2.453	FALSE
BF2	ME1	-1.772	0.310	2.393	FALSE

download these results as csv

https://music-ir.org/mirex/results/2009/ams/evalutron/small.evalutron.cat.friedman.tukeyKramerHSD.png

Summary Results by Query

FINE Scores

These are the mean FINE scores per query assigned by Evalutron graders. The FINE scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0.0 and 10.0. A perfect score would be 10. Genre labels have been included for reference.

Genre	Query	ANO	BF1	BF2	BSWH1	BSWH2	CL1	CL2	GT	LR	ME1	ME2	PS1	PS2	SH1	SH2
BAROQUE	d005166	3.060	2.120	1.900	2.520	3.700	3.280	5.560	3.560	2.620	0.540	1.860	3.040	3.700	3.960	2.040
BAROQUE	d007244	5.260	0.540	2.620	6.060	5.540	4.400	5.940	5.900	6.140	3.500	1.840	4.880	7.100	5.180	6.360
BAROQUE	d004490	7.880	3.940	6.680	3.400	6.060	5.340	8.520	7.500	8.300	5.160	3.440	7.800	9.200	7.080	8.280
BAROQUE	d009737	7.620	5.180	0.420	4.540	8.940	5.860	6.080	2.480	8.180	1.320	1.060	8.820	8.840	3.440	2.780
BAROQUE	d009837	4.140	1.520	0.360	1.700	1.640	2.860	1.000	3.100	3.860	0.700	0.720	3.960	4.680	2.320	3.520
BAROQUE	d008174	5.000	3.880	2.260	7.680	7.600	7.140	7.460	7.480	7.020	3.320	1.860	7.700	7.680	5.080	7.580
BAROQUE	d011054	7.320	4.160	0.000	4.360	6.140	5.500	5.280	5.460	5.980	0.940	1.000	5.980	7.160	5.780	5.560
BAROQUE	d018083	6.940	1.380	2.540	5.600	3.040	2.540	5.940	3.400	6.240	0.460	0.260	3.660	3.580	3.440	4.020
BAROQUE	d017335	4.680	0.280	0.960	1.940	4.640	2.340	3.440	5.780	3.380	2.540	0.000	3.560	4.240	3.880	1.980
BAROQUE	d019774	5.500	3.600	4.660	4.060	4.000	3.860	4.400	5.560	4.060	3.440	5.160	4.500	6.300	4.960	4.820
BLUES	e002545	5.760	6.260	1.320	5.600	7.140	0.280	6.760	6.140	5.160	1.600	4.180	7.340	8.060	4.760	5.200
BLUES	e000111	3.200	5.900	4.300	3.800	3.200	1.980	7.400	5.500	3.000	2.900	7.000	6.000	6.660	7.300	6.800
BLUES	e001417	6.700	3.320	2.920	4.760	4.980	7.480	8.800	7.660	7.000	6.480	4.940	7.160	6.980	5.020	7.160
BLUES	e004211	2.880	2.840	2.680	2.720	5.880	6.840	6.520	4.280	5.380	0.180	2.020	7.840	8.460	6.640	0.120
BLUES	e014267	5.400	2.200	0.960	7.040	4.700	0.000	4.880	5.400	2.780	1.380	3.220	4.460	6.640	4.440	2.280
BLUES	e013973	8.480	0.700	4.140	6.680	8.100	0.480	9.060	8.560	8.400	4.300	3.560	8.520	8.520	8.660	8.140
BLUES	e014486	3.800	3.080	2.280	4.280	5.180	0.160	1.760	5.380	3.160	3.420	1.300	5.920	4.360	4.660	5.000
BLUES	e010067	4.160	2.180	2.520	5.640	7.320	4.620	6.620	5.480	5.100	7.020	2.960	4.680	7.960	3.880	4.460
BLUES	e012895	7.760	3.360	1.580	5.880	7.560	6.720	7.740	6.460	7.840	2.280	1.740	8.360	8.540	7.700	6.060
BLUES	e015354	5.400	2.700	0.000	4.000	5.600	0.000	6.400	4.800	4.800	0.800	2.200	1.200	6.900	5.600	6.800
CLASSICAL	d000239	7.580	5.080	0.720	6.400	7.280	7.640	6.160	9.100	8.020	0.580	3.620	8.860	8.760	5.360	6.500
CLASSICAL	d000762	6.100	2.040	2.520	7.200	6.840	5.260	6.660	6.660	6.940	0.800	0.720	8.300	8.520	5.400	5.800
CLASSICAL	d002538	7.200	0.200	3.800	6.600	7.000	4.200	6.400	6.800	7.000	2.400	3.800	6.800	7.200	6.600	6.800
CLASSICAL	d001502	8.700	4.300	7.560	8.600	8.900	8.400	8.820	8.840	9.100	6.740	5.800	8.920	8.800	9.100	8.660
CLASSICAL	d012972	7.960	3.960	2.080	8.100	7.860	6.860	7.640	8.820	7.200	3.860	4.640	8.420	7.940	7.660	7.580
CLASSICAL	d010713	5.000	3.820	2.500	5.100	5.200	4.200	1.740	3.600	3.100	2.400	4.100	2.800	3.300	3.200	3.100
CLASSICAL	d012985	8.680	4.200	1.920	3.120	9.080	5.900	8.040	6.460	8.740	0.420	5.520	8.760	8.920	7.720	8.240
CLASSICAL	d019802	6.660	2.060	1.500	5.580	7.380	4.680	5.440	4.680	3.100	2.300	3.540	5.700	6.220	6.420	6.540
CLASSICAL	d019790	6.600	5.200	6.000	6.600	7.200	4.800	6.200	5.400	6.200	1.800	3.400	7.000	6.600	6.600	6.800
CLASSICAL	d019783	7.800	2.400	4.200	6.800	7.400	7.000	7.800	6.400	7.600	3.600	2.200	6.100	8.400	8.200	7.800
COUNTRY	e002580	3.880	1.660	2.300	3.740	3.340	0.000	2.440	4.800	2.900	3.080	1.920	3.760	3.820	2.640	2.500
COUNTRY	e003090	3.200	0.800	2.800	2.200	4.000	0.000	2.200	3.800	4.400	1.600	0.800	6.200	7.200	3.000	2.500
COUNTRY	e001591	2.000	4.760	3.780	3.380	6.340	1.700	4.880	5.140	2.740	2.640	3.060	7.200	6.680	5.600	5.500
COUNTRY	e006811	4.820	2.480	3.860	5.400	5.760	0.860	4.220	3.580	5.040	1.460	2.800	4.900	8.360	6.040	3.460
COUNTRY	e003155	0.000	0.060	1.200	0.200	2.800	0.000	3.340	1.720	2.100	0.200	0.500	3.700	5.320	5.220	2.680
COUNTRY	e003544	6.200	0.880	2.660	5.200	3.720	0.000	4.040	4.600	4.500	1.100	2.900	5.400	5.240	5.720	6.280
COUNTRY	e014385	4.800	0.000	0.600	2.600	3.800	0.000	2.000	2.900	1.700	0.900	1.700	5.300	5.000	1.800	1.820
COUNTRY	e011855	6.860	0.740	1.160	4.280	6.340	0.440	4.960	6.200	6.560	1.500	0.820	7.380	5.040	5.480	5.120
COUNTRY	e015843	5.540	1.940	2.140	3.620	2.880	0.000	4.400	4.980	4.400	2.180	4.780	5.500	4.760	4.000	5.520
COUNTRY	e015137	7.120	2.600	2.860	4.880	6.420	0.840	4.260	6.700	7.740	3.100	2.800	7.160	7.560	5.360	3.360
EDANCE	a009068	2.400	1.940	3.840	4.800	5.280	2.320	3.760	3.940	2.680	1.940	2.360	3.520	4.640	3.540	4.520
EDANCE	b001840	3.920	0.660	0.300	4.640	4.960	0.100	0.280	5.120	4.120	0.400	0.000	5.020	4.460	1.520	1.200
EDANCE	b015999	7.780	2.000	1.700	6.640	6.480	0.000	6.340	8.340	8.980	0.000	0.680	7.680	8.280	7.400	7.300
EDANCE	b015503	4.200	1.200	1.000	5.300	6.500	0.200	2.900	2.900	6.400	2.300	3.400	1.000	4.200	4.600	2.300
EDANCE	b019464	5.060	2.900	0.760	5.620	4.520	0.000	0.200	4.400	2.680	4.760	5.780	2.520	6.100	4.420	1.480
EDANCE	b019570	5.880	0.400	4.600	7.320	6.660	0.900	6.300	7.860	6.500	3.920	3.480	8.220	7.140	6.860	5.220
EDANCE	f014939	1.880	0.820	0.860	1.220	0.660	0.260	1.360	1.880	1.980	1.480	1.200	1.560	3.020	1.880	3.020
EDANCE	f008160	4.680	2.420	4.480	7.420	7.040	1.200	6.100	6.600	6.180	3.240	2.880	7.200	8.100	6.880	5.460
EDANCE	f011114	3.400	3.060	4.000	5.320	5.560	1.360	4.520	2.180	2.080	3.480	4.760	1.580	5.620	2.620	2.500
EDANCE	f003748	6.800	1.660	4.240	6.540	5.840	0.260	6.560	6.380	5.140	4.860	1.160	6.800	4.720	4.980	4.800
JAZZ	a003703	1.000	3.260	0.180	2.540	1.120	0.000	3.820	0.680	3.620	0.580	0.120	3.060	2.200	0.600	1.020
JAZZ	e002394	1.000	0.000	1.000	1.800	1.800	3.200	4.000	6.800	3.600	1.700	1.800	3.800	4.000	0.400	2.500
JAZZ	e001113	6.980	2.120	3.480	7.640	8.360	3.520	7.400	4.560	8.220	2.320	2.940	7.780	7.120	7.100	5.180
JAZZ	e004292	6.000	3.940	1.300	7.520	8.000	5.800	6.740	7.080	5.700	1.140	2.040	5.600	7.940	3.760	3.720
JAZZ	e004662	2.780	0.860	2.160	1.060	3.880	0.800	4.640	2.660	4.300	1.280	0.640	3.580	6.120	1.660	2.680
JAZZ	e004944	7.600	2.860	2.740	5.220	5.640	6.600	8.320	8.520	8.660	2.560	2.900	5.280	8.540	8.060	5.080
JAZZ	e004070	6.440	4.540	1.440	3.980	4.900	1.480	4.960	5.680	5.760	4.120	5.140	5.180	6.380	4.520	3.440
JAZZ	e012026	2.140	4.180	0.160	4.740	5.400	1.000	2.440	3.900	2.640	3.460	0.760	1.920	7.920	2.880	6.280
JAZZ	e015744	7.540	2.120	1.560	7.860	8.120	3.440	7.300	7.900	8.140	5.360	4.900	7.700	9.080	6.560	7.180
JAZZ	e015566	6.440	2.800	1.700	4.720	6.700	7.020	5.780	5.560	7.720	5.020	2.720	6.840	8.300	5.560	2.940
METAL	a003208	7.440	1.620	5.800	6.280	7.220	1.100	7.880	6.980	7.000	3.880	4.260	7.840	5.400	7.820	8.120
METAL	b006262	5.920	3.500	3.360	6.520	4.060	0.000	5.420	2.380	3.240	1.260	1.740	5.800	6.920	3.280	5.340
METAL	b007445	7.240	3.680	4.500	4.760	6.600	6.580	7.020	7.380	7.820	2.760	4.560	7.140	7.920	7.380	6.260
METAL	b010346	6.600	1.780	2.340	5.980	7.060	2.520	6.820	6.160	7.260	3.800	4.800	6.720	6.940	7.320	6.960
METAL	b014327	1.760	0.920	3.860	6.000	3.400	0.620	5.960	4.240	2.800	1.560	2.260	5.200	5.880	4.820	5.200
METAL	b017546	4.300	2.340	3.120	6.140	5.400	0.960	5.980	4.480	5.920	1.020	0.980	6.420	7.880	5.100	3.680
METAL	b019571	6.760	2.000	3.720	5.460	7.540	1.380	6.300	7.600	7.380	3.160	2.200	7.700	7.000	7.080	6.080
METAL	f002408	6.220	0.140	5.320	5.480	6.260	1.560	7.120	5.880	6.940	2.960	8.120	8.260	7.540	5.860	6.200
METAL	f000530	6.440	1.520	2.020	6.240	7.300	0.000	8.520	7.520	6.860	2.000	4.900	6.780	7.980	7.440	7.520
METAL	f005072	5.640	3.420	4.360	5.060	4.800	2.700	5.140	6.640	5.540	1.920	3.740	4.760	5.760	3.620	4.380
RAPHIPHOP	a000293	7.480	3.540	2.080	7.780	8.580	0.480	7.540	7.160	5.320	2.580	3.660	8.120	9.020	2.160	5.760
RAPHIPHOP	a000827	5.700	4.020	3.880	4.440	5.860	0.120	6.500	6.420	4.440	2.940	2.080	6.260	3.100	3.000	4.220
RAPHIPHOP	a000897	5.940	3.560	4.120	6.020	6.580	0.400	5.500	6.740	6.760	2.000	2.160	6.480	6.860	6.220	6.400
RAPHIPHOP	a002740	6.560	2.540	3.160	6.040	7.180	0.940	5.640	6.080	6.600	2.500	2.280	7.000	6.640	5.240	6.720
RAPHIPHOP	a009059	6.200	1.920	2.540	6.520	6.640	0.020	7.000	2.200	5.040	2.900	2.320	2.280	7.760	3.680	2.460
RAPHIPHOP	b010144	6.380	1.820	2.100	7.120	7.500	0.080	8.180	7.280	6.700	3.520	1.860	6.700	8.920	7.820	7.880
RAPHIPHOP	b011546	5.680	3.040	0.860	5.960	7.800	0.000	6.480	7.600	5.800	2.680	1.980	7.320	5.640	5.520	6.800
RAPHIPHOP	b010454	7.820	1.680	5.060	8.300	8.340	0.000	7.600	7.100	8.900	2.320	3.160	8.140	8.220	6.880	8.300
RAPHIPHOP	b017461	3.660	1.780	4.280	7.500	7.100	0.180	7.840	6.900	5.780	4.220	4.100	7.320	7.700	6.600	7.160
RAPHIPHOP	b018747	7.300	0.920	0.640	8.280	8.240	0.000	8.060	7.960	7.040	1.920	0.980	7.820	8.240	7.260	7.200
ROCKROLL	b001069	6.000	2.760	5.260	5.780	3.600	4.080	6.520	3.520	6.700	1.540	3.820	6.220	7.420	5.860	6.800
ROCKROLL	b001751	3.640	1.300	0.800	2.540	1.540	0.940	0.980	1.780	3.120	1.380	2.100	3.660	2.860	3.760	4.100
ROCKROLL	b003625	1.600	2.100	3.140	5.340	3.080	0.800	3.040	4.040	2.280	1.160	1.300	4.600	4.220	5.280	5.500
ROCKROLL	b004162	5.040	3.180	3.380	3.700	4.760	1.740	5.840	4.260	5.020	2.340	2.200	4.660	5.540	5.420	4.560
ROCKROLL	b004353	6.600	1.800	3.600	7.000	6.600	1.000	5.400	4.400	5.400	4.000	2.800	4.200	8.300	6.200	5.000
ROCKROLL	b006456	5.400	1.760	8.240	7.800	5.740	0.480	6.880	6.000	6.720	1.660	3.200	8.060	6.020	7.980	6.540
ROCKROLL	b009365	5.960	1.020	4.220	4.840	7.940	0.200	4.220	3.920	6.160	2.940	3.180	6.860	7.440	7.600	5.200
ROCKROLL	b008878	5.480	2.400	3.160	5.820	4.740	0.840	4.300	5.860	4.940	2.720	2.360	4.220	4.040	3.140	1.700
ROCKROLL	b011553	1.740	0.960	1.540	2.400	2.660	0.140	3.020	3.240	3.280	0.120	0.800	1.800	3.560	2.220	1.460
ROCKROLL	e017211	2.080	1.580	1.640	1.460	2.820	1.340	1.980	1.700	2.120	1.580	1.340	2.000	2.020	2.140	1.980
ROMANTIC	d004429	8.860	6.340	3.840	7.860	8.800	7.920	7.140	8.400	8.820	5.140	3.540	8.700	9.180	8.560	8.860
ROMANTIC	d001688	4.020	0.600	1.260	2.160	3.420	2.440	2.660	3.080	3.780	0.060	0.540	3.780	3.100	2.100	2.460
ROMANTIC	d004908	5.640	2.220	0.360	8.400	7.060	5.720	5.880	5.940	5.220	1.720	0.740	6.280	7.500	4.800	6.000
ROMANTIC	d007929	4.700	1.500	2.060	6.860	7.700	7.180	5.080	3.340	6.740	0.420	2.120	6.480	6.800	4.720	4.640
ROMANTIC	d011624	6.400	0.000	1.800	3.900	6.700	3.500	5.800	3.700	6.100	0.400	0.800	7.100	7.480	0.200	3.200
ROMANTIC	d016855	4.700	3.900	0.000	3.600	5.800	4.100	4.900	4.300	4.600	0.400	0.900	4.400	4.400	4.400	4.900
ROMANTIC	d014946	7.140	2.200	1.800	6.400	6.300	7.060	2.000	5.040	7.360	0.480	2.880	5.880	7.460	5.900	4.720
ROMANTIC	d014940	5.680	2.920	0.280	6.460	7.160	4.300	6.160	5.860	7.680	1.460	0.880	6.400	7.380	3.220	4.900
ROMANTIC	d018672	2.100	0.200	1.080	0.580	2.300	1.660	1.720	2.540	2.240	0.320	0.000	2.860	2.440	2.320	1.640
ROMANTIC	d017019	3.680	3.520	1.320	1.180	3.320	3.540	4.260	4.720	2.960	2.500	1.440	3.520	4.240	3.560	3.020

download these results as csv

BROAD Scores

These are the mean BROAD scores per query assigned by Evalutron graders. The BROAD scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 (not similar) and 2 (very similar). A perfect score would be 2. Genre labels have been included for reference.

Genre	Query	ANO	BF1	BF2	BSWH1	BSWH2	CL1	CL2	GT	LR	ME1	ME2	PS1	PS2	SH1	SH2
BAROQUE	d005166	1.000	0.600	0.600	1.000	1.000	0.800	1.600	1.000	1.000	0.200	0.600	1.000	1.200	1.000	0.600
BAROQUE	d007244	1.200	0.000	0.400	1.400	1.200	0.800	1.400	1.600	1.400	0.800	0.200	0.800	1.800	1.200	1.600
BAROQUE	d004490	2.000	0.800	1.600	0.800	1.600	1.200	2.000	1.800	2.000	1.200	0.800	1.800	2.000	1.800	2.000
BAROQUE	d009737	2.000	1.200	0.000	0.800	2.000	1.600	1.200	0.400	1.800	0.000	0.000	2.000	2.000	0.800	0.400
BAROQUE	d009837	1.200	0.400	0.000	0.400	0.400	0.800	0.200	0.800	1.200	0.200	0.000	1.000	1.400	0.600	0.800
BAROQUE	d008174	1.200	0.800	0.400	2.000	2.000	1.800	2.000	2.000	1.600	0.600	0.200	2.000	2.000	1.200	2.000
BAROQUE	d011054	1.800	1.200	0.000	1.000	1.400	1.400	1.000	1.400	1.400	0.200	0.200	1.600	1.600	1.200	1.200
BAROQUE	d018083	2.000	0.400	0.600	1.600	0.600	0.600	1.400	1.000	1.800	0.000	0.000	1.000	1.000	0.600	1.000
BAROQUE	d017335	0.400	0.000	0.000	0.000	0.600	0.000	0.400	0.600	0.200	0.200	0.000	0.000	0.400	0.400	0.200
BAROQUE	d019774	1.000	0.200	0.800	1.000	1.400	1.200	1.200	1.600	1.400	0.200	1.200	2.000	2.000	1.200	1.800
BLUES	e002545	1.400	1.200	0.200	1.400	1.400	0.000	1.600	1.400	1.200	0.200	0.800	1.800	2.000	1.000	1.400
BLUES	e000111	0.400	1.200	0.800	0.600	0.400	0.000	1.600	1.200	0.400	0.600	1.400	1.000	1.200	1.400	1.400
BLUES	e001417	1.800	0.600	0.400	1.000	1.000	1.800	2.000	2.000	1.600	1.400	0.800	1.600	1.800	1.000	1.600
BLUES	e004211	0.400	0.600	0.600	0.600	1.400	1.600	1.600	0.600	0.800	0.000	0.400	1.800	2.000	1.600	0.000
BLUES	e014267	1.200	0.600	0.200	1.400	0.800	0.000	1.200	1.200	0.600	0.200	0.600	1.000	1.400	0.800	0.400
BLUES	e013973	1.600	0.200	0.600	1.400	1.800	0.000	2.000	2.000	1.600	0.800	0.800	1.800	1.800	2.000	1.800
BLUES	e014486	0.800	0.600	0.200	1.200	1.200	0.000	0.400	1.200	0.600	0.800	0.400	1.200	1.200	1.000	1.200
BLUES	e010067	0.400	0.000	0.200	1.200	1.600	0.800	1.400	0.800	0.800	1.600	0.400	0.800	2.000	0.400	0.600
BLUES	e012895	1.600	0.800	0.200	1.000	1.600	1.600	1.600	1.600	1.800	0.400	0.200	1.800	2.000	1.400	1.200
BLUES	e015354	1.200	0.600	0.000	1.000	1.400	0.000	1.400	1.200	1.400	0.200	0.400	0.200	1.600	1.400	2.000
CLASSICAL	d000239	1.600	1.000	0.000	1.400	1.600	1.600	1.200	2.000	1.800	0.000	0.800	2.000	2.000	1.000	1.200
CLASSICAL	d000762	1.400	0.400	0.400	1.600	1.400	1.000	1.400	1.400	1.600	0.000	0.000	2.000	2.000	1.000	1.000
CLASSICAL	d002538	1.200	0.000	0.600	1.000	1.200	0.800	1.200	1.200	1.200	0.400	0.800	1.200	1.400	1.000	1.400
CLASSICAL	d001502	2.000	0.800	1.600	2.000	2.000	1.800	2.000	2.000	2.000	1.200	1.000	2.000	2.000	2.000	2.000
CLASSICAL	d012972	1.800	0.800	0.000	1.800	2.000	1.600	1.800	2.000	1.600	0.400	0.800	2.000	2.000	1.800	1.800
CLASSICAL	d010713	1.200	0.600	0.000	1.400	1.600	1.000	1.400	1.800	1.200	0.200	0.000	1.800	1.600	0.800	1.000
CLASSICAL	d012985	2.000	0.800	0.400	0.600	1.800	1.200	1.800	1.200	2.000	0.000	1.000	2.000	1.800	1.400	2.000
CLASSICAL	d019802	1.200	0.400	0.200	0.800	1.600	0.600	1.000	0.800	0.400	0.400	0.400	1.200	1.000	1.200	1.400
CLASSICAL	d019790	1.000	1.000	1.200	1.400	1.400	0.800	1.000	1.000	1.000	0.200	0.600	1.200	1.400	1.000	1.400
CLASSICAL	d019783	1.800	0.200	0.600	1.400	1.800	1.600	1.800	1.200	1.800	0.600	0.000	1.200	2.000	2.000	1.800
COUNTRY	e002580	1.000	0.200	0.400	0.800	0.600	0.000	0.400	1.000	0.400	0.600	0.200	0.800	1.000	0.400	0.400
COUNTRY	e003090	0.600	0.200	0.600	0.400	1.000	0.000	0.800	0.800	1.000	0.400	0.200	1.000	1.600	0.800	0.800
COUNTRY	e001591	0.000	0.800	0.600	0.200	1.400	0.000	1.000	1.000	0.200	0.600	0.400	1.600	1.600	1.200	1.200
COUNTRY	e006811	1.200	0.400	0.600	1.200	1.200	0.000	0.800	0.600	1.200	0.200	0.400	1.000	2.000	1.400	0.400
COUNTRY	e003155	0.000	0.000	0.200	0.000	0.600	0.000	0.800	0.400	0.400	0.000	0.000	0.800	1.200	1.200	0.600
COUNTRY	e003544	1.400	0.200	0.400	1.200	0.600	0.000	1.000	0.800	0.800	0.000	0.400	1.400	1.200	1.400	1.400
COUNTRY	e014385	1.000	0.000	0.000	0.600	1.000	0.000	0.400	0.400	0.200	0.200	0.200	1.200	1.200	0.200	0.400
COUNTRY	e011855	1.600	0.000	0.000	0.600	1.800	0.000	1.000	1.000	1.400	0.200	0.000	2.000	1.200	1.400	1.200
COUNTRY	e015843	1.600	0.200	0.400	1.000	0.800	0.000	1.000	1.200	1.000	0.400	1.200	1.600	1.200	0.800	1.200
COUNTRY	e015137	1.600	0.600	0.600	1.000	1.200	0.200	0.600	1.600	1.600	0.600	0.400	1.600	1.800	1.200	0.800
EDANCE	a009068	0.000	0.200	0.600	1.000	1.000	0.200	0.600	0.800	0.200	0.200	0.200	0.600	0.800	0.600	0.800
EDANCE	b001840	0.400	0.000	0.000	0.600	0.600	0.000	0.000	0.800	0.800	0.000	0.000	0.800	0.600	0.000	0.000
EDANCE	b015999	1.800	0.400	0.200	1.400	1.400	0.000	1.200	1.800	2.000	0.000	0.000	1.800	1.800	1.600	1.600
EDANCE	b015503	0.600	0.200	0.000	1.000	1.200	0.000	0.400	0.400	1.000	0.200	0.400	0.000	0.400	0.400	0.200
EDANCE	b019464	1.200	0.600	0.000	1.600	1.000	0.000	0.000	1.000	0.600	0.800	1.400	0.600	1.600	1.000	0.200
EDANCE	b019570	1.200	0.000	0.600	1.600	1.400	0.000	1.200	1.800	1.000	0.600	0.600	2.000	1.400	1.400	1.200
EDANCE	f014939	0.200	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.200	0.000	0.000	0.000	0.400	0.000	0.600
EDANCE	f008160	1.000	0.200	1.000	1.800	1.600	0.000	1.400	1.600	1.400	0.400	0.600	1.800	2.000	1.600	1.400
EDANCE	f011114	0.600	0.400	0.600	1.000	1.200	0.000	1.000	0.200	0.000	0.400	0.800	0.000	1.000	0.400	0.400
EDANCE	f003748	1.600	0.200	0.800	1.600	1.200	0.000	1.200	1.200	1.200	1.000	0.200	1.400	1.000	1.000	1.000
JAZZ	a003703	0.200	0.600	0.000	0.600	0.200	0.000	1.000	0.200	0.600	0.200	0.000	0.800	0.400	0.000	0.200
JAZZ	e002394	0.200	0.000	0.200	0.400	0.200	0.800	1.000	1.400	0.600	0.200	0.400	0.800	1.000	0.000	0.600
JAZZ	e001113	1.400	0.400	0.800	1.800	1.800	0.800	1.600	1.000	1.800	0.400	0.400	1.600	1.600	1.600	1.000
JAZZ	e004292	1.000	0.800	0.000	1.600	1.600	1.200	1.600	1.400	1.000	0.000	0.200	1.000	1.600	0.400	0.800
JAZZ	e004662	0.400	0.000	0.000	0.000	0.600	0.000	0.600	0.000	0.600	0.000	0.000	0.400	1.200	0.000	0.200
JAZZ	e004944	2.000	0.400	0.400	1.200	1.200	1.600	2.000	2.000	2.000	0.200	0.600	1.200	2.000	2.000	1.000
JAZZ	e004070	1.400	1.000	0.000	0.600	1.000	0.200	1.000	1.000	1.200	0.600	0.800	1.000	1.400	0.800	0.600
JAZZ	e012026	0.400	0.800	0.000	1.000	1.000	0.200	0.600	0.600	0.600	0.600	0.200	0.400	2.000	0.400	1.600
JAZZ	e015744	1.400	0.200	0.200	1.800	1.800	0.400	1.200	1.600	1.800	1.000	0.800	1.800	2.000	1.200	1.200
JAZZ	e015566	1.400	0.600	0.400	1.000	1.600	1.600	1.200	1.000	1.600	1.200	0.400	1.600	2.000	1.000	0.400
METAL	a003208	1.400	0.000	0.800	1.000	1.200	0.000	1.800	1.400	1.200	0.400	0.400	1.600	0.800	1.600	1.800
METAL	b006262	1.200	0.800	1.000	1.400	0.600	0.000	1.400	0.400	0.800	0.200	0.400	1.600	1.600	0.800	1.000
METAL	b007445	2.000	0.400	0.800	0.600	1.800	1.400	1.800	2.000	2.000	0.200	0.800	2.000	2.000	1.600	1.400
METAL	b010346	1.600	0.000	0.000	1.400	1.800	0.200	1.400	1.400	1.800	0.400	1.000	1.600	1.800	1.800	1.400
METAL	b014327	0.000	0.000	0.400	0.800	0.200	0.000	0.800	0.600	0.200	0.000	0.200	0.600	1.000	0.600	0.800
METAL	b017546	0.800	0.400	0.400	1.400	1.200	0.000	1.200	0.800	1.400	0.000	0.000	1.600	1.800	1.000	0.600
METAL	b019571	1.600	0.200	1.000	1.200	1.800	0.200	1.400	1.600	1.400	0.400	0.400	1.600	1.600	1.400	1.400
METAL	f002408	1.000	0.000	1.200	1.200	1.000	0.400	1.600	1.000	1.400	0.600	1.800	1.800	1.400	1.200	1.400
METAL	f000530	1.000	0.200	0.200	1.200	1.400	0.000	1.600	1.400	1.200	0.400	0.800	1.200	1.600	1.600	1.600
METAL	f005072	0.600	0.400	0.600	0.800	0.600	0.200	0.800	1.200	0.600	0.000	0.400	0.400	1.000	0.400	0.200
RAPHIPHOP	a000293	1.600	0.600	0.200	1.800	2.000	0.000	1.800	1.600	1.000	0.600	0.600	2.000	2.000	0.200	1.000
RAPHIPHOP	a000827	1.200	0.800	0.600	1.000	1.200	0.000	1.400	1.600	1.000	0.600	0.400	1.400	0.600	0.200	0.800
RAPHIPHOP	a000897	1.600	0.600	1.000	1.400	1.800	0.000	1.400	2.000	2.000	0.200	0.200	1.800	2.000	1.600	1.400
RAPHIPHOP	a002740	1.400	0.400	0.400	1.200	1.600	0.000	1.400	1.200	1.400	0.200	0.200	1.600	1.600	1.200	1.600
RAPHIPHOP	a009059	1.200	0.400	0.200	1.000	1.400	0.000	1.600	0.000	1.000	0.200	0.400	0.200	1.600	0.600	0.400
RAPHIPHOP	b010144	1.200	0.400	0.200	1.400	1.800	0.000	2.000	1.800	1.400	1.000	0.200	1.600	2.000	1.800	1.800
RAPHIPHOP	b011546	1.400	0.600	0.000	1.400	1.800	0.000	1.600	1.600	1.600	0.600	0.400	2.000	1.400	1.200	1.800
RAPHIPHOP	b010454	1.600	0.400	1.200	2.000	1.800	0.000	1.800	1.400	2.000	0.200	0.600	1.800	1.800	1.600	2.000
RAPHIPHOP	b017461	0.800	0.400	1.000	2.000	1.800	0.000	1.800	1.800	1.400	1.000	1.000	1.800	2.000	1.600	2.000
RAPHIPHOP	b018747	1.800	0.200	0.000	2.000	2.000	0.000	2.000	2.000	2.000	0.400	0.000	2.000	2.000	1.800	1.800
ROCKROLL	b001069	1.000	0.400	0.800	1.000	0.600	0.600	1.400	0.600	1.400	0.200	0.600	1.200	1.600	1.000	1.400
ROCKROLL	b001751	1.000	0.600	0.000	0.600	0.200	0.000	0.200	0.600	0.800	0.400	0.800	0.800	0.800	0.800	1.000
ROCKROLL	b003625	0.200	0.200	0.600	1.000	0.600	0.000	0.600	0.600	0.400	0.000	0.000	0.800	1.000	1.200	1.200
ROCKROLL	b004162	1.000	0.200	0.200	0.800	1.000	0.000	1.000	1.000	0.800	0.200	0.200	1.000	1.000	0.800	0.800
ROCKROLL	b004353	1.600	0.000	0.000	1.800	1.600	0.000	1.000	0.400	0.800	0.200	0.000	0.200	2.000	1.200	0.600
ROCKROLL	b006456	1.400	0.200	2.000	1.600	1.200	0.000	1.600	1.400	1.600	0.000	0.600	2.000	1.400	2.000	1.600
ROCKROLL	b009365	1.000	0.200	1.000	1.000	2.000	0.000	0.600	0.600	1.000	0.400	0.400	1.400	1.600	1.600	1.000
ROCKROLL	b008878	1.400	0.200	0.400	1.400	1.000	0.000	0.800	1.400	1.000	0.400	0.200	1.000	0.800	0.400	0.200
ROCKROLL	b011553	0.400	0.400	0.400	0.600	0.800	0.000	0.800	0.800	0.800	0.000	0.200	0.400	0.800	0.600	0.200
ROCKROLL	e017211	0.000	0.000	0.000	0.000	0.200	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
ROMANTIC	d004429	2.000	1.400	0.600	1.800	2.000	1.800	1.600	2.000	2.000	1.000	0.600	2.000	2.000	2.000	2.000
ROMANTIC	d001688	0.600	0.000	0.000	0.200	0.400	0.200	0.400	0.200	0.600	0.000	0.000	0.600	0.200	0.000	0.000
ROMANTIC	d004908	1.200	0.400	0.000	2.000	1.600	1.200	1.400	1.400	1.200	0.400	0.200	1.400	1.600	0.800	1.200
ROMANTIC	d007929	1.000	0.400	0.400	1.600	1.400	1.600	1.000	1.000	1.600	0.000	0.600	1.400	1.600	1.200	1.000
ROMANTIC	d011624	1.600	0.000	0.400	1.000	1.800	0.800	1.200	1.000	1.600	0.000	0.000	1.600	1.800	0.000	0.600
ROMANTIC	d016855	0.800	0.800	0.000	1.000	1.400	0.800	1.200	1.000	1.000	0.000	0.200	0.800	1.000	0.800	1.000
ROMANTIC	d014946	1.400	0.400	0.400	1.400	1.400	1.600	0.400	1.000	1.600	0.000	0.600	1.400	1.600	1.400	1.200
ROMANTIC	d014940	1.200	0.400	0.000	1.600	1.600	0.800	1.600	1.200	1.800	0.200	0.200	1.800	1.800	0.600	1.000
ROMANTIC	d018672	0.200	0.000	0.200	0.000	0.200	0.000	0.000	0.200	0.200	0.000	0.000	0.400	0.200	0.400	0.000
ROMANTIC	d017019	0.400	0.600	0.000	0.000	0.400	0.600	0.600	0.800	0.200	0.200	0.000	0.400	0.600	0.000	0.200

download these results as csv

Raw Scores

The raw data derived from the Evalutron 6000 human evaluations are located on the 2009:Audio Music Similarity and Retrieval Raw Data page.

Metadata and Distance Space Evaluation

The following reports provide evaluation statistics based on analysis of the distance space and metadata matches and include:

Neighbourhood clustering by artist, album and genre
Artist-filtered genre clustering
How often the triangular inequality holds
Statistics on 'hubs' (tracks similar to many tracks) and orphans (tracks that are not similar to any other tracks at N results).

Reports

ANO = Anonymous
BF1 = Benjamin Fields (chr12)
BF2 = Benjamin Fields (mfcc10)
BSWH1 = Dmitry Bogdanov, Joan Serrà, Nicolas Wack, and Perfecto Herrera (clas)
BSWH2 = Dmitry Bogdanov, Joan Serrà, Nicolas Wack, and Perfecto Herrera (hybrid)
CL1 = Chuan Cao, Ming Li
CL2 = Chuan Cao, Ming Li
GT = George Tzanetakis
LR = Thomas Lidy, Andreas Rauber]
ME1 = François Maillet, Douglas Eck (mlp)
ME2 = François Maillet, Douglas Eck (sda)
PS1 = Tim Pohle, Dominik Schnitzer (2007)
PS2 = Tim Pohle, Dominik Schnitzer (2009)
SH1 = Stephan Hübler
SH2 = Stephan Hübler

Run Times

Participant	Machine	Runtime (dd:hh:mm)
ANO	ALE	Feat/Dist 00:00:22/00:00:04
BF1	ALE	10:00:00
BF2	ALE	10:00:00
BSWH1	ALE	Feat/Dist 00:20:06/00:00:22
BSWH2	ALE	Feat/Dist 00:20:06/00:01:12
CL1	FAST3	Lost - HD Failure
CL2	FAST3	Lost - HD Failure
GT	ALE	Feat/Dist 00:00:23/00:00:20
LR	ALE	Feat/Dist 00:05:09/00:02:05
ME1	ALE	Feat/Dist 00:16:10/00:12:37
ME2	ALE	Feat/Dist 00:16:08/02:23:27
PS1	ALE	Feat/Dist 00:00:42/00:03:51
PS2	ALE	Feat/Dist 00:03:40/00:05:56
SH1	ALE	00:04:06
SH2	ALE	00:05:50

download these results as csv

@@ Line 12: / Line 12: @@
 ==== Team ID ====
-'''ANO''' = [[Anonymous]]<br />
+'''ANO''' = [https://www.music-ir.org/mirex/abstracts/2009/ANO_train_simi.pdf Anonymous]<br />
-'''BF1''' = [[Benjamin Fields (chr12)]]<br />
+'''BF1''' = [https://www.music-ir.org/mirex/abstracts/2009/BF.pdf Benjamin Fields (chr12)]<br />
-'''BF2''' = [[Benjamin Fields (mfcc10)]]<br />
+'''BF2''' = [https://www.music-ir.org/mirex/abstracts/2009/BF.pdf Benjamin Fields (mfcc10)]<br />
-'''BSWH1''' = [[Dmitry Bogdanov, Joan Serr├á, Nicolas Wack, and Perfecto Herrera (clas)]]<br />
+'''BSWH1''' = [https://www.music-ir.org/mirex/abstracts/2009/MIREX2009-sim-BSWH1-BSWH2.pdf Dmitry Bogdanov, Joan Serrà, Nicolas Wack, Perfecto Herrera (clas)]<br />
-'''BSWH2''' = [[Dmitry Bogdanov, Joan Serr├á, Nicolas Wack, and Perfecto Herrera (hybrid)]]<br />
+'''BSWH2''' = [https://www.music-ir.org/mirex/abstracts/2009/MIREX2009-sim-BSWH1-BSWH2.pdf Dmitry Bogdanov, Joan Serrà, Nicolas Wack, Perfecto Herrera (hybrid)]<br />
-'''CL1''' = [[Chuan Cao, Ming Li]]<br />
+'''CL1''' = [https://www.music-ir.org/mirex/abstracts/2009/CL.pdf Chuan Cao, Ming Li]<br />
-'''CL2''' = [[Chuan Cao, Ming Li]]<br />
+'''CL2''' = [https://www.music-ir.org/mirex/abstracts/2009/CL.pdf Chuan Cao, Ming Li]<br />
-'''GT''' = [[George Tzanetakis]]<br />
+'''GT''' = [https://www.music-ir.org/mirex/abstracts/2009/GTfinal.pdf George Tzanetakis]<br />
-'''LR''' = [[Thomas Lidy, Andreas Rauber]]<br />
+'''LR''' = [https://www.music-ir.org/mirex/abstracts/2009/LR.pdf Thomas Lidy, Andreas Rauber]<br />
-'''ME1'''  = [[Fran├ºois Maillet, Douglas Eck (mlp)]]<br />
+'''ME1'''  = [https://www.music-ir.org/mirex/abstracts/2009/ME.pdf François Maillet, Douglas Eck (mlp)]<br />
-'''ME2'''  = [[Fran├ºois Maillet, Douglas Eck (sda)]]<br />
+'''ME2'''  = [https://www.music-ir.org/mirex/abstracts/2009/ME.pdf François Maillet, Douglas Eck (sda)]<br />
-'''PS1''' = [[Tim Pohle, Dominik Schnitzer (2007)]]<br />
+'''PS1''' = [https://www.music-ir.org/mirex/abstracts/2009/PS.pdf Tim Pohle, Dominik Schnitzer (2007)]<br />
-'''PS2''' = [[Tim Pohle, Dominik Schnitzer (2009)]]<br />
+'''PS2''' = [https://www.music-ir.org/mirex/abstracts/2009/PS.pdf Tim Pohle, Dominik Schnitzer (2009)]<br />
-'''SH1''' = [[Stephan H├╝bler]]<br />
+'''SH1''' = [https://www.music-ir.org/mirex/abstracts/2009/SH.pdf Stephan Hübler]<br />
-'''SH2''' = [[Stephan H├╝bler]]<br />
+'''SH2''' = [https://www.music-ir.org/mirex/abstracts/2009/SH.pdf Stephan Hübler]<br />
 ====Broad Categories====
@@ Line 37: / Line 37: @@
 '''Broad''' = Has a range from 0 (failure) to 2 (perfection) as each query/candidate pair is scored with either NS=0, SS=1 or VS=2. <br />
-==Overall Summary Results==
+==Human Evaluation==
+===Overall Summary Results===
-<csv p=3>ams/evalutron/summary_evalutron.csv</csv>
+<csv p=3>2009/ams/evalutron/summary_evalutron.csv</csv>
-==Friedman's Tests==
+===Friedman's Tests===
-===Friedman's Test (FINE Scores)===
+====Friedman's Test (FINE Scores)====
 The Friedman test was run in MATLAB against the '''Fine''' summary data over the 100 queries.<br />
 Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
-<csv p=3>ams/evalutron/evalutron.fine.friedman.tukeyKramerHSD.csv</csv>
+<csv p=3>2009/ams/evalutron/evalutron.fine.friedman.tukeyKramerHSD.csv</csv>
-https://music-ir.org/mirex/2009/results/ams/evalutron/small.evalutron.fine.friedman.tukeyKramerHSD.png
+https://music-ir.org/mirex/results/2009/ams/evalutron/small.evalutron.fine.friedman.tukeyKramerHSD.png
-===Friedman's Test (BROAD Scores)===
+====Friedman's Test (BROAD Scores)====
 The Friedman test was run in MATLAB against the '''BROAD''' summary data over the 100 queries.<br />
 Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
-<csv p=3>ams/evalutron/evalutron.cat.friedman.tukeyKramerHSD.csv</csv>
+<csv p=3>2009/ams/evalutron/evalutron.cat.friedman.tukeyKramerHSD.csv</csv>
-https://music-ir.org/mirex/2009/results/ams/evalutron/small.evalutron.cat.friedman.tukeyKramerHSD.png
+https://music-ir.org/mirex/results/2009/ams/evalutron/small.evalutron.cat.friedman.tukeyKramerHSD.png
-==Summary Results by Query==
+===Summary Results by Query===
-===FINE Scores===
+====FINE Scores====
 These are the mean FINE scores per query assigned by Evalutron graders. The FINE scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0.0 and 10.0. A perfect score would be 10. Genre labels have been included for reference.
-<csv>ams/evalutron/fine_scores.csv</csv>
+<csv p=3>2009/ams/evalutron/fine_scores.csv</csv>
-===BROAD Scores===
+====BROAD Scores====
 These are the mean BROAD scores per query assigned by Evalutron graders. The BROAD scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 (not similar) and 2 (very similar). A perfect score would be 2. Genre labels have been included for reference.
-<csv p=3>ams/evalutron/cat_scores.csv</csv>
+<csv p=3>2009/ams/evalutron/cat_scores.csv</csv>
-==Anonymized Metadata==
-[https://www.music-ir.org/mirex2007/results/anonymizedAudioSim07metaData.csv Anonymized Metadata]<br />
-==Raw Scores==
-The raw data derived from the Evalutron 6000 human evaluations are located on the [[Audio Music Similarity and Retrieval Raw Data]] page.
-==Individual Performance Reports==
+===Raw Scores===
-=== Team ID ===
+The raw data derived from the Evalutron 6000 human evaluations are located on the [[2009:Audio Music Similarity and Retrieval Raw Data]] page.
-'''ANO''' = [https://music-ir.org/mirex/2009/results/ams/statistics/ANO/report.txt Anonymous]<br />
+==Metadata and Distance Space Evaluation==
-'''BF1''' = [https://music-ir.org/mirex/2009/results/ams/statistics/BF1/report.txt Benjamin Fields (chr12)]<br />
+The following reports provide evaluation statistics based on analysis of the distance space and metadata matches and  include:
-'''BF2''' = [https://music-ir.org/mirex/2009/results/ams/statistics/BF2/report.txt Benjamin Fields (mfcc10)]<br />
+* Neighbourhood clustering by artist, album and genre
-'''BSWH1''' = [https://music-ir.org/mirex/2009/results/ams/statistics/BSWH1/report.txt Dmitry Bogdanov, Joan Serr├á, Nicolas Wack, and Perfecto Herrera (clas)]<br />
+* Artist-filtered genre clustering
-'''BSWH2''' = [https://music-ir.org/mirex/2009/results/ams/statistics/BSWH2/report.txt Dmitry Bogdanov, Joan Serr├á, Nicolas Wack, and Perfecto Herrera (hybrid)]<br />
+* How often the triangular inequality holds
-'''CL1''' = [https://music-ir.org/mirex/2009/results/ams/statistics/CL1/report.txt Chuan Cao, Ming Li]<br />
+* Statistics on 'hubs' (tracks similar to many tracks) and orphans (tracks that are not similar to any other tracks at N results).
-'''CL2''' = [https://music-ir.org/mirex/2009/results/ams/statistics/CL2/report.txt Chuan Cao, Ming Li]<br />
-'''GT''' = [https://music-ir.org/mirex/2009/results/ams/statistics/GT/report.txt George Tzanetakis]<br />
+=== Reports ===
-'''LR''' = [https://music-ir.org/mirex/2009/results/ams/statistics/LR/report.txt Thomas Lidy, Andreas Rauber]]<br />
-'''ME1'''  = [https://music-ir.org/mirex/2009/results/ams/statistics/ME1/report.txt Fran├ºois Maillet, Douglas Eck (mlp)]<br />
+'''ANO''' = [https://music-ir.org/mirex/results/2009/ams/statistics/ANO/report.txt Anonymous]<br />
-'''ME2'''  = [https://music-ir.org/mirex/2009/results/ams/statistics/ME2/report.txt Fran├ºois Maillet, Douglas Eck (sda)]<br />
+'''BF1''' = [https://music-ir.org/mirex/results/2009/ams/statistics/BF1/report.txt Benjamin Fields (chr12)]<br />
-'''PS1''' = [https://music-ir.org/mirex/2009/results/ams/statistics/PS1/report.txt Tim Pohle, Dominik Schnitzer (2007)]<br />
+'''BF2''' = [https://music-ir.org/mirex/results/2009/ams/statistics/BF2/report.txt Benjamin Fields (mfcc10)]<br />
-'''PS2''' = [https://music-ir.org/mirex/2009/results/ams/statistics/PS2/report.txt Tim Pohle, Dominik Schnitzer (2009)]<br />
+'''BSWH1''' = [https://music-ir.org/mirex/results/2009/ams/statistics/BSWH1/report.txt Dmitry Bogdanov, Joan Serrà, Nicolas Wack, and Perfecto Herrera (clas)]<br />
-'''SH1''' = [https://music-ir.org/mirex/2009/results/ams/statistics/SH1/report.txt Stephan H├╝bler]<br />
+'''BSWH2''' = [https://music-ir.org/mirex/results/2009/ams/statistics/BSWH2/report.txt Dmitry Bogdanov, Joan Serrà, Nicolas Wack, and Perfecto Herrera (hybrid)]<br />
-'''SH2''' = [https://music-ir.org/mirex/2009/results/ams/statistics/SH2/report.txt Stephan H├╝bler]<br />
+'''CL1''' = [https://music-ir.org/mirex/results/2009/ams/statistics/CL1/report.txt Chuan Cao, Ming Li]<br />
+'''CL2''' = [https://music-ir.org/mirex/results/2009/ams/statistics/CL2/report.txt Chuan Cao, Ming Li]<br />
+'''GT''' = [https://music-ir.org/mirex/results/2009/ams/statistics/GT/report.txt George Tzanetakis]<br />
+'''LR''' = [https://music-ir.org/mirex/results/2009/ams/statistics/LR/report.txt Thomas Lidy, Andreas Rauber]]<br />
+'''ME1'''  = [https://music-ir.org/mirex/results/2009/ams/statistics/ME1/report.txt François Maillet, Douglas Eck (mlp)]<br />
+'''ME2'''  = [https://music-ir.org/mirex/results/2009/ams/statistics/ME2/report.txt François Maillet, Douglas Eck (sda)]<br />
+'''PS1''' = [https://music-ir.org/mirex/results/2009/ams/statistics/PS1/report.txt Tim Pohle, Dominik Schnitzer (2007)]<br />
+'''PS2''' = [https://music-ir.org/mirex/results/2009/ams/statistics/PS2/report.txt Tim Pohle, Dominik Schnitzer (2009)]<br />
+'''SH1''' = [https://music-ir.org/mirex/results/2009/ams/statistics/SH1/report.txt Stephan Hübler]<br />
+'''SH2''' = [https://music-ir.org/mirex/results/2009/ams/statistics/SH2/report.txt Stephan Hübler]<br />
+== Run Times ==
+<csv>2009/ams/audiosim.runtime.csv</csv>

Difference between revisions of "2009:Audio Music Similarity and Retrieval Results"

Latest revision as of 15:31, 23 July 2010

Contents

Introduction

General Legend

Team ID

Broad Categories

Understanding Summary Measures

Human Evaluation

Overall Summary Results

Friedman's Tests

Friedman's Test (FINE Scores)

Friedman's Test (BROAD Scores)

Summary Results by Query

FINE Scores

BROAD Scores

Raw Scores

Metadata and Distance Space Evaluation

Reports

Run Times

Navigation menu

Views

Personal tools

MIREX by Year

Results by Year

Account Request

Search

Navigation

Tools