2006:Symbolic Melodic Similarity Results

Introduction

These are the results for the 2006 running of the Symbolic Melodic Similarity task set. For background information about this task set please refer to the 2006:Symbolic Melodic Similarity page.

Each system was given a query and returned the 10 most melodically similar songs from a given collection where the collections were RISM (monophonic; 10,000), Karoke (polyphonic; 1,000), Mixed (polyphonic; 15,741). Then, for each query, the returned results from all participants were grouped and were evaluated by human graders, each query being evaluated by 3 different graders with two scores (using the Evalutron 6000 system). Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 10).

Evalutron 6000 Summary Data

Number of evaluators = 20
Number of evaluations per query/candidate pair = 3
Number of queries per grader = 15
Ave. size of the candidate lists = 15
Ave. number of query/candidate pairs evaluated per grader: 225
Number of queries (across all subtasks = 17

General Legend

Team ID

Prefix R = RISM collection, K = Karaoke collection, M = Polyphonic collection

FH = Pascal Ferraro and Pierre Hanna
NM = Kjell Lemström, Niko Mikkilä, Veli Mäkinen and Esko Ukkonen
RT = Rainer Typke, Frans Wiering and Remco C. Veltkamp
KF = Klaus Frieler
AU = Alexandra Uitdenbogerd

Broad Categories

NS = Not Similar
SS = Somewhat Similar
VS = Very Similar

Table Headings

ADR = Average Dynamic Recall
NRGB = Normalize Recall at Group Boundaries
AP = Average Precision (non-interpolated)
PND = Precision at N Documents

Calculating Summary Measures

Fine⁽¹⁾ = Sum of fine-grained human similarity decisions (0-10).
PSum⁽¹⁾ = Sum of human broad similarity decisions: NS=0, SS=1, VS=2.
WCsum⁽¹⁾ = 'World Cup' scoring: NS=0, SS=1, VS=3 (rewards Very Similar).
SDsum⁽¹⁾ = 'Stephen Downie' scoring: NS=0, SS=1, VS=4 (strongly rewards Very Similar).
Greater0⁽¹⁾ = NS=0, SS=1, VS=1 (binary relevance judgement).
Greater1⁽¹⁾ = NS=0, SS=0, VS=1 (binary relevance judgement using only Very Similar).

⁽¹⁾Normalized to the range 0 to 1.

Overall Summary Results

Visualizations

Rainer Typke has created a series of 2006:Symbolic Melodic Similarity Graphs that help us visualize the results.
Rainer Typke has also created a set of detailed representations of the results that is definitely with exploring at [http://rainer.typke.org/mirex06.0.html].

Task I: RISM Overall Summary

	RFH	RRT	RKF2	RAU	RKF3	RNM2	RNM1	RKF1
ADR	0.707	0.715	0.670	0.577	0.555	0.541	0.268	0.000
NRGB	0.622	0.626	0.568	0.515	0.466	0.484	0.277	0.000
AP	0.623	0.607	0.594	0.461	0.391	0.393	0.167	0.000
PND	0.526	0.547	0.504	0.441	0.426	0.403	0.250	0.000
Fine	0.490	0.488	0.478	0.433	0.418	0.292	0.208	0.145
Psum	0.594	0.592	0.567	0.522	0.500	0.364	0.258	0.172
WCsum	0.544	0.544	0.519	0.457	0.450	0.344	0.250	0.119
SDsum	0.519	0.521	0.494	0.425	0.425	0.335	0.246	0.092
Greater0	0.744	0.733	0.711	0.717	0.650	0.422	0.283	0.333
Greater1	0.444	0.450	0.422	0.328	0.333	0.306	0.233	0.011

download these results as csv

Task I: RISM Runtime Data

Team ID		Machine	Run-time(seconds)
AU	indexing	beer 4	33
AU	query	beer 4	31
FH	query	beer 4	807
KF	indexing	black	210
KF1	query	black	2880
KF2	query	black	2220
KF3	optip	black	3960
NM1	query	beer 6	68
NM2	query	beer 6	188
RT	query	beer 4	59

download these results as csv

Task IIa: Karaoke Overall Summary

	KRT	KAU	KFH	KNM2	KNM1
ADR	0.819	0.378	0.150	0.000	0.000
NRGB	0.764	0.333	0.150	0.000	0.000
AP	0.875	0.363	0.100	0.000	0.000
PND	0.833	0.333	0.100	0.000	0.000
Fine	0.267	0.207	0.153	0.112	0.105
Psum	0.327	0.260	0.200	0.147	0.120
WCsum	0.289	0.211	0.151	0.098	0.082
SDsum	0.270	0.187	0.127	0.073	0.063
Greater0	0.440	0.407	0.347	0.293	0.233
Greater1	0.213	0.113	0.053	0.007	0.000

download these results as csv

Task IIa: Karaoke Runtime Data

Team ID		Machine	Run-time(seconds)
AU	indexing	beer 4	397
AU	query	beer 4	5
FH	query	beer 4	1338
NM1	query	beer 6	386
NM2	query	beer 6	1875
RT	query	beer 4	32

download these results as csv

Task IIb: Mixed Polyphonic Overall Summary

	MRT	MAU	MFH	MNM1	MNH2
ADR	0.784	0.587	0.218	0.070	0.000
NRGB	0.806	0.516	0.174	0.040	0.000
AP	0.903	0.517	0.170	0.032	0.004
PND	0.903	0.549	0.187	0.044	0.000
Fine	0.663	0.489	0.283	0.128	0.114
Psum	0.772	0.558	0.342	0.153	0.153
WCsum	0.752	0.535	0.294	0.115	0.113
SDsum	0.742	0.524	0.271	0.096	0.093
Greater0	0.833	0.628	0.483	0.267	0.272
Greater1	0.711	0.489	0.200	0.039	0.033

download these results as csv

Task IIb: Mixed Polyphonic Runtime Data

Team ID		Machine	Run-time(seconds)
AU	indexing	beer 4	2785
AU	query	beer 4	51
FH	query	beer 4	14440
NM1	both	beer 6	3271
NM2	both	beer 6	16314
RT	query	beer 4	108

download these results as csv

Task I: RISM Collection Summary Results

There is an error with this data set...please stand by.

ADR:
queryID	RFH	RRT	RKF2	RAU	RKF3	RNM2	RNM1	RKF1
qr000001	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
qr000002	0.768	0.802	0.518	0.144	0.298	0.572	0.103	0.000
qr000003	0.786	0.840	0.707	0.740	0.642	0.000	0.555	0.000
qr000004	0.930	0.795	0.866	0.707	0.609	0.817	0.222	0.000
qr000005	0.870	0.870	0.960	0.960	0.870	0.870	0.613	0.000
qr000006	0.887	0.984	0.967	0.912	0.912	0.984	0.113	0.000
Ave. ADR score:	0.707	0.715	0.670	0.577	0.555	0.541	0.268	0.000
NRGB:
queryID	RFH	RRT	RKF2	RAU	RKF3	RNM2	RNM1	RKF1
qr000001	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
qr000002	0.630	0.630	0.296	0.296	0.333	0.370	0.185	0.000
qr000003	0.625	0.688	0.563	0.625	0.500	0.000	0.375	0.000
qr000004	0.917	0.615	0.771	0.531	0.406	0.708	0.260	0.000
qr000005	0.840	0.840	0.920	0.920	0.840	0.840	0.560	0.000
qr000006	0.719	0.984	0.859	0.719	0.719	0.984	0.281	0.000
Ave. NRGB score:	0.622	0.626	0.568	0.515	0.466	0.484	0.277	0.000
AP:
queryID	RFH	RRT	RKF2	RAU	RKF3	RNM2	RNM1	RKF1
qr000001	0.500	0.500	0.500	0.333	0.000	0.000	0.000	0.000
qr000002	0.604	0.514	0.222	0.059	0.137	0.383	0.050	0.000
qr000003	0.500	0.604	0.375	0.446	0.384	0.000	0.250	0.000
qr000004	0.750	0.548	0.683	0.375	0.375	0.500	0.186	0.000
qr000005	0.600	0.600	0.800	0.800	0.600	0.600	0.419	0.000
qr000006	0.785	0.875	0.986	0.750	0.847	0.875	0.098	0.000
Ave. AP score:	0.623	0.607	0.594	0.461	0.391	0.393	0.167	0.000
PND:
queryID	RFH	RRT	RKF2	RAU	RKF3	RNM2	RNM1	RKF1
qr000001	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
qr000002	0.556	0.556	0.222	0.222	0.333	0.444	0.222	0.000
qr000003	0.500	0.625	0.375	0.500	0.500	0.000	0.250	0.000
qr000004	0.750	0.625	0.750	0.375	0.375	0.500	0.375	0.000
qr000005	0.600	0.600	0.800	0.800	0.600	0.600	0.400	0.000
qr000006	0.750	0.875	0.875	0.750	0.750	0.875	0.250	0.000
Ave. PND score:	0.526	0.547	0.504	0.441	0.426	0.403	0.250	0.000
Fine:
queryID	RFH	RRT	RKF2	RAU	RKF3	RNM2	RNM1	RKF1
qr000001	0.322	0.265	0.359	0.309	0.230	0.042	0.007	0.154
qr000002	0.566	0.569	0.407	0.423	0.452	0.358	0.153	0.214
qr000003	0.401	0.489	0.352	0.396	0.355	0.017	0.191	0.076
qr000004	0.581	0.544	0.554	0.357	0.419	0.390	0.341	0.111
qr000005	0.387	0.386	0.475	0.483	0.379	0.276	0.278	0.140
qr000006	0.681	0.675	0.720	0.632	0.672	0.671	0.278	0.178
Ave. Fine Score:	0.490	0.488	0.478	0.433	0.418	0.292	0.208	0.145
Psum:
queryID	RFH	RRT	RKF2	RAU	RKF3	RNM2	RNM1	RKF1
qr000001	0.417	0.317	0.433	0.383	0.267	0.067	0.033	0.200
qr000002	0.717	0.700	0.483	0.533	0.533	0.467	0.200	0.250
qr000003	0.500	0.617	0.417	0.483	0.450	0.050	0.250	0.100
qr000004	0.667	0.650	0.633	0.417	0.517	0.450	0.400	0.100
qr000005	0.483	0.450	0.550	0.567	0.450	0.350	0.350	0.183
qr000006	0.783	0.817	0.883	0.750	0.783	0.800	0.317	0.200
Ave. Psum Score:	0.594	0.592	0.567	0.522	0.500	0.364	0.258	0.172
WCsum:
queryID	RFH	RRT	RKF2	RAU	RKF3	RNM2	RNM1	RKF1
qr000001	0.322	0.244	0.344	0.289	0.189	0.044	0.022	0.133
qr000002	0.689	0.667	0.422	0.456	0.467	0.444	0.200	0.178
qr000003	0.444	0.556	0.378	0.433	0.422	0.033	0.233	0.067
qr000004	0.633	0.600	0.600	0.378	0.456	0.433	0.400	0.067
qr000005	0.433	0.411	0.500	0.489	0.400	0.333	0.333	0.122
qr000006	0.744	0.789	0.867	0.700	0.767	0.778	0.311	0.144
Ave. WCsum Score:	0.544	0.544	0.519	0.457	0.450	0.344	0.250	0.119
SDsum:
queryID	RRT	RFH	RKF2	RAU	RKF3	RNM2	RNM1	RKF1
qr000001	0.208	0.275	0.300	0.242	0.150	0.033	0.017	0.100
qr000002	0.650	0.675	0.392	0.417	0.433	0.433	0.200	0.142
qr000003	0.525	0.417	0.358	0.408	0.408	0.025	0.225	0.050
qr000004	0.575	0.617	0.583	0.358	0.425	0.425	0.400	0.050
qr000005	0.392	0.408	0.475	0.450	0.375	0.325	0.325	0.092
qr000006	0.775	0.725	0.858	0.675	0.758	0.767	0.308	0.117
Ave. SDsum Score:	0.521	0.519	0.494	0.425	0.425	0.335	0.246	0.092
Greater0:
queryID	RFH	RRT	RAU	RKF2	RKF3	RNM2	RNM1	RKF1
qr000001	0.700	0.533	0.667	0.700	0.500	0.133	0.067	0.400
qr000002	0.800	0.800	0.767	0.667	0.733	0.533	0.200	0.467
qr000003	0.667	0.800	0.633	0.533	0.533	0.100	0.300	0.200
qr000004	0.767	0.800	0.533	0.733	0.700	0.500	0.400	0.200
qr000005	0.633	0.567	0.800	0.700	0.600	0.400	0.400	0.367
qr000006	0.900	0.900	0.900	0.933	0.833	0.867	0.333	0.367
Ave. greater0 Score:	0.744	0.733	0.717	0.711	0.650	0.422	0.283	0.333
Greater1:
queryID	RRT	RFH	RKF2	RKF3	RAU	RNM2	RNM1	RKF1
qr000001	0.100	0.133	0.167	0.033	0.100	0.000	0.000	0.000
qr000002	0.600	0.633	0.300	0.233	0.300	0.400	0.200	0.033
qr000003	0.433	0.333	0.300	0.367	0.333	0.000	0.200	0.000
qr000004	0.500	0.567	0.533	0.333	0.300	0.400	0.400	0.000
qr000005	0.333	0.333	0.400	0.300	0.333	0.300	0.300	0.000
qr000006	0.733	0.667	0.833	0.733	0.600	0.733	0.300	0.033
Ave. greater1 Score:	0.450	0.444	0.422	0.333	0.328	0.306	0.233	0.011

download these results as csv

Task IIa: Karaoke Collection Summary Results

ADR:
queryID	KRT	KAU	KFH	KNM2	KNM1
qk000001	1.000	0.889	0.000	0.000	0.000
qk000002	0.542	0.000	0.000	0.000	0.000
qk000003	0.556	0.000	0.000	0.000	0.000
qk000004	1.000	0.000	0.000	0.000	0.000
qk000005	1.000	1.000	0.750	0.000	0.000
Ave. ADR score:	0.820	0.378	0.150	0.000	0.000
NRGB:
queryID	KRT	KAU	KFH	KNM2	KNM1
qk000001	1.000	0.667	0.000	0.000	0.000
qk000002	0.375	0.000	0.000	0.000	0.000
qk000003	0.444	0.000	0.000	0.000	0.000
qk000004	1.000	0.000	0.000	0.000	0.000
qk000005	1.000	1.000	0.750	0.000	0.000
Ave. NRGB score:	0.764	0.333	0.150	0.000	0.000
AP:
queryID	KRT	KAU	KFH	KNM2	KNM1
qk000001	1.000	0.667	0.000	0.000	0.000
qk000002	0.707	0.000	0.000	0.000	0.000
qk000003	0.667	0.067	0.000	0.000	0.000
qk000004	1.000	0.083	0.000	0.000	0.000
qk000005	1.000	1.000	0.500	0.000	0.000
Ave. AP score:	0.875	0.363	0.100	0.000	0.000
PND:
queryID	KRT	KAU	KFH	KNM2	KNM1
qk000001	1.000	0.667	0.000	0.000	0.000
qk000002	0.500	0.000	0.000	0.000	0.000
qk000003	0.667	0.000	0.000	0.000	0.000
qk000004	1.000	0.000	0.000	0.000	0.000
qk000005	1.000	1.000	0.500	0.000	0.000
Ave. PND score:	0.833	0.333	0.100	0.000	0.000
Fine
queryID	KRT	KAU	KFH	KNM2	KNM1
qk000001	0.337	0.268	0.108	0.079	0.116
qk000002	0.297	0.094	0.106	0.048	0.109
qk000003	0.222	0.189	0.169	0.177	0.073
qk000004	0.267	0.199	0.152	0.088	0.078
qk000005	0.214	0.283	0.230	0.168	0.149
Ave. Fine Score:	0.267	0.207	0.153	0.112	0.105
Psum:
queryID	KRT	KAU	KFH	KNM2	KNM1
qk000001	0.417	0.317	0.150	0.100	0.133
qk000002	0.350	0.117	0.117	0.067	0.133
qk000003	0.300	0.250	0.233	0.233	0.050
qk000004	0.317	0.267	0.217	0.117	0.117
qk000005	0.250	0.350	0.283	0.217	0.167
Ave. Psum Score:	0.327	0.260	0.200	0.147	0.120
WCsum:
queryID	KRT	KAU	KFH	KNM2	KNM1
qk000001	0.378	0.278	0.100	0.067	0.089
qk000002	0.322	0.078	0.078	0.044	0.089
qk000003	0.244	0.189	0.189	0.156	0.033
qk000004	0.278	0.211	0.167	0.078	0.078
qk000005	0.222	0.300	0.222	0.144	0.122
Ave. WCsum Score:	0.289	0.211	0.151	0.098	0.082
SDsum:
queryID	KRT	KAU	KFH	KNM2	KNM1
qk000001	0.358	0.258	0.075	0.050	0.067
qk000002	0.308	0.058	0.058	0.033	0.067
qk000003	0.217	0.158	0.167	0.117	0.025
qk000004	0.258	0.183	0.142	0.058	0.058
qk000005	0.208	0.275	0.192	0.108	0.100
Ave. SDsum Score:	0.270	0.187	0.127	0.073	0.063
Greater0:
queryID	KRT	KAU	KFH	KNM2	KNM1
qk000001	0.533	0.433	0.300	0.200	0.267
qk000002	0.433	0.233	0.233	0.133	0.267
qk000003	0.467	0.433	0.367	0.467	0.100
qk000004	0.433	0.433	0.367	0.233	0.233
qk000005	0.333	0.500	0.467	0.433	0.300
Ave. greater0 Score:	0.440	0.407	0.347	0.293	0.233
Greater1:
queryID	KRT	KAU	KFH	KNM1	KNM2
qk000001	0.300	0.200	0.000	0.000	0.000
qk000002	0.267	0.000	0.000	0.000	0.000
qk000003	0.133	0.067	0.100	0.000	0.000
qk000004	0.200	0.100	0.067	0.000	0.000
qk000005	0.167	0.200	0.100	0.033	0.000
Ave. greater1 Score:	0.213	0.113	0.053	0.007	0.000

download these results as csv

Task IIb: Mixed Polyphonic Collection Summary Results

ADR:
queryID	MRT	MAU	MFH	MNM1	MNM2
qm000001	0.569	0.158	0.000	0.000	0.000
qm000002	0.592	0.322	0.000	0.408	0.000
qm000003	0.793	0.663	0.000	0.000	0.000
qm000004	0.893	0.602	0.012	0.000	0.000
qm000005	1.000	1.000	1.000	0.000	0.000
qm000006	0.857	0.776	0.293	0.010	0.000
Ave. ADR score:	0.784	0.587	0.218	0.070	0.000
NRGB:
queryID	MRT	MAU	MFH	MNM1	MNM2
qm000001	0.583	0.139	0.000	0.000	0.000
qm000002	0.772	0.517	0.000	0.228	0.000
qm000003	0.857	0.510	0.000	0.000	0.000
qm000004	0.836	0.383	0.012	0.000	0.000
qm000005	0.909	0.909	0.909	0.000	0.000
qm000006	0.879	0.637	0.121	0.010	0.000
Ave. NRGB score:	0.806	0.516	0.174	0.040	0.000
AP:
queryID	MRT	MAU	MFH	MNM1	MNM2
qm000001	1.000	0.167	0.000	0.000	0.000
qm000002	0.833	0.403	0.000	0.167	0.021
qm000003	1.000	0.524	0.000	0.000	0.000
qm000004	0.778	0.397	0.012	0.000	0.000
qm000005	0.909	0.909	0.909	0.000	0.000
qm000006	0.900	0.700	0.100	0.025	0.000
Ave. AP score:	0.903	0.517	0.170	0.032	0.004
PND:
queryID	MRT	MAU	MFH	MNM1	MNM2
qm000001	1.000	0.167	0.000	0.000	0.000
qm000002	0.833	0.500	0.000	0.167	0.000
qm000003	1.000	0.571	0.000	0.000	0.000
qm000004	0.778	0.444	0.111	0.000	0.000
qm000005	0.909	0.909	0.909	0.000	0.000
qm000006	0.900	0.700	0.100	0.100	0.000
Ave. AP score:	0.903	0.549	0.187	0.044	0.000
Fine:
queryID	MRT	MAU	MFH	MNM1	MNM2
qm000001	0.502	0.162	0.130	0.097	0.101
qm000002	0.479	0.306	0.108	0.203	0.088
qm000003	0.670	0.438	0.087	0.105	0.073
qm000004	0.621	0.430	0.226	0.097	0.235
qm000005	0.866	0.880	0.870	0.147	0.078
qm000006	0.838	0.719	0.279	0.122	0.110
Ave. Fine Score:	0.663	0.489	0.283	0.128	0.114
Psum:
queryID	MRT	MAU	MFH	MNM1	MNM2
qm000001	0.600	0.217	0.167	0.083	0.150
qm000002	0.583	0.367	0.133	0.233	0.150
qm000003	0.783	0.483	0.133	0.117	0.083
qm000004	0.717	0.450	0.267	0.117	0.250
qm000005	1.000	1.000	1.000	0.167	0.100
qm000006	0.950	0.833	0.350	0.200	0.183
Ave. Psum Score:	0.772	0.558	0.342	0.153	0.153
WCsum:
queryID	MRT	MAU	MFH	MNM1	MNM2
qm000001	0.567	0.167	0.122	0.056	0.100
qm000002	0.533	0.344	0.089	0.189	0.122
qm000003	0.767	0.456	0.089	0.078	0.056
qm000004	0.700	0.422	0.189	0.078	0.200
qm000005	1.000	1.000	1.000	0.122	0.067
qm000006	0.944	0.822	0.278	0.167	0.133
Ave. WCsum Score:	0.752	0.535	0.294	0.115	0.113
SDsum:
queryID	MRT	MAU	MFH	MNM1	MNM2
qm000001	0.550	0.142	0.100	0.042	0.075
qm000002	0.508	0.333	0.067	0.167	0.108
qm000003	0.758	0.442	0.067	0.058	0.042
qm000004	0.692	0.408	0.150	0.058	0.175
qm000005	1.000	1.000	1.000	0.100	0.050
qm000006	0.942	0.817	0.242	0.150	0.108
Ave. SDsum Score:	0.742	0.524	0.271	0.096	0.093
Greater0:
queryID	MRT	MAU	MFH	MNM2	MNM1
qm000001	0.700	0.367	0.300	0.300	0.167
qm000002	0.733	0.433	0.267	0.233	0.367
qm000003	0.833	0.567	0.267	0.167	0.233
qm000004	0.767	0.533	0.500	0.400	0.233
qm000005	1.000	1.000	1.000	0.200	0.300
qm000006	0.967	0.867	0.567	0.333	0.300
Ave. greater0 Score:	0.833	0.628	0.483	0.272	0.267
Greater1:
queryID	MRT	MAU	MFH	MNM1	MNM2
qm000001	0.500	0.067	0.033	0.000	0.000
qm000002	0.433	0.300	0.000	0.100	0.067
qm000003	0.733	0.400	0.000	0.000	0.000
qm000004	0.667	0.367	0.033	0.000	0.100
qm000005	1.000	1.000	1.000	0.033	0.000
qm000006	0.933	0.800	0.133	0.100	0.033
Ave. greater1 Score:	0.711	0.489	0.200	0.039	0.033

download these results as csv

Raw Scores

The raw data derived from the Evalutron 6000 human evaluations are located on the 2006:Symbolic Melodic Similarity Raw Data page.