Difference between revisions of "2008:Query-by-Singing/Humming Results"

Latest revision as of 16:26, 23 July 2010

Introduction

These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the 2008:Query by Singing/Humming page.

Task Descriptions

Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. See ESAC Data Homepage for more information about the Essen Collection. The query database consists of 2797 sung queries.

Task 2 Goto Task 2 Results: The second subtask is the same as last year too. In the second subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.

General Legend

Team ID

JL1 = J-S. R. Jang 1
JL2 = J-S. R. Jang 2
JL3 = J-S. R. Jang 3
JL4 = J-S. R. Jang 4
RK = M. Ryynänen, A. Klapuri
LW1 = L. Wang 1
LW2 = L. Wang 2
LW3 = L. Wang 3
WL1 = X. Wu, M. Li 1
WL2 = X. Wu, M. Li 2

Task 1 Results

Task 1 Overall Results

songID	JL1	JL2	JL3	JL4	LW1	RK	WL1	WL2
Avg. MRR	0.81	0.69	0.82	0.72	0.93	0.93	0.92	0.91

download these results as csv

Task 1 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
JL1	JL2	0.2791	1.7708	3.2626	TRUE
JL1	JL3	-2.1063	-0.6146	0.8772	FALSE
JL1	JL4	-0.3147	1.1771	2.6688	FALSE
JL1	LW1	-4.5334	-3.0417	-1.5499	TRUE
JL1	RK	-4.3980	-2.9062	-1.4145	TRUE
JL1	WL1	-4.5751	-3.0833	-1.5916	TRUE
JL1	WL2	-3.6272	-2.1354	-0.6437	TRUE
JL2	JL3	-3.8772	-2.3854	-0.8937	TRUE
JL2	JL4	-2.0855	-0.5938	0.8980	FALSE
JL2	LW1	-6.3043	-4.8125	-3.3207	TRUE
JL2	RK	-6.1688	-4.6771	-3.1853	TRUE
JL2	WL1	-6.3459	-4.8542	-3.3624	TRUE
JL2	WL2	-5.3980	-3.9063	-2.4145	TRUE
JL3	JL4	0.2999	1.7917	3.2834	TRUE
JL3	LW1	-3.9188	-2.4271	-0.9353	TRUE
JL3	RK	-3.7834	-2.2917	-0.7999	TRUE
JL3	WL1	-3.9605	-2.4688	-0.9770	TRUE
JL3	WL2	-3.0126	-1.5208	-0.0291	TRUE
JL4	LW1	-5.7105	-4.2188	-2.7270	TRUE
JL4	RK	-5.5751	-4.0833	-2.5916	TRUE
JL4	WL1	-5.7522	-4.2604	-2.7687	TRUE
JL4	WL2	-4.8043	-3.3125	-1.8207	TRUE
LW1	RK	-1.3563	0.1354	1.6272	FALSE
LW1	WL1	-1.5334	-0.0417	1.4501	FALSE
LW1	WL2	-0.5855	0.9063	2.3980	FALSE
RK	WL1	-1.6688	-0.1771	1.3147	FALSE
RK	WL2	-0.7209	0.7708	2.2626	FALSE
WL1	WL2	-0.5438	0.9479	2.4397	FALSE

download these results as csv

Task 1 Summary Results by Query Group

Query Group	JL1	JL2	JL3	JL4	LW1	RK	WL1	WL2
1	0.82	0.67	0.94	0.78	1.00	0.93	1.00	1.00
2	0.78	0.64	0.65	0.59	0.90	0.69	0.45	0.44
3	0.75	0.75	0.88	0.77	0.88	1.00	0.88	0.88
4	1.00	0.94	0.85	0.81	1.00	1.00	1.00	0.94
5	0.56	0.22	1.00	0.56	1.00	0.90	1.00	0.92
6	1.00	0.90	1.00	1.00	1.00	0.75	1.00	1.00
7	0.94	0.81	1.00	0.93	1.00	0.89	0.94	0.90
8	0.56	0.63	0.83	0.94	1.00	1.00	1.00	1.00
9	0.88	0.81	1.00	0.88	1.00	1.00	1.00	1.00
10	0.80	1.00	0.90	0.95	1.00	1.00	1.00	1.00
11	0.85	0.75	0.84	0.78	0.95	0.92	0.93	0.92
12	0.85	0.76	0.87	0.86	0.94	0.93	0.93	0.93
13	0.84	0.76	0.86	0.78	0.93	0.95	0.96	0.95
14	0.82	0.64	0.66	0.43	0.90	0.94	0.94	0.92
15	0.77	0.79	0.87	0.87	0.84	0.87	0.83	0.83
16	0.82	0.78	0.86	0.81	0.95	0.87	0.94	0.92
17	0.74	0.48	0.75	0.59	0.96	0.85	0.96	0.94
18	0.87	0.69	0.87	0.79	0.99	0.99	1.00	0.99
19	0.92	0.81	0.92	0.83	0.97	0.95	0.95	0.94
20	0.89	0.78	0.86	0.85	0.99	0.97	0.97	0.96
21	0.65	0.56	0.63	0.56	0.64	0.77	0.66	0.66
22	0.89	0.87	0.90	0.86	0.98	0.97	0.99	0.99
23	0.84	0.68	0.91	0.81	0.95	0.97	0.96	0.94
24	0.76	0.65	0.64	0.64	0.72	0.88	0.79	0.77
25	0.46	0.30	0.64	0.45	0.60	0.78	0.71	0.69
26	0.88	0.74	0.90	0.86	0.95	0.90	0.90	0.88
27	0.84	0.65	0.84	0.70	0.96	0.91	0.97	0.97
28	0.41	0.22	0.40	0.18	0.93	0.67	0.84	0.81
29	0.79	0.73	0.72	0.64	0.96	0.97	0.96	0.94
30	0.83	0.65	0.87	0.65	0.98	0.94	0.94	0.91
31	0.86	0.81	0.70	0.66	0.97	0.99	1.00	0.97
32	0.86	0.68	0.90	0.79	0.98	0.98	0.97	0.95
33	0.70	0.53	0.91	0.71	0.94	0.96	0.72	0.65
34	0.79	0.66	0.87	0.69	0.97	0.93	0.93	0.90
35	0.92	0.75	0.85	0.70	0.97	0.95	0.92	0.89
36	0.85	0.81	0.88	0.71	0.88	1.00	0.96	0.92
37	0.74	0.53	0.80	0.68	0.93	0.90	0.95	0.93
38	0.87	0.87	0.84	0.79	0.93	0.96	0.91	0.93
39	0.81	0.71	0.85	0.76	0.76	0.85	0.89	0.91
40	0.73	0.60	0.82	0.73	0.91	0.91	0.90	0.88
41	0.76	0.69	0.74	0.64	0.94	0.99	0.96	0.96
42	0.84	0.70	0.89	0.73	0.95	0.93	0.95	0.92
43	0.89	0.58	0.88	0.67	0.98	0.94	0.97	0.95
44	0.82	0.78	0.94	0.92	1.00	0.98	0.96	0.96
45	0.82	0.76	0.87	0.82	0.92	0.86	0.93	0.90
46	0.80	0.64	0.85	0.80	0.92	0.99	0.96	0.95
47	0.78	0.65	0.86	0.75	0.93	1.00	0.97	0.97
48	0.80	0.75	0.78	0.70	0.88	0.94	0.94	0.96

download these results as csv

Task 2 Results

In this subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.

Task 2 Overall Results

songID	JL1	JL2	JL3	JL4	LW2	LW3	RK	WL1	WL2
Avg. MRR	0.32	0.53	0.32	0.59	0.92	0.94	0.89	0.94	0.92

download these results as csv

Task 2 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
JL1	JL2	-1.9291	-0.9198	0.0895	FALSE
JL1	JL3	-1.0093	0.0000	1.0093	FALSE
JL1	JL4	-2.5282	-1.5189	-0.5096	TRUE
JL1	LW2	-4.9574	-3.9481	-2.9388	TRUE
JL1	LW3	-5.1650	-4.1557	-3.1463	TRUE
JL1	RK	-4.7169	-3.7075	-2.6982	TRUE
JL1	WL1	-5.2027	-4.1934	-3.1841	TRUE
JL1	WL2	-4.8961	-3.8868	-2.8775	TRUE
JL2	JL3	-0.0895	0.9198	1.9291	FALSE
JL2	JL4	-1.6084	-0.5991	0.4103	FALSE
JL2	LW2	-4.0376	-3.0283	-2.0190	TRUE
JL2	LW3	-4.2452	-3.2358	-2.2265	TRUE
JL2	RK	-3.7970	-2.7877	-1.7784	TRUE
JL2	WL1	-4.2829	-3.2736	-2.2643	TRUE
JL2	WL2	-3.9763	-2.9670	-1.9577	TRUE
JL3	JL4	-2.5282	-1.5189	-0.5096	TRUE
JL3	LW2	-4.9574	-3.9481	-2.9388	TRUE
JL3	LW3	-5.1650	-4.1557	-3.1463	TRUE
JL3	RK	-4.7169	-3.7075	-2.6982	TRUE
JL3	WL1	-5.2027	-4.1934	-3.1841	TRUE
JL3	WL2	-4.8961	-3.8868	-2.8775	TRUE
JL4	LW2	-3.4386	-2.4292	-1.4199	TRUE
JL4	LW3	-3.6461	-2.6368	-1.6275	TRUE
JL4	RK	-3.1980	-2.1887	-1.1794	TRUE
JL4	WL1	-3.6838	-2.6745	-1.6652	TRUE
JL4	WL2	-3.3772	-2.3679	-1.3586	TRUE
LW2	LW3	-1.2169	-0.2075	0.8018	FALSE
LW2	RK	-0.7687	0.2406	1.2499	FALSE
LW2	WL1	-1.2546	-0.2453	0.7640	FALSE
LW2	WL2	-0.9480	0.0613	1.0706	FALSE
LW3	RK	-0.5612	0.4481	1.4574	FALSE
LW3	WL1	-1.0470	-0.0377	0.9716	FALSE
LW3	WL2	-0.7404	0.2689	1.2782	FALSE
RK	WL1	-1.4952	-0.4858	0.5235	FALSE
RK	WL2	-1.1886	-0.1792	0.8301	FALSE
WL1	WL2	-0.7027	0.3066	1.3159	FALSE

download these results as csv

Task 2 Summary Results by Query Group

Query Group	JL1	JL2	JL3	JL4	LW2	LW3	RK	WL1	WL2
1	0.32	0.74	0.28	0.62	1.00	1.00	0.91	0.95	0.93
2	0.60	0.70	0.40	0.61	1.00	1.00	1.00	1.00	1.00
3	0.00	0.00	0.00	0.50	0.11	0.00	1.00	0.25	0.11
4	0.00	0.50	0.25	0.75	0.75	1.00	1.00	1.00	1.00
5	0.00	0.00	0.00	0.00	1.00	1.00	0.00	0.00	0.00
6	0.00	0.56	0.00	0.11	0.67	0.68	0.83	0.69	0.68
7	0.83	0.83	0.68	0.45	1.00	1.00	1.00	1.00	1.00
8	0.04	0.00	0.11	0.05	0.75	1.00	1.00	1.00	1.00
9	0.00	0.60	0.00	0.70	1.00	1.00	1.00	1.00	1.00
10	0.00	0.50	0.00	0.50	1.00	1.00	1.00	1.00	1.00
11	0.00	0.22	0.00	1.00	1.00	1.00	1.00	0.75	0.63
12	0.33	0.11	0.00	0.07	1.00	1.00	1.00	1.00	1.00
13	0.03	0.60	0.23	0.60	1.00	1.00	0.80	1.00	1.00
14	0.00	0.63	0.00	0.56	1.00	1.00	0.50	1.00	1.00
15	0.00	0.17	0.00	0.00	0.67	1.00	0.13	1.00	0.60
16	0.38	0.38	0.31	0.39	1.00	1.00	0.75	1.00	1.00
17	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
18	0.00	0.63	0.25	0.17	0.50	0.50	0.50	1.00	1.00
19	0.00	1.00	0.00	0.75	1.00	1.00	1.00	1.00	1.00
20	0.00	0.00	0.00	0.00	1.00	1.00	0.75	0.42	0.33
21	0.16	0.03	0.03	0.60	0.75	1.00	0.63	0.55	0.53
22	0.39	0.58	0.44	0.62	0.74	0.81	1.00	0.92	0.94
23	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
24	0.00	1.00	0.00	1.00	1.00	1.00	1.00	1.00	1.00
25	0.50	0.67	0.50	1.00	1.00	1.00	1.00	1.00	1.00
26	0.42	0.50	0.30	0.60	1.00	1.00	0.92	1.00	1.00
27	0.06	0.75	0.25	0.65	1.00	1.00	1.00	1.00	1.00
28	0.50	0.58	0.33	0.80	1.00	1.00	0.63	1.00	1.00
29	0.33	0.05	1.00	1.00	1.00	1.00	0.25	1.00	1.00
30	1.00	1.00	0.75	0.58	1.00	1.00	1.00	1.00	1.00
31	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
32	0.00	0.50	0.00	0.71	1.00	1.00	0.86	1.00	1.00
33	0.00	0.80	0.00	0.80	0.63	0.67	1.00	0.72	0.66
34	0.00	0.25	0.50	0.00	0.50	0.50	0.58	0.50	0.50
35	0.00	0.00	0.00	0.00	1.00	1.00	1.00	1.00	1.00
36	0.00	0.67	0.00	0.52	0.82	0.80	0.87	0.87	0.80
37	0.00	0.00	0.00	1.00	1.00	1.00	0.11	0.06	0.00
38	0.50	0.55	0.50	1.00	0.63	1.00	1.00	1.00	1.00
39	0.36	0.73	0.43	1.00	1.00	1.00	0.90	1.00	1.00
40	0.54	0.50	1.00	0.50	1.00	1.00	1.00	1.00	1.00
41	0.25	0.57	0.06	0.56	0.78	0.81	0.88	0.88	0.88
42	0.63	0.38	0.59	0.68	1.00	1.00	0.92	1.00	0.88
43	0.48	0.35	0.67	0.54	0.73	0.85	0.81	0.83	0.76
44	0.48	0.68	0.56	0.83	1.00	1.00	1.00	1.00	1.00
45	0.69	0.67	1.00	0.67	1.00	1.00	1.00	1.00	1.00
46	0.00	0.25	0.00	0.25	1.00	1.00	0.29	1.00	1.00
47	0.75	0.63	0.50	0.51	1.00	1.00	1.00	1.00	1.00
48	0.07	0.00	0.00	1.00	1.00	1.00	1.00	1.00	1.00
49	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
50	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
51	0.05	0.25	0.25	0.26	1.00	1.00	1.00	1.00	1.00
52	0.08	0.54	0.25	0.54	1.00	1.00	0.77	1.00	1.00
53	0.33	0.55	0.33	1.00	1.00	1.00	1.00	1.00	1.00
54	0.20	0.66	0.13	0.60	1.00	1.00	1.00	1.00	1.00
55	0.00	0.20	0.00	0.63	1.00	1.00	1.00	1.00	1.00
56	0.50	0.50	0.50	0.50	0.50	0.50	0.58	0.50	0.50
57	0.90	0.90	1.00	1.00	1.00	1.00	1.00	1.00	1.00
58	0.00	1.00	0.00	1.00	1.00	1.00	1.00	1.00	1.00
59	0.00	0.73	0.00	1.00	1.00	1.00	1.00	1.00	1.00
60	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
61	0.00	0.50	0.00	0.08	1.00	1.00	0.00	1.00	1.00
62	0.08	0.67	0.33	0.67	1.00	1.00	1.00	1.00	1.00
63	0.63	0.53	0.53	0.57	1.00	1.00	1.00	1.00	1.00
64	0.00	1.00	0.00	1.00	1.00	1.00	1.00	1.00	1.00
65	0.00	0.08	0.00	0.17	1.00	1.00	1.00	1.00	1.00
66	0.50	0.75	0.50	0.55	1.00	1.00	1.00	1.00	1.00
67	0.00	0.83	0.00	1.00	1.00	1.00	0.67	1.00	1.00
68	1.00	0.54	0.55	0.54	0.75	1.00	0.50	0.67	0.67
69	0.00	0.03	0.00	0.50	1.00	1.00	1.00	1.00	1.00
70	1.00	0.88	0.81	0.79	1.00	1.00	1.00	1.00	1.00
71	0.00	0.37	0.00	0.67	0.75	1.00	1.00	0.42	0.42
72	1.00	0.71	1.00	1.00	1.00	1.00	1.00	1.00	1.00
73	0.75	0.75	0.65	0.75	1.00	1.00	1.00	1.00	1.00
74	1.00	0.54	1.00	1.00	1.00	1.00	1.00	1.00	1.00
75	0.17	0.35	0.33	0.38	1.00	1.00	0.83	1.00	1.00
76	0.00	0.30	0.00	0.75	1.00	1.00	1.00	1.00	0.55
77	1.00	0.80	0.64	0.45	1.00	1.00	1.00	1.00	1.00
78	0.00	0.33	0.07	0.00	1.00	1.00	1.00	1.00	1.00
79	0.50	0.50	0.50	0.50	1.00	1.00	1.00	1.00	1.00
80	0.33	0.73	0.00	0.67	1.00	1.00	1.00	1.00	1.00
81	0.17	0.60	0.17	0.58	0.92	0.92	1.00	1.00	0.92
82	0.00	0.75	0.00	1.00	1.00	1.00	1.00	1.00	1.00
83	0.00	0.05	0.02	0.22	0.36	0.50	1.00	1.00	1.00
84	0.05	0.02	0.00	0.00	0.27	0.34	0.54	0.75	0.75
85	0.50	1.00	0.50	1.00	1.00	1.00	1.00	1.00	1.00
86	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
87	0.00	1.00	0.00	0.45	1.00	1.00	1.00	1.00	1.00
88	0.00	0.00	0.00	0.00	1.00	1.00	0.17	1.00	1.00
89	0.02	0.16	0.43	0.36	0.93	0.93	0.93	0.86	0.86
90	0.00	0.11	0.00	0.60	1.00	1.00	1.00	1.00	1.00
91	0.00	0.25	0.00	0.50	1.00	1.00	1.00	1.00	1.00
92	0.50	0.54	0.50	0.50	1.00	1.00	1.00	1.00	1.00
93	0.25	0.17	0.50	1.00	1.00	1.00	1.00	1.00	1.00
94	0.00	0.00	0.00	0.00	1.00	1.00	1.00	1.00	1.00
95	0.00	1.00	0.00	1.00	1.00	1.00	1.00	1.00	1.00
96	0.00	0.00	1.00	0.05	1.00	1.00	0.00	1.00	1.00
97	0.00	0.00	0.00	0.00	1.00	1.00	1.00	1.00	1.00
98	0.00	0.00	0.00	0.00	1.00	1.00	0.00	1.00	1.00
99	0.55	0.03	0.38	0.15	1.00	1.00	0.63	1.00	0.88
100	0.33	1.00	0.33	1.00	0.68	0.67	1.00	1.00	1.00
101	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
102	0.00	0.00	0.00	0.00	1.00	1.00	1.00	1.00	1.00
103	0.27	0.39	0.26	0.38	0.94	1.00	1.00	1.00	1.00
104	0.00	0.67	0.00	0.67	1.00	1.00	1.00	1.00	1.00
105	1.00	0.00	0.17	0.08	0.00	0.00	0.11	0.14	0.05
106	0.00	0.36	0.00	0.35	1.00	1.00	0.61	1.00	0.69

download these results as csv

Runtime Results

Participant	Task	Runtime (min'sec")	Machine
JL1	1	20'10"	FAST
JL2	1	417'36"	FAST
JL3	1	54'25"	FAST
JL4	1	297'32"	FAST2
LW1	1	143'40"	FAST2
RK	1	71'52"	cluster
WL1	1	31'03"	FAST
WL2	1	28'19"	FAST
JL1	2	2'08"	FAST2
JL2	2	54'05"	FAST2
JL3	2	7'32"	FAST2
JL4	2	59'23"	FAST2
LW2	2	561'42"	FAST2
LW3	2	62'39"	FAST2
RK	2	19'36"	cluster
WL1	2	8'30"	FAST
WL2	2	6'03"	FAST

download these results as csv

@@ Line 1: / Line 1: @@
-Runtime Results:
+==Introduction==
+These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the [[2008:Query by Singing/Humming]] page.
-<csv>qbsh.runtime.csv</csv>
+===Task Descriptions===
+'''Task 1 [[#Task 1 Results|Goto Task 1 Results]]''': The first subtask is the same as last year. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files.  See [http://www.esac-data.org/  ESAC Data Homepage] for more information about the Essen Collection. The query database consists of 2797 sung queries.
+'''Task 2 [[#Task 2 Results|Goto Task 2 Results]]''': The second subtask is the same as last year too. In the second subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.
+===General Legend===
+====Team ID====
+'''JL1''' = [https://www.music-ir.org/mirex/abstracts/2008/MIREX2008_QBSH_Davidson_abstract.pdf J-S. R. Jang 1]<br />
+'''JL2''' = [https://www.music-ir.org/mirex/abstracts/2008/MIREX2008_QBSH_Davidson_abstract.pdf J-S. R. Jang 2]<br />
+'''JL3''' = [https://www.music-ir.org/mirex/abstracts/2008/MIREX2008_QBSH_Davidson_abstract.pdf J-S. R. Jang 3]<br />
+'''JL4''' = [https://www.music-ir.org/mirex/abstracts/2008/MIREX2008_QBSH_Davidson_abstract.pdf J-S. R. Jang 4]<br />
+'''RK''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_ryynanen.pdf M. Ryynänen, A. Klapuri]<br />
+'''LW1''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_leiwang.pdf L. Wang 1]<br />
+'''LW2''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_leiwang.pdf L. Wang 2]<br />
+'''LW3''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_leiwang.pdf L. Wang 3]<br />
+'''WL1''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_wu.pdf X. Wu, M. Li 1]<br />
+'''WL2''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_wu.pdf X. Wu, M. Li 2]<br />
+===Task 1 Results===
+=====Task 1 Overall Results=====
+<csv>2008/qbsh/qbsh_task1_summary.csv</csv>
+====Task 1 Friedman's Test for Significant Differences====
+The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
+Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
+<csv>2008/qbsh/qbsh.task1.friedman_detailed.csv</csv>
+[[Image:2008_qbsh.task1.friedman.small.png]]
+====Task 1 Summary Results by Query Group====
+<csv>2008/qbsh/qbsh.task1.res.byQueryGroup.csv</csv>
+===Task 2 Results===
+In this subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.
+=====Task 2 Overall Results=====
+<csv>2008/qbsh/qbsh.task2.summary.csv</csv>
+====Task 2 Friedman's Test for Significant Differences====
+The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
+Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
+<csv>2008/qbsh/qbsh.task2.friedman_detailed.csv</csv>
+[[Image:2008_qbsh.task2.friedman.s.png]]
+====Task 2 Summary Results by Query Group====
+<csv>2008/qbsh/qbsh_task2_res_byQueryGroup.csv</csv>
+===Runtime Results===
+<csv>2008/qbsh.runtime.csv</csv>
+[[Category: Results]]

Difference between revisions of "2008:Query-by-Singing/Humming Results"

Latest revision as of 16:26, 23 July 2010

Contents

Introduction

Task Descriptions

General Legend

Team ID

Task 1 Results

Task 1 Overall Results

Task 1 Friedman's Test for Significant Differences

Task 1 Summary Results by Query Group

Task 2 Results

Task 2 Overall Results

Task 2 Friedman's Test for Significant Differences

Task 2 Summary Results by Query Group

Runtime Results

Navigation menu

Views

Personal tools

MIREX by Year

Results by Year

Account Request

Search

Navigation

Tools