2009:Query-by-Tapping Results

Introduction

These are the results for the 2008 running of the Query-by-tappingn task. For background information about this task set please refer to the 2009:Query_by_Tapping page.

Task Descriptions

Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:

• Jang's Dataset Roger Jang's MIR-QBT: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries.
• Hsiao's Dataset Show Hsiao's QBT_symbolic: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries.

Task 2 Goto Task 2 Results:The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.

General Legend

Team ID

CSJ = Chun-Ta Chen and Jyh-Shing Roger Jang
HAFR = Pierre Hanna, Julien Allali, Pascal Ferraro,Matthias Robine
HL = Shu-Jen Show Hsiao,Tyne Liang

Task 1a, Jang's Dataset Results

Task 1a Overall Results

	CSJ	HAFR	HL
Simple Count	0.87	0.88	0.89
MRR	0.78	0.77	0.79
Total count	890	890	890

download these results as csv

Task 1a Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
HL	CSJ	-0.1863	-0.0184	0.1496	FALSE
HL	HAFR	-0.1716	-0.0037	0.1643	FALSE
CSJ	HAFR	-0.1533	0.0147	0.1827	FALSE

download these results as csv

MRR Method:

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
HL	CSJ	-0.2337	-0.0074	0.2190	FALSE
HL	HAFR	-0.0646	0.1618	0.3881	FALSE
CSJ	HAFR	-0.0572	0.1691	0.3955	FALSE

download these results as csv

Task 1a Summary Results by Query Group

Simple Hit/Miss Counting

	CSJ	HAFR	HL
1	1	0.92	0.92
2	1	1	1
3	1	1	1
4	1	0	1
5	1	1	1
6	1	1	0.50
7	1	1	1
8	1	1	1
9	1	1	1
10	1	1	1
11	0.50	1	0.50
12	1	1	1
13	1	1	1
14	1	1	1
15	1	1	1
16	1	1	0.95
17	0.87	0.93	1
18	0.77	0.85	0.69
19	0.38	0.50	1
20	1	1	1
21	1	1	1
22	1	0.50	0.50
23	0	0	0
24	1	1	1
25	1	1	1
26	0	0	1
27	1	0.94	1
28	1	1	1
29	1	1	1
30	0.58	0.75	0.67
31	1	1	1
32	1	1	0.67
33	0.78	0.75	0.94
34	0.92	0.92	0.92
35	1	1	1
36	1	1	1
37	0	1	1
38	1	1	1
39	1	1	1
40	1	1	1
41	0.83	0.75	0.92
42	1	1	1
43	0.80	1	1
44	0.75	0.75	0.50
45	1	1	0.50
46	1	1	1
47	1	1	1
48	1	1	1
49	1	1	1
50	1	1	1
51	0.60	0.80	0.60
52	1	1	1
53	0.80	0.60	0.60
54	1	1	1
55	1	0.98	1
56	1	1	1
57	1	0.80	1
58	1	1	1
59	0.20	0.60	0.20
60	0.43	1	0.71
61	1	1	1
62	1	1	1
63	1	1	1
64	0.50	0.50	0.50
65	1	1	1
66	1	1	0.50
67	1	1	1
68	0.67	0.78	0.61
69	0.82	0.82	0.94
70	1	1	0.75
71	1	0	1
72	0.83	0.92	0.75
73	0.93	0.86	0.93
74	1	1	1
75	0.83	1	0.83
76	1	1	1
77	0	0	1
78	1	0	0
79	1	0.50	0.75
80	1	1	1
81	1	1	1
82	1	1	1
83	1	1	1
84	1	1	1
85	0.86	0.86	0.86
86	1	1	1
87	0.58	0.75	0.75
88	1	1	1
89	1	0.67	1
90	0.50	1	1
91	1	1	1
92	0.25	0.25	0.50
93	1	1	1
94	0.50	0.67	0.67
95	1	1	1
96	1	1	1
97	1	1	1
98	0.50	0.50	0.50
99	0.88	0.88	0.81
100	1	1	1
101	0.83	1	0.83
102	0.88	1	1
103	1	1	1
104	1	1	1
105	1	1	0.96
106	0.73	0.68	0.50
107	0.94	0.94	1
108	0.69	0.56	0.88
109	0.94	0.81	0.94
110	1	1	1
111	1	1	0.78
112	1	1	1
113	1	1	1
114	1	1	1
115	0.80	0.60	0.60
116	0.75	0.75	0.75
117	1	1	1
118	1	1	1
119	0.67	0.67	0.67
120	0.67	0.67	1
121	0.33	0.67	0.33
122	0.33	0.67	0.67
123	0.67	0.33	0.33
124	0.33	0.33	0.33
125	1	0.33	0.67
126	0.67	0.67	1
127	1	1	1
128	0.67	0.67	0.67
129	0.50	0.50	0
130	1	0.50	1
131	1	1	1
132	1	1	1
133	1	1	1
134	1	1	1
135	1	1	1
136	1	1	1

download these results as csv

MRR Method

	CSJ	HAFR	HL
1	0.94	0.88	0.88
2	1	1	1
3	1	1	1
4	1	0	1
5	1	1	1
6	0.78	0.88	0.50
7	1	1	1
8	0.75	0.75	0.75
9	1	0.25	1
10	1	1	1
11	0.50	0.60	0.50
12	1	1	1
13	1	1	0.50
14	1	0.94	0.94
15	1	1	1
16	0.93	0.82	0.35
17	0.76	0.93	1
18	0.71	0.64	0.59
19	0.28	0.30	0.50
20	1	1	1
21	0.56	1	1
22	1	0.50	0.50
23	0	0	0
24	1	0.50	1
25	0.90	0.90	0.89
26	0	0	0.14
27	0.97	0.76	0.88
28	0.83	1	1
29	1	1	1
30	0.52	0.75	0.53
31	1	0.33	1
32	1	1	0.67
33	0.50	0.72	0.91
34	0.92	0.92	0.92
35	1	1	1
36	0.88	0.78	0.88
37	0	1	0.25
38	1	1	1
39	1	1	1
40	0.96	0.96	1
41	0.63	0.58	0.60
42	1	1	1
43	0.80	0.88	0.84
44	0.75	0.63	0.50
45	0.60	0.42	0.17
46	1	0.71	0.92
47	1	0.57	1
48	0.41	1	0.78
49	1	1	0.75
50	0.73	0.44	0.78
51	0.33	0.63	0.27
52	1	1	1
53	0.67	0.60	0.60
54	1	0.50	1
55	0.98	0.96	0.98
56	1	1	1
57	0.90	0.80	1
58	1	1	0.89
59	0.03	0.43	0.20
60	0.43	0.59	0.52
61	1	1	1
62	1	0.96	0.94
63	0.33	0.11	1
64	0.38	0.28	0.38
65	1	1	1
66	1	1	0.50
67	1	1	0.50
68	0.56	0.55	0.56
69	0.77	0.67	0.77
70	1	0.78	0.50
71	1	0	0.50
72	0.69	0.82	0.68
73	0.87	0.82	0.89
74	1	0.80	1
75	0.71	0.85	0.83
76	1	1	1
77	0	0	0.12
78	1	0	0
79	0.65	0.38	0.75
80	0.73	1	1
81	1	1	1
82	1	1	1
83	1	0.88	1
84	1	0.50	1
85	0.63	0.16	0.86
86	1	1	1
87	0.33	0.69	0.45
88	1	1	1
89	0.83	0.42	1
90	0.25	1	1
91	1	1	1
92	0.25	0.13	0.13
93	1	1	1
94	0.42	0.29	0.36
95	0.91	0.94	1
96	1	1	1
97	1	1	0.88
98	0.50	0.50	0.50
99	0.88	0.79	0.70
100	1	1	1
101	0.83	0.57	0.83
102	0.88	1	1
103	1	1	1
104	0.98	1	1
105	0.94	0.94	0.96
106	0.60	0.59	0.48
107	0.73	0.68	0.82
108	0.48	0.46	0.69
109	0.90	0.65	0.89
110	1	0.97	1
111	0.89	0.82	0.78
112	0.78	0.94	0.94
113	1	1	0.90
114	0.80	0.80	0.90
115	0.48	0.31	0.25
116	0.53	0.54	0.63
117	1	1	1
118	0.47	1	1
119	0.38	0.67	0.67
120	0.67	0.67	1
121	0.33	0.40	0.17
122	0.08	0.50	0.39
123	0.38	0.33	0.33
124	0.33	0.33	0.33
125	0.70	0.17	0.67
126	0.23	0.67	0.67
127	1	1	1
128	0.67	0.67	0.67
129	0.50	0.25	0
130	0.58	0.50	1
131	0.67	1	1
132	0.75	1	0.67
133	0.17	0.14	0.20
134	1	1	1
135	1	1	1
136	0.50	1	1

download these results as csv

Task 1a Summary Results by Query

Simple Hit/Miss Counting [1]

MRR Method [2]

Task 1b, Hsiao's Dataset Results

Task 1b Overall Results

	CSJ	HAFR	HL
Simple Count	0.82	0.88	0.90
MRR	0.66	0.70	0.69
Total count	410.00	410.00	410.00

download these results as csv

Task 1b Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
HL	HAFR	-0.1073	0.0352	0.1776	FALSE
HL	CSJ	0.0451	0.1875	0.3299	TRUE
HAFR	CSJ	0.0099	0.1523	0.2948	TRUE

download these results as csv

MRR Method:

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
HAFR	HL	-0.1002	0.1328	0.3659	FALSE
HAFR	CSJ	-0.0260	0.2070	0.4401	FALSE
HL	CSJ	-0.1588	0.0742	0.3073	FALSE

download these results as csv

Task 1b Summary Results by Query Group

Simple Hit/Miss Counting

	CSJ	HAFR	HL
1	1	1	1
2	NaN	NaN	NaN
3	1	1	1
4	1	1	1
5	1	1	1
6	1	1	1
7	1	1	1
8	1	1	1
9	1	1	1
10	0.67	0.67	0.67
11	1	1	1
12	0.75	1	0.50
13	1	1	1
14	1	1	1
15	NaN	NaN	NaN
16	1	1	1
17	1	1	1
18	1	1	1
19	1	1	1
20	1	1	1
21	0.67	1	1
22	1	1	1
23	1	0.83	0.83
24	0.67	0.67	0.67
25	0	1	1
26	0	0	0
27	1	1	1
28	NaN	NaN	NaN
29	1	1	1
30	1	1	1
31	NaN	NaN	NaN
32	1	1	1
33	1	1	1
34	1	1	1
35	NaN	NaN	NaN
36	0	0	0
37	0.50	0	0.50
38	1	0.86	1
39	1	1	1
40	0	0.50	0.50
41	1	1	1
42	0.67	1	1
43	1	1	1
44	0	1	0.50
45	1	1	1
46	1	1	1
47	1	1	1
48	1	1	1
49	1	0.67	1
50	1	1	1
51	1	1	1
52	1	1	1
53	0.89	0.89	0.89
54	0.33	1	1
55	0.50	1	1
56	1	0.50	1
57	1	1	1
58	NaN	NaN	NaN
59	1	1	1
60	1	1	1
61	0.75	0.75	1
62	0	0.50	0.50
63	1	1	1
64	NaN	NaN	NaN
65	1	1	1
66	NaN	NaN	NaN
67	1	1	1
68	NaN	NaN	NaN
69	0.40	1	1
70	0.20	1	1
71	NaN	NaN	NaN
72	1	1	1
73	1	1	0.50
74	NaN	NaN	NaN
75	1	1	1
76	0.40	1	0.40
77	NaN	NaN	NaN
78	1	1	1
79	0.50	1	1
80	1	1	0
81	0	1	1
82	0	1	1
83	1	1	1
84	1	1	1
85	0	0	1
86	1	1	1
87	1	1	1
88	0.80	0.60	0.80
89	0.80	1	0.20
90	1	1	1
91	1	1	1
92	1	1	0.80
93	0.50	1	1
94	1	0.75	1
95	0.67	0.67	0.67
96	1	1	1
97	0	0	0
98	1	1	1
99	1	1	1
100	1	1	1
101	1	1	1
102	1	1	1
103	NaN	NaN	NaN
104	1	1	1
105	1	1	1
106	0	0.33	0
107	NaN	NaN	NaN
108	1	1	1
109	1	1	1
110	0	1	1
111	1	1	1
112	0.67	1	0.67
113	1	1	1
114	0.50	0.50	0.50
115	NaN	NaN	NaN
116	0	0	1
117	1	1	1
118	1	1	1
119	0	0	0
120	0.67	0.67	0.67
121	1	0.67	1
122	1	1	1
123	1	1	1
124	1	1	1
125	1	0	1
126	1	1	1
127	1	1	1
128	1	1	1
129	1	1	1
130	0	0	0
131	0	0	1
132	1	1	1
133	0.33	1	0.67
134	0.50	0.50	0.50
135	1	1	1
136	1	1	1
137	1	1	1
138	1	1	1
139	0.60	1	1
140	0	0	1
141	1	1	1
142	1	1	1
143	1	1	1

download these results as csv

MRR Method

	CSJ	HAFR	HL
1	0.56	1	1
2	NaN	NaN	NaN
3	0.88	0.83	0.83
4	0.50	0.75	0.50
5	1	1	1
6	1	0.50	1
7	1	1	1
8	1	1	1
9	1	1	1
10	0.44	0.50	0.67
11	1	1	1
12	0.75	0.81	0.50
13	1	1	1
14	1	0.42	1
15	NaN	NaN	NaN
16	1	1	1
17	1	1	1
18	1	1	1
19	1	1	1
20	0.83	1	0.61
21	0.45	1	1
22	1	1	0.88
23	0.88	0.47	0.64
24	0.67	0.67	0.67
25	0	0.75	0.15
26	0	0	0
27	0.50	0.67	0.44
28	NaN	NaN	NaN
29	1	1	0.75
30	1	1	0.75
31	NaN	NaN	NaN
32	1	0.50	1
33	0.33	0.56	0.50
34	0.78	0.61	0.58
35	NaN	NaN	NaN
36	0	0	0
37	0.13	0	0.25
38	0.89	0.26	0.90
39	0.18	1	1
40	0	0.07	0.08
41	1	1	0.50
42	0.50	0.83	0.75
43	0.75	0.75	0.88
44	0	0.42	0.10
45	0.33	0.56	0.22
46	0.78	1	1
47	1	1	0.75
48	1	1	1
49	0.53	0.14	0.28
50	1	1	1
51	0.10	0.13	0.17
52	1	1	1
53	0.89	0.83	0.89
54	0.11	0.37	0.32
55	0.06	1	0.75
56	0.42	0.25	0.67
57	0.80	0.90	0.60
58	NaN	NaN	NaN
59	1	0.50	0.50
60	1	0.63	1
61	0.50	0.75	0.54
62	0	0.50	0.50
63	1	0.67	0.50
64	NaN	NaN	NaN
65	0.69	0.49	1
66	NaN	NaN	NaN
67	0.58	0.20	0.67
68	NaN	NaN	NaN
69	0.05	0.85	1
70	0.20	0.41	0.90
71	NaN	NaN	NaN
72	1	1	0.60
73	1	1	0.50
74	NaN	NaN	NaN
75	1	1	1
76	0.20	0.85	0.30
77	NaN	NaN	NaN
78	0.67	0.83	0.67
79	0.13	0.58	0.88
80	0.13	0.33	0
81	0	0.35	0.29
82	0	0.50	0.25
83	1	1	1
84	1	0.63	1
85	0	0	0.25
86	1	1	1
87	1	1	1
88	0.67	0.43	0.47
89	0.38	0.69	0.02
90	0.83	0.78	0.53
91	0.88	1	0.88
92	1	1	0.80
93	0.50	1	1
94	0.33	0.16	0.29
95	0.67	0.67	0.67
96	1	1	1
97	0	0	0
98	1	0.75	1
99	1	1	1
100	1	1	1
101	1	1	0.80
102	1	1	1
103	NaN	NaN	NaN
104	0.33	1	1
105	0.35	0.33	0.40
106	0	0.11	0
107	NaN	NaN	NaN
108	1	1	1
109	0.33	0.33	0.33
110	0	0.10	0.17
111	0.50	0.50	0.13
112	0.50	0.78	0.20
113	1	1	1
114	0.50	0.17	0.50
115	NaN	NaN	NaN
116	0	0	0.60
117	0.75	0.75	0.75
118	0.83	0.67	0.33
119	0	0	0
120	0.21	0.50	0.20
121	0.56	0.67	0.71
122	0.65	0.88	0.43
123	0.79	0.78	1
124	0.70	0.70	0.70
125	0.19	0	0.23
126	1	1	1
127	0.50	0.88	0.50
128	0.56	0.78	0.50
129	1	1	1
130	0	0	0
131	0	0	0.29
132	1	0.75	0.63
133	0.03	0.48	0.15
134	0.50	0.06	0.07
135	1	1	0.78
136	0.56	1	1
137	1	1	1
138	0.23	0.38	0.33
139	0.35	0.44	0.80
140	0	0	0.50
141	0.56	0.69	0.42
142	0.50	0.50	0.39
143	1	1	1

download these results as csv

Task 1b Summary Results by Query

Simple Hit/Miss Counting [3]

MRR Method [4]

Task 2 Results

Task 2 Overall Results

	CSJ	HAFR	HL
Simple Count	0.86	0.79	0.84
MRR	0.77	0.66	0.73
Total count	890	890	880

download these results as csv

Task 2 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
CSJ	HL	-0.1404	0.0515	0.2434	FALSE
CSJ	HAFR	0.0103	0.2022	0.3941	TRUE
HL	HAFR	-0.0412	0.1507	0.3426	FALSE

download these results as csv

MRR Method:

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
CSJ	HL	-0.1350	0.1066	0.3482	FALSE
CSJ	HAFR	0.1702	0.4118	0.6533	TRUE
HL	HAFR	0.0636	0.3051	0.5467	TRUE

download these results as csv

Task 2 Summary Results by Query Group

Simple Hit/Miss Counting

	CSJ	HAFR	HL
1	1	0.69	0.77
2	1	1	1
3	1	1	1
4	1	0	1
5	1	1	1
6	1	1	0.75
7	1	1	1
8	1	1	1
9	1	1	1
10	1	1	1
11	0.50	1	1
12	1	1	1
13	1	1	1
14	1	1	1
15	1	1	1
16	1	0.90	0.85
17	0.80	0.93	1
18	0.77	0.62	0.69
19	0.38	0.75	1
20	1	1	1
21	1	1	1
22	1	1	0.50
23	0	0	0
24	1	1	1
25	1	1	1
26	0	0	1
27	1	0.71	0.88
28	1	0.67	1
29	1	1	1
30	0.67	0.67	0.75
31	1	0	1
32	1	1	1
33	0.71	0.59	0.82
34	0.83	0.92	0.92
35	1	1	1
36	1	1	1
37	0	1	1
38	1	1	1
39	1	1	1
40	1	0.86	0.93
41	0.75	0.83	0.92
42	1	1	1
43	0.80	0.90	0.90
44	0.75	0.75	0.50
45	1	1	0.50
46	1	0.83	1
47	1	1	1
48	1	1	1
49	1	1	0.50
50	1	1	1
51	0.60	0.80	0.80
52	1	1	1
53	0.80	0.20	0.60
54	1	1	1
55	1	0.84	0.98
56	1	1	1
57	1	0.80	0.80
58	1	1	1
59	0.20	0.40	0.20
60	0.43	0.57	0.57
61	1	0.82	0.91
62	0.92	0.75	0.83
63	1	0	1
64	0.50	0.50	0.50
65	1	1	1
66	1	1	0.50
67	1	1	1
68	0.56	0.56	0.61
69	0.82	0.64	0.91
70	1	0.75	0.75
71	1	0	1
72	0.83	0.92	0.75
73	0.86	0.71	0.93
74	1	1	1
75	0.83	0.83	0.83
76	1	1	1
77	0	0	1
78	1	0	0
79	1	0.75	0.75
80	1	1	1
81	1	1	1
82	1	1	1
83	1	1	1
84	1	1	1
85	0.86	0.43	0.71
86	1	1	1
87	0.58	0.67	0.67
88	1	1	1
89	1	1	1
90	0.50	1	1
91	1	1	1
92	0.25	0.25	0.25
93	1	1	1
94	0.50	0.83	0.67
95	1	0.88	1
96	1	1	1
97	1	1	1
98	0.50	0.50	0.50
99	0.81	0.88	0.75
100	1	1	1
101	0.83	1	0.83
102	0.88	1	1
103	1	1	1
104	0.96	0.89	0.89
105	1	0.81	0.89
106	0.64	0.73	0.45
107	0.94	0.72	0.89
108	0.69	0.44	0.81
109	0.88	0.81	0.75
110	0.93	0.93	0.87
111	1	1	1
112	1	1	0.88
113	0.86	0.71	0.86
114	1	0.60	0.60
115	0.80	0.60	0.40
116	0.50	0.75	0.75
117	1	1	1
118	1	1	1
119	0.67	0.67	1
120	1	0.67	1
121	1	1	1
122	0.33	0.67	1
123	0.67	0.67	1
124	0.33	0.33	0.50
125	0.67	1	1
126	1	1	1
127	1	1	0.67
128	0.67	0.67	0.67
129	0.50	0.50	0
130	1	0.50	1
131	1	1	1
132	1	1	1
133	1	0	0
134	1	1	1
135	1	1	1
136	1	0	0

download these results as csv

MRR Method

	CSJ	HAFR	HL
1	0.94	0.69	0.77
2	1	1	1
3	1	1	1
4	1	0	1
5	1	0.67	1
6	0.78	0.83	0.75
7	1	1	1
8	0.75	1	0.50
9	1	0.10	1
10	1	1	1
11	0.50	0.67	0.63
12	1	1	1
13	1	1	0.50
14	1	0.89	0.94
15	1	1	1
16	0.91	0.70	0.32
17	0.74	0.79	0.81
18	0.71	0.34	0.59
19	0.28	0.33	0.44
20	1	1	1
21	0.55	1	1
22	1	0.55	0.50
23	0	0	0
24	1	0.50	1
25	0.90	0.93	0.89
26	0	0	0.14
27	0.94	0.63	0.68
28	0.83	0.37	1
29	1	1	1
30	0.60	0.53	0.61
31	1	0	1
32	1	1	0.72
33	0.43	0.56	0.76
34	0.83	0.92	0.92
35	1	1	1
36	0.88	0.58	0.88
37	0	1	0.25
38	1	1	1
39	1	1	1
40	0.96	0.82	0.93
41	0.52	0.47	0.60
42	1	1	1
43	0.73	0.78	0.83
44	0.75	0.75	0.50
45	0.58	0.33	0.17
46	1	0.63	0.92
47	1	1	1
48	0.42	0.43	0.78
49	1	1	0.25
50	0.73	0.51	0.78
51	0.32	0.70	0.47
52	1	1	0.63
53	0.67	0.20	0.60
54	1	0.33	1
55	0.98	0.75	0.98
56	1	1	1
57	1	0.80	0.80
58	1	0.91	0.89
59	0.03	0.09	0.20
60	0.43	0.40	0.50
61	1	0.82	0.91
62	0.92	0.69	0.79
63	0.33	0	1
64	0.38	0.50	0.38
65	1	1	1
66	1	1	0.50
67	1	0.25	0.50
68	0.50	0.44	0.56
69	0.76	0.49	0.75
70	1	0.54	0.50
71	1	0	0.50
72	0.69	0.80	0.61
73	0.86	0.61	0.93
74	1	0.90	1
75	0.71	0.83	0.75
76	1	1	1
77	0	0	0.12
78	1	0	0
79	0.65	0.38	0.75
80	0.73	1	1
81	1	1	1
82	1	1	1
83	1	0.88	1
84	1	0.50	1
85	0.66	0.22	0.62
86	1	1	1
87	0.33	0.50	0.37
88	1	0.75	1
89	0.83	0.58	0.83
90	0.25	0.56	1
91	1	0.87	1
92	0.25	0.06	0.08
93	1	1	1
94	0.42	0.49	0.36
95	0.91	0.70	1
96	1	0.50	1
97	1	0.79	0.88
98	0.50	0.50	0.50
99	0.81	0.79	0.69
100	1	1	1
101	0.83	0.52	0.83
102	0.88	1	1
103	1	0.89	1
104	0.90	0.78	0.85
105	0.97	0.81	0.89
106	0.58	0.58	0.40
107	0.78	0.67	0.66
108	0.47	0.33	0.57
109	0.83	0.62	0.65
110	0.88	0.90	0.87
111	1	0.81	1
112	0.88	1	0.88
113	0.76	0.71	0.64
114	0.80	0.44	0.37
115	0.30	0.35	0.22
116	0.50	0.58	0.35
117	1	0.75	1
118	0.47	1	1
119	0.38	0.50	1
120	0.83	0.67	1
121	0.83	0.83	0.75
122	0.08	0.25	0.25
123	0.40	0.44	0.57
124	0.33	0.17	0.50
125	0.67	0.28	0.75
126	0.48	1	0.75
127	1	1	0.67
128	0.67	0.67	0.67
129	0.50	0.05	0
130	1	0.50	1
131	1	1	1
132	0.75	0.60	1
133	0.14	0	0
134	1	1	1
135	1	1	1
136	0.20	0	0

download these results as csv

Task 2 Summary Results by Query

Simple Hit/Miss Counting [5]

MRR Method [6]

Runtime Results

Participant	Task	Runtime (min)	Machine
CSJ	1a	~30	BEER 2
CSJ	1b	7	BEER 2
CSJ	2	~35	BEER 2
HAFR	1a	15	BEER 2
HAFR	1b	5	BEER 2
HAFR	2	35	BEER 2
HL	1a	~95	FAST
HL	1b	~40	FAST
HL	2	~95	FAST

download these results as csv