2011:Query-by-Tapping Results

Introduction

These are the results for the 2011 running of the Query-by-tappingn task. For background information about this task set please refer to the 2011:Query_by_Tapping page.

Task Descriptions

Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:

• Jang's Dataset Roger Jang's MIR-QBT: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries.
• Hsiao's Dataset Show Hsiao's QBT_symbolic: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries.

Task 2 Goto Task 2 Results:The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.

General Legend

Team ID

Sub code	Submission name	Abstract	Contributors
CJ1	QBT	PDF	ChunTa Chen, Roger Jang

Task 1a, Jang's Dataset Results

Task 1a Overall Results

	CJ1
1	0.846
2	1.000
3	1.000
4	1.000
5	1.000
6	0.271
7	1.000
8	0.750
9	0.000
10	0.200
11	0.625
12	0.750
13	0.250
14	0.875
15	1.000
16	0.349
17	0.667
18	0.701
19	0.250
20	1.000
21	0.250
22	0.750
23	0.000
24	1.000
25	0.873
26	0.000
27	0.379
28	0.833
29	1.000
30	0.600
31	0.143
32	0.778
33	0.238
34	0.850
35	1.000
36	0.333
37	0.000
38	1.000
39	0.571
40	0.929
41	0.583
42	1.000
43	0.700
44	0.563
45	0.050
46	1.000
47	0.375
48	0.333
49	0.625
50	0.500
51	0.222
52	1.000
53	0.367
54	1.000
55	0.538
56	1.000
57	0.850
58	1.000
59	0.020
60	0.112
61	0.955
62	1.000
63	0.100
64	0.286
65	1.000
66	1.000
67	0.000
68	0.472
69	0.641
70	0.500
71	0.000
72	0.667
73	0.810
74	1.000
75	0.833
76	0.875
77	0.000
78	0.000
79	0.375
80	0.667
81	0.167
82	1.000
83	1.000
84	1.000
85	0.390
86	1.000
87	0.308
88	1.000
89	0.733
90	0.000
91	1.000
92	0.125
93	0.583
94	0.167
95	0.891
96	1.000
97	1.000
98	0.500
99	0.797
100	0.625
101	0.052
102	0.875
103	1.000
104	0.905
105	0.907
106	0.466
107	0.423
108	0.438
109	0.819
110	1.000
111	1.000
112	0.463
113	0.583
114	0.507
115	0.250
116	0.500
117	0.375
118	0.000
119	0.333
120	0.667
121	0.333
122	0.000
123	0.333
124	0.333
125	0.117
126	0.000
127	1.000
128	0.667
129	0.500
130	0.167
131	0.500
132	0.500
133	0.100
134	1.000
135	0.333
136	0.333

download these results as csv

Task 1a Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask1aSimpleByGroup.friedman.tukeyKramerHSD.csv not found

500px

MRR Method: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask1aMrrByGroup.friedman.tukeyKramerHSD.csv not found

500px

Task 1a Summary Results by Query Group

Simple Hit/Miss Counting file /nema-raid/www/mirex/results/2011/qbt/QbtTask1aSimpleByGroup.csv not found

MRR Method file /nema-raid/www/mirex/results/2011/qbt/QbtTask1aMrrByGroup.csv not found

Task 1a Summary Results by Query

Simple Hit/Miss Counting [1]

MRR Method [2]

Task 1b, Hsiao's Dataset Results

Task 1b Overall Results

	CJ1
1	0.50
2	NaN
3	0.50
4	0.42
5	1.00
6	0.50
7	1.00
8	1.00
9	1.00
10	0.33
11	1.00
12	0.12
13	0.75
14	0.00
15	NaN
16	1.00
17	1.00
18	0.33
19	0.84
20	0.33
21	0.58
22	1.00
23	0.71
24	0.33
25	0.00
26	0.00
27	0.27
28	NaN
29	1.00
30	1.00
31	NaN
32	1.00
33	0.25
34	0.21
35	NaN
36	0.00
37	0.00
38	0.03
39	0.50
40	0.00
41	0.33
42	0.50
43	0.48
44	0.00
45	0.04
46	0.14
47	0.33
48	1.00
49	0.20
50	1.00
51	0.00
52	0.67
53	0.69
54	0.08
55	0.00
56	0.25
57	0.42
58	NaN
59	1.00
60	0.32
61	0.27
62	0.00
63	0.67
64	NaN
65	0.42
66	NaN
67	0.00
68	NaN
69	0.04
70	0.00
71	NaN
72	1.00
73	0.50
74	NaN
75	1.00
76	0.05
77	NaN
78	0.67
79	0.03
80	0.00
81	0.00
82	0.00
83	1.00
84	0.56
85	0.00
86	1.00
87	1.00
88	0.40
89	0.21
90	0.17
91	0.75
92	1.00
93	0.10
94	0.00
95	0.75
96	1.00
97	0.00
98	0.67
99	0.65
100	0.62
101	0.90
102	1.00
103	NaN
104	1.00
105	0.71
106	0.00
107	NaN
108	1.00
109	0.25
110	0.00
111	0.20
112	0.17
113	1.00
114	0.25
115	NaN
116	0.00
117	1.00
118	0.83
119	0.00
120	0.17
121	0.19
122	0.38
123	0.33
124	0.90
125	0.00
126	1.00
127	0.50
128	0.43
129	1.00
130	0.00
131	0.00
132	1.00
133	0.00
134	0.00
135	0.71
136	0.25
137	1.00
138	0.10
139	0.23
140	0.00
141	0.21
142	0.44
143	1.00

download these results as csv

Task 1b Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask1bSimpleByGroup.friedman.tukeyKramerHSD.csv not found

500px

MRR Method: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask1bMrrByGroup.friedman.tukeyKramerHSD.csv not found

500px

Task 1b Summary Results by Query Group

Simple Hit/Miss Counting file /nema-raid/www/mirex/results/2011/qbt/QbtTask1bSimpleByGroup.csv not found

MRR Method file /nema-raid/www/mirex/results/2011/qbt/QbtTask1bMrrByGroup.csv not found

Task 1b Summary Results by Query

Simple Hit/Miss Counting [3]

MRR Method [4]

Task 2 Results

Task 2 Overall Results

	CJ1
1	0.846
2	1.000
3	1.000
4	1.000
5	1.000
6	0.271
7	1.000
8	0.750
9	0.250
10	0.200
11	0.625
12	0.750
13	0.250
14	0.875
15	1.000
16	0.334
17	0.689
18	0.641
19	0.250
20	1.000
21	0.250
22	0.750
23	0.000
24	1.000
25	0.871
26	0.000
27	0.394
28	0.833
29	1.000
30	0.600
31	0.143
32	0.778
33	0.202
34	0.767
35	1.000
36	0.400
37	0.000
38	1.000
39	0.571
40	0.869
41	0.583
42	1.000
43	0.700
44	0.563
45	0.050
46	1.000
47	0.500
48	0.333
49	0.625
50	0.500
51	0.222
52	1.000
53	0.367
54	1.000
55	0.509
56	1.000
57	0.867
58	1.000
59	0.000
60	0.124
61	0.955
62	0.917
63	0.100
64	0.286
65	1.000
66	1.000
67	0.000
68	0.435
69	0.611
70	0.500
71	0.000
72	0.667
73	0.810
74	1.000
75	0.833
76	0.875
77	0.000
78	0.000
79	0.375
80	0.667
81	0.167
82	1.000
83	1.000
84	1.000
85	0.376
86	1.000
87	0.308
88	1.000
89	0.733
90	0.000
91	1.000
92	0.125
93	0.583
94	0.250
95	0.893
96	1.000
97	1.000
98	0.500
99	0.750
100	0.625
101	0.064
102	0.875
103	1.000
104	0.866
105	0.873
106	0.443
107	0.437
108	0.438
109	0.735
110	0.867
111	1.000
112	0.588
113	0.493
114	0.373
115	0.400
116	0.500
117	0.375
118	0.000
119	0.333
120	0.667
121	0.667
122	0.000
123	0.333
124	0.333
125	0.120
126	0.000
127	1.000
128	0.667
129	0.500
130	0.500
131	1.000
132	0.500
133	0.100
134	1.000
135	1.000
136	0.000

download these results as csv

Task 2 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask2SimpleByGroup.friedman.tukeyKramerHSD.csv not found

500px

MRR Method: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask2MrrByGroup.friedman.tukeyKramerHSD.csv not found

500px

Task 2 Summary Results by Query Group

Simple Hit/Miss Counting file /nema-raid/www/mirex/results/2011/qbt/QbtTask2SimpleByGroup.csv not found

MRR Method file /nema-raid/www/mirex/results/2011/qbt/QbtTask2MrrByGroup.csv not found

Task 2 Summary Results by Query

Simple Hit/Miss Counting [5]

MRR Method [6]

Runtime Results

file /nema-raid/www/mirex/results/2011/qbt/qbtRunTime.csv not found