Difference between revisions of "2011:Query-by-Tapping Results"

From MIREX Wiki
(→‎Task 1a Overall Results)
(→‎Task 1b Summary Results by Query Group)
 
(6 intermediate revisions by the same user not shown)
Line 32: Line 32:
 
=====Task 1a Overall Results=====
 
=====Task 1a Overall Results=====
 
<csv>2011/qbt/2011qbtTask1a.csv</csv>
 
<csv>2011/qbt/2011qbtTask1a.csv</csv>
−
 
−
====Task 1a Friedman's Test for Significant Differences====
 
−
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
−
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
−
 
−
Simple Hit/Miss Count:
 
−
<csv>2011/qbt/friedman/QbtTask1aSimpleByGroup.friedman.tukeyKramerHSD.csv</csv>
 
−
 
−
[[Image:2011QbtTask1aSimpleByGroup.friedman.tukeyKramerHSD.png|500px]]
 
−
 
−
MRR Method:
 
−
<csv>2011/qbt/friedman/QbtTask1aMrrByGroup.friedman.tukeyKramerHSD.csv</csv>
 
−
 
−
[[Image:2011QbtTask1aMrrByGroup.friedman.tukeyKramerHSD.png|500px]]
 
  
 
====Task 1a Summary Results by Query Group====
 
====Task 1a Summary Results by Query Group====
 
Simple Hit/Miss Counting
 
Simple Hit/Miss Counting
−
<csv p=2>2011/qbt/QbtTask1aSimpleByGroup.csv</csv>
+
<csv p=2>2011/qbt/qbtTask1aSimple.csv</csv>
  
 
MRR Method
 
MRR Method
−
<csv p=2>2011/qbt/QbtTask1aMrrByGroup.csv</csv>
+
<csv p=2>2011/qbt/qbtTask1aMRR.csv</csv>
  
 
====Task 1a Summary Results by Query ====
 
====Task 1a Summary Results by Query ====
 
Simple Hit/Miss Counting
 
Simple Hit/Miss Counting
−
[https://www.music-ir.org/mirex/results/2011/qbt/QbtTask1aSimpleByQuery.csv]
+
[https://www.music-ir.org/mirex/results/2011/qbt/qbtTask1aQuerySimple.csv]
  
 
MRR Method
 
MRR Method
−
[https://www.music-ir.org/mirex/results/2011/qbt/QbtTask1aMrrByQuery.csv]
+
[https://www.music-ir.org/mirex/results/2011/qbt/qbtTask1aQueryMRR.csv]
−
 
 
  
 
===Task 1b, Hsiao's Dataset Results===
 
===Task 1b, Hsiao's Dataset Results===
  
 
=====Task 1b Overall Results=====
 
=====Task 1b Overall Results=====
−
<csv p=2>2011/qbt/qbtTask1bMRR.csv</csv>
+
<csv p=2>2011/qbt/2011qbtTask1b.csv</csv>
  
−
====Task 1b Friedman's Test for Significant Differences====
 
−
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
−
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
−
 
−
Simple Hit/Miss Count:
 
−
<csv>2011/qbt/friedman/QbtTask1bSimpleByGroup.friedman.tukeyKramerHSD.csv</csv>
 
−
 
−
[[Image:2011QbtTask1bSimpleByGroup.friedman.tukeyKramerHSD.png|500px]]
 
−
 
−
MRR Method:
 
−
<csv>2011/qbt/friedman/QbtTask1bMrrByGroup.friedman.tukeyKramerHSD.csv</csv>
 
−
 
−
[[Image:2011QbtTask1bMrrByGroup.friedman.tukeyKramerHSD.png|500px ]]
 
  
 
====Task 1b Summary Results by Query Group====
 
====Task 1b Summary Results by Query Group====
 
Simple Hit/Miss Counting
 
Simple Hit/Miss Counting
−
<csv p=2>2011/qbt/QbtTask1bSimpleByGroup.csv</csv>
+
<csv p=2>2011/qbt/qbtTask1bSimple.csv</csv>
  
 
MRR Method
 
MRR Method
−
<csv p=2>2011/qbt/QbtTask1bMrrByGroup.csv</csv>
+
<csv p=2>2011/qbt/qbtTask1bMRR.csv</csv>
  
 
====Task 1b Summary Results by Query ====
 
====Task 1b Summary Results by Query ====
 
Simple Hit/Miss Counting
 
Simple Hit/Miss Counting
−
[https://www.music-ir.org/mirex/results/2011/qbt/QbtTask1bSimpleByQuery.csv]
+
[https://www.music-ir.org/mirex/results/2011/qbt/qbtTask1bQuerySimple.csv]
  
 
MRR Method
 
MRR Method
−
[https://www.music-ir.org/mirex/results/2011/qbt/QbtTask1bMrrByQuery.csv]
+
[https://www.music-ir.org/mirex/results/2011/qbt/qbtTask1bQueryMRR.csv]
  
 
===Task 2 Results===
 
===Task 2 Results===
Line 99: Line 71:
  
 
=====Task 2 Overall Results=====
 
=====Task 2 Overall Results=====
−
<csv>2011/qbt/qbtTask2MRR.csv</csv>
+
<csv>2011/qbt/2011qbtTask2.csv</csv>
−
 
 
−
====Task 2 Friedman's Test for Significant Differences====
 
−
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
−
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
−
 
 
−
Simple Hit/Miss Count:
 
−
<csv>2011/qbt/friedman/QbtTask2SimpleByGroup.friedman.tukeyKramerHSD.csv</csv>
 
−
 
 
−
[[Image:2011QbtTask2SimpleByGroup.friedman.tukeyKramerHSD.png|500px]]
 
  
−
MRR Method:
 
−
<csv>2011/qbt/friedman/QbtTask2MrrByGroup.friedman.tukeyKramerHSD.csv</csv>
 
  
−
[[Image:2011QbtTask2MrrByGroup.friedman.tukeyKramerHSD.png|500px]]
 
  
 
====Task 2 Summary Results by Query Group====
 
====Task 2 Summary Results by Query Group====
 
Simple Hit/Miss Counting
 
Simple Hit/Miss Counting
−
<csv p=2>2011/qbt/QbtTask2SimpleByGroup.csv</csv>
+
<csv p=2>2011/qbt/qbtTask2Simple.csv</csv>
  
 
MRR Method
 
MRR Method
−
<csv p=2>2011/qbt/QbtTask2MrrByGroup.csv</csv>
+
<csv p=2>2011/qbt/qbtTask2MRR.csv</csv>
  
 
====Task 2 Summary Results by Query ====
 
====Task 2 Summary Results by Query ====
 
Simple Hit/Miss Counting
 
Simple Hit/Miss Counting
−
[https://www.music-ir.org/mirex/results/2011/qbt/QbtTask2SimpleByQuery.csv]
+
[https://www.music-ir.org/mirex/results/2011/qbt/qbtTask2QuerySimple.csv]
  
 
MRR Method
 
MRR Method
−
[https://www.music-ir.org/mirex/results/2011/qbt/QbtTask2MrrByQuery.csv]
+
[https://www.music-ir.org/mirex/results/2011/qbt/qbtTask2QuerySimple.csv]
  
 
===Runtime Results===
 
===Runtime Results===

Latest revision as of 12:32, 31 October 2011

Introduction

These are the results for the 2011 running of the Query-by-tappingn task. For background information about this task set please refer to the 2011:Query_by_Tapping page.

Task Descriptions

Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:

  • Jang's Dataset Roger Jang's MIR-QBT: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries.
  • Hsiao's Dataset Show Hsiao's QBT_symbolic: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries.


Task 2 Goto Task 2 Results:The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.

General Legend

Team ID

Sub code Submission name Abstract Contributors
CJ1 QBT PDF ChunTa Chen, Roger Jang

Task 1 Results

Task 1a, Jang's Dataset Results

Task 1a Overall Results
CJ1
Simple Count 0.747
MRR 0.599
Total Count 890

download these results as csv

Task 1a Summary Results by Query Group

Simple Hit/Miss Counting

CJ1
1 0.85
2 1.00
3 1.00
4 1.00
5 1.00
6 0.75
7 1.00
8 1.00
9 0.00
10 1.00
11 1.00
12 1.00
13 1.00
14 0.88
15 1.00
16 0.80
17 0.73
18 0.77
19 0.25
20 1.00
21 0.50
22 1.00
23 0.00
24 1.00
25 1.00
26 0.00
27 0.79
28 1.00
29 1.00
30 0.67
31 1.00
32 1.00
33 0.47
34 0.92
35 1.00
36 0.50
37 0.00
38 1.00
39 1.00
40 0.93
41 0.75
42 1.00
43 0.70
44 0.75
45 0.50
46 1.00
47 1.00
48 0.33
49 1.00
50 0.67
51 0.40
52 1.00
53 0.60
54 1.00
55 0.93
56 1.00
57 1.00
58 1.00
59 0.20
60 0.43
61 1.00
62 1.00
63 1.00
64 0.50
65 1.00
66 1.00
67 0.00
68 0.50
69 0.79
70 0.50
71 0.00
72 0.67
73 0.86
74 1.00
75 0.83
76 1.00
77 0.00
78 0.00
79 0.50
80 0.67
81 1.00
82 1.00
83 1.00
84 1.00
85 0.71
86 1.00
87 0.50
88 1.00
89 1.00
90 0.00
91 1.00
92 0.25
93 1.00
94 0.33
95 1.00
96 1.00
97 1.00
98 0.50
99 0.88
100 0.75
101 0.33
102 0.88
103 1.00
104 1.00
105 0.93
106 0.50
107 0.61
108 0.44
109 0.94
110 1.00
111 1.00
112 0.62
113 1.00
114 1.00
115 0.40
116 0.50
117 0.50
118 0.00
119 0.33
120 0.67
121 0.33
122 0.00
123 0.33
124 0.33
125 0.67
126 0.00
127 1.00
128 0.67
129 0.50
130 0.50
131 0.50
132 0.50
133 1.00
134 1.00
135 1.00
136 1.00

download these results as csv

MRR Method

CJ1
1 0.85
2 1.00
3 1.00
4 1.00
5 1.00
6 0.27
7 1.00
8 0.75
9 0.00
10 0.20
11 0.62
12 0.75
13 0.25
14 0.88
15 1.00
16 0.35
17 0.67
18 0.70
19 0.25
20 1.00
21 0.25
22 0.75
23 0.00
24 1.00
25 0.87
26 0.00
27 0.38
28 0.83
29 1.00
30 0.60
31 0.14
32 0.78
33 0.24
34 0.85
35 1.00
36 0.33
37 0.00
38 1.00
39 0.57
40 0.93
41 0.58
42 1.00
43 0.70
44 0.56
45 0.05
46 1.00
47 0.38
48 0.33
49 0.62
50 0.50
51 0.22
52 1.00
53 0.37
54 1.00
55 0.54
56 1.00
57 0.85
58 1.00
59 0.02
60 0.11
61 0.95
62 1.00
63 0.10
64 0.29
65 1.00
66 1.00
67 0.00
68 0.47
69 0.64
70 0.50
71 0.00
72 0.67
73 0.81
74 1.00
75 0.83
76 0.88
77 0.00
78 0.00
79 0.38
80 0.67
81 0.17
82 1.00
83 1.00
84 1.00
85 0.39
86 1.00
87 0.31
88 1.00
89 0.73
90 0.00
91 1.00
92 0.12
93 0.58
94 0.17
95 0.89
96 1.00
97 1.00
98 0.50
99 0.80
100 0.62
101 0.05
102 0.88
103 1.00
104 0.91
105 0.91
106 0.47
107 0.42
108 0.44
109 0.82
110 1.00
111 1.00
112 0.46
113 0.58
114 0.51
115 0.25
116 0.50
117 0.38
118 0.00
119 0.33
120 0.67
121 0.33
122 0.00
123 0.33
124 0.33
125 0.12
126 0.00
127 1.00
128 0.67
129 0.50
130 0.17
131 0.50
132 0.50
133 0.10
134 1.00
135 0.33
136 0.33

download these results as csv

Task 1a Summary Results by Query

Simple Hit/Miss Counting [1]

MRR Method [2]

Task 1b, Hsiao's Dataset Results

Task 1b Overall Results
CJ1
Simple Count 0.68
MRR 0.51
Total Count 410.00

download these results as csv


Task 1b Summary Results by Query Group

Simple Hit/Miss Counting

CJ1
1 0.50
2 NaN
3 0.75
4 1.00
5 1.00
6 1.00
7 1.00
8 1.00
9 1.00
10 0.33
11 1.00
12 0.50
13 1.00
14 0.00
15 NaN
16 1.00
17 1.00
18 1.00
19 1.00
20 0.33
21 0.67
22 1.00
23 0.83
24 0.33
25 0.00
26 0.00
27 1.00
28 NaN
29 1.00
30 1.00
31 NaN
32 1.00
33 0.50
34 1.00
35 NaN
36 0.00
37 0.00
38 0.29
39 1.00
40 0.00
41 1.00
42 0.67
43 1.00
44 0.00
45 0.25
46 0.67
47 1.00
48 1.00
49 0.67
50 1.00
51 0.00
52 1.00
53 0.89
54 0.33
55 0.00
56 0.50
57 1.00
58 NaN
59 1.00
60 1.00
61 0.75
62 0.00
63 1.00
64 NaN
65 0.75
66 NaN
67 0.00
68 NaN
69 0.40
70 0.00
71 NaN
72 1.00
73 0.50
74 NaN
75 1.00
76 0.20
77 NaN
78 1.00
79 0.25
80 0.00
81 0.00
82 0.00
83 1.00
84 1.00
85 0.00
86 1.00
87 1.00
88 0.40
89 0.80
90 0.33
91 1.00
92 1.00
93 0.50
94 0.00
95 1.00
96 1.00
97 0.00
98 1.00
99 1.00
100 1.00
101 1.00
102 1.00
103 NaN
104 1.00
105 1.00
106 0.00
107 NaN
108 1.00
109 1.00
110 0.00
111 1.00
112 0.33
113 1.00
114 0.50
115 NaN
116 0.00
117 1.00
118 1.00
119 0.00
120 0.33
121 0.67
122 0.75
123 0.50
124 1.00
125 0.00
126 1.00
127 1.00
128 1.00
129 1.00
130 0.00
131 0.00
132 1.00
133 0.00
134 0.00
135 1.00
136 0.50
137 1.00
138 0.50
139 0.40
140 0.00
141 0.50
142 1.00
143 1.00

download these results as csv

MRR Method

CJ1
1 0.50
2 NaN
3 0.50
4 0.42
5 1.00
6 0.50
7 1.00
8 1.00
9 1.00
10 0.33
11 1.00
12 0.12
13 0.75
14 0.00
15 NaN
16 1.00
17 1.00
18 0.33
19 0.84
20 0.33
21 0.58
22 1.00
23 0.71
24 0.33
25 0.00
26 0.00
27 0.27
28 NaN
29 1.00
30 1.00
31 NaN
32 1.00
33 0.25
34 0.21
35 NaN
36 0.00
37 0.00
38 0.03
39 0.50
40 0.00
41 0.33
42 0.50
43 0.48
44 0.00
45 0.04
46 0.14
47 0.33
48 1.00
49 0.20
50 1.00
51 0.00
52 0.67
53 0.69
54 0.08
55 0.00
56 0.25
57 0.42
58 NaN
59 1.00
60 0.32
61 0.27
62 0.00
63 0.67
64 NaN
65 0.42
66 NaN
67 0.00
68 NaN
69 0.04
70 0.00
71 NaN
72 1.00
73 0.50
74 NaN
75 1.00
76 0.05
77 NaN
78 0.67
79 0.03
80 0.00
81 0.00
82 0.00
83 1.00
84 0.56
85 0.00
86 1.00
87 1.00
88 0.40
89 0.21
90 0.17
91 0.75
92 1.00
93 0.10
94 0.00
95 0.75
96 1.00
97 0.00
98 0.67
99 0.65
100 0.62
101 0.90
102 1.00
103 NaN
104 1.00
105 0.71
106 0.00
107 NaN
108 1.00
109 0.25
110 0.00
111 0.20
112 0.17
113 1.00
114 0.25
115 NaN
116 0.00
117 1.00
118 0.83
119 0.00
120 0.17
121 0.19
122 0.38
123 0.33
124 0.90
125 0.00
126 1.00
127 0.50
128 0.43
129 1.00
130 0.00
131 0.00
132 1.00
133 0.00
134 0.00
135 0.71
136 0.25
137 1.00
138 0.10
139 0.23
140 0.00
141 0.21
142 0.44
143 1.00

download these results as csv

Task 1b Summary Results by Query

Simple Hit/Miss Counting [3]

MRR Method [4]

Task 2 Results

Task 2 Overall Results
CJ1
Simple Count 0.727
MRR 0.588
Total Count 890

download these results as csv


Task 2 Summary Results by Query Group

Simple Hit/Miss Counting

CJ1
1 0.85
2 1.00
3 1.00
4 1.00
5 1.00
6 0.75
7 1.00
8 1.00
9 1.00
10 1.00
11 1.00
12 1.00
13 1.00
14 0.88
15 1.00
16 0.75
17 0.73
18 0.69
19 0.25
20 1.00
21 0.50
22 1.00
23 0.00
24 1.00
25 1.00
26 0.00
27 0.71
28 1.00
29 1.00
30 0.67
31 1.00
32 1.00
33 0.37
34 0.83
35 1.00
36 0.75
37 0.00
38 1.00
39 1.00
40 0.93
41 0.75
42 1.00
43 0.70
44 0.75
45 0.50
46 1.00
47 1.00
48 0.33
49 1.00
50 0.67
51 0.40
52 1.00
53 0.60
54 1.00
55 0.93
56 1.00
57 1.00
58 1.00
59 0.00
60 0.43
61 1.00
62 0.92
63 1.00
64 0.50
65 1.00
66 1.00
67 0.00
68 0.50
69 0.76
70 0.50
71 0.00
72 0.67
73 0.86
74 1.00
75 0.83
76 1.00
77 0.00
78 0.00
79 0.50
80 0.67
81 1.00
82 1.00
83 1.00
84 1.00
85 0.71
86 1.00
87 0.50
88 1.00
89 1.00
90 0.00
91 1.00
92 0.25
93 1.00
94 0.33
95 1.00
96 1.00
97 1.00
98 0.50
99 0.81
100 0.75
101 0.50
102 0.88
103 1.00
104 0.89
105 0.93
106 0.50
107 0.67
108 0.44
109 0.88
110 0.87
111 1.00
112 0.75
113 0.86
114 0.80
115 0.40
116 0.50
117 0.50
118 0.00
119 0.33
120 0.67
121 0.67
122 0.00
123 0.33
124 0.33
125 0.67
126 0.00
127 1.00
128 0.67
129 0.50
130 0.50
131 1.00
132 0.50
133 1.00
134 1.00
135 1.00
136 0.00

download these results as csv

MRR Method

CJ1
1 0.85
2 1.00
3 1.00
4 1.00
5 1.00
6 0.27
7 1.00
8 0.75
9 0.25
10 0.20
11 0.62
12 0.75
13 0.25
14 0.88
15 1.00
16 0.33
17 0.69
18 0.64
19 0.25
20 1.00
21 0.25
22 0.75
23 0.00
24 1.00
25 0.87
26 0.00
27 0.39
28 0.83
29 1.00
30 0.60
31 0.14
32 0.78
33 0.20
34 0.77
35 1.00
36 0.40
37 0.00
38 1.00
39 0.57
40 0.87
41 0.58
42 1.00
43 0.70
44 0.56
45 0.05
46 1.00
47 0.50
48 0.33
49 0.62
50 0.50
51 0.22
52 1.00
53 0.37
54 1.00
55 0.51
56 1.00
57 0.87
58 1.00
59 0.00
60 0.12
61 0.95
62 0.92
63 0.10
64 0.29
65 1.00
66 1.00
67 0.00
68 0.43
69 0.61
70 0.50
71 0.00
72 0.67
73 0.81
74 1.00
75 0.83
76 0.88
77 0.00
78 0.00
79 0.38
80 0.67
81 0.17
82 1.00
83 1.00
84 1.00
85 0.38
86 1.00
87 0.31
88 1.00
89 0.73
90 0.00
91 1.00
92 0.12
93 0.58
94 0.25
95 0.89
96 1.00
97 1.00
98 0.50
99 0.75
100 0.62
101 0.06
102 0.88
103 1.00
104 0.87
105 0.87
106 0.44
107 0.44
108 0.44
109 0.73
110 0.87
111 1.00
112 0.59
113 0.49
114 0.37
115 0.40
116 0.50
117 0.38
118 0.00
119 0.33
120 0.67
121 0.67
122 0.00
123 0.33
124 0.33
125 0.12
126 0.00
127 1.00
128 0.67
129 0.50
130 0.50
131 1.00
132 0.50
133 0.10
134 1.00
135 1.00
136 0.00

download these results as csv

Task 2 Summary Results by Query

Simple Hit/Miss Counting [5]

MRR Method [6]

Runtime Results

file /nema-raid/www/mirex/results/2011/qbt/qbtRunTime.csv not found

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2011:Query-by-Tapping_Results&oldid=8555"