Difference between revisions of "2009:Query-by-Tapping Results"

From MIREX Wiki
(→‎Introduction)
 
(22 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
==Introduction==
 
==Introduction==
−
These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the [[Query by Singing/Humming]] page.  
+
These are the results for the 2008 running of the Query-by-tappingn task. For background information about this task set please refer to the [[2009:Query_by_Tapping]] page.  
  
 
===Task Descriptions===
 
===Task Descriptions===
Line 14: Line 14:
 
===General Legend===
 
===General Legend===
 
====Team ID====
 
====Team ID====
−
'''CSJ''' = [[Chun-Ta Chen and Jyh-Shing Roger Jang]]
+
'''CSJ''' = [https://www.music-ir.org/mirex/results/2009/qbt/QbtChenJang.pdf  Chun-Ta Chen and Jyh-Shing Roger Jang]<br/>
−
 
+
'''HAFR''' = [https://www.music-ir.org/mirex/results/2009/qbt/HAFR.pdf  Pierre Hanna, Julien Allali, Pascal Ferraro,Matthias Robine]<br/>
−
'''HAFR''' = [[Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine]]
+
'''HL''' = [https://www.music-ir.org/mirex/results/2009/qbt/QBT_Show.pdf  Shu-Jen Show Hsiao,Tyne Liang]<br/>
−
 
 
−
'''HL''' = [[Shu-Jen Show and Hsiao Tyne Liang]]
 
−
 
 
  
 
===Task 1 Results===
 
===Task 1 Results===
Line 27: Line 24:
  
 
=====Task 1a Overall Results=====
 
=====Task 1a Overall Results=====
−
<csv>qbt/QbtFinalTask1a.csv</csv>
+
<csv>2009/qbt/QbtFinalTask1a.csv</csv>
  
 
====Task 1a Friedman's Test for Significant Differences====
 
====Task 1a Friedman's Test for Significant Differences====
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
−
<csv>qbt/friedman/QbtTask1aSimple_friedman_tukeyKramerHSD.csv</csv>
+
 
−
[[Image:QbtTask1aSimple_Friedman_Mean_Ranks.png|800x600px]]
+
Simple Hit/Miss Count:
 +
<csv>2009/qbt/friedman/QbtTask1aSimple_friedman_tukeyKramerHSD.csv</csv>
 +
 
 +
[[Image:2009_sqbttask1asimple_friedman_mean_ranks.png]]
 +
 
 +
MRR Method:
 +
<csv>2009/qbt/friedman/Qbt1aTask2Mrr_friedman_tukeyKramerHSD.csv</csv>
 +
 
 +
[[Image:2009_sqbt1atask2mrr_friedman_mean_ranks.png]]
  
 
====Task 1a Summary Results by Query Group====
 
====Task 1a Summary Results by Query Group====
 
Simple Hit/Miss Counting
 
Simple Hit/Miss Counting
−
<csv>qbt/QbtTask1aSimpleByGroup.csv</csv>
+
<csv p=2>2009/qbt/QbtTask1aSimpleByGroup.csv</csv>
  
 
MRR Method
 
MRR Method
−
<csv>qbt/QbtTask1aMrrByGroup.csv</csv>
+
<csv p=2>2009/qbt/QbtTask1aMrrByGroup.csv</csv>
−
 
 
  
 
====Task 1a Summary Results by Query ====
 
====Task 1a Summary Results by Query ====
 
Simple Hit/Miss Counting
 
Simple Hit/Miss Counting
−
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask1aSimpleByQuery.csv]
+
[https://www.music-ir.org/mirex/results/2009/qbt/QbtTask1aSimpleByQuery.csv]
  
 
MRR Method
 
MRR Method
−
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask1aMrrByQuery.csv]
+
[https://www.music-ir.org/mirex/results/2009/qbt/QbtTask1aMrrByQuery.csv]
  
  
Line 54: Line 58:
  
 
=====Task 1b Overall Results=====
 
=====Task 1b Overall Results=====
−
<csv>qbt/QbtFinalTask1b.csv</csv>
+
<csv p=2>2009/qbt/QbtFinalTask1b.csv</csv>
  
 
====Task 1b Friedman's Test for Significant Differences====
 
====Task 1b Friedman's Test for Significant Differences====
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
−
<csv>qbt/friedman/QbtTask1aSimple_friedman_tukeyKramerHSD.csv</csv>
 
  
−
[[Image:qbt/friedman/QbtTask1a2Simple_Friedman_Mean_Ranks.png]]
+
Simple Hit/Miss Count:
 +
<csv>2009/qbt/friedman/QbtTask1bSimple_friedman_tukeyKramerHSD.csv</csv>
 +
 
 +
[[Image:2009_sqbttask1bsimple_friedman_mean_ranks.png]]
 +
 
 +
MRR Method:
 +
<csv>2009/qbt/friedman/QbtTask1bMrr_friedman_tukeyKramerHSD.csv</csv>
 +
 
 +
[[Image:2009_sqbttask1bmrr_friedman_mean_ranks.png ]]
  
 
====Task 1b Summary Results by Query Group====
 
====Task 1b Summary Results by Query Group====
 
Simple Hit/Miss Counting
 
Simple Hit/Miss Counting
−
<csv>qbt/QbtTask1bSimpleByGroup.csv</csv>
+
<csv p=2>2009/qbt/QbtTask1bSimpleByGroup.csv</csv>
  
 
MRR Method
 
MRR Method
−
<csv>qbt/QbtTask1bMrrByGroup.csv</csv>
+
<csv p=2>2009/qbt/QbtTask1bMrrByGroup.csv</csv>
−
 
 
  
 
====Task 1b Summary Results by Query ====
 
====Task 1b Summary Results by Query ====
 
Simple Hit/Miss Counting
 
Simple Hit/Miss Counting
−
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask1bSimpleByQuery.csv]
+
[https://www.music-ir.org/mirex/results/2009/qbt/QbtTask1bSimpleByQuery.csv]
  
 
MRR Method
 
MRR Method
−
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask1bMrrByQuery.csv]
+
[https://www.music-ir.org/mirex/results/2009/qbt/QbtTask1bMrrByQuery.csv]
  
 
===Task 2 Results===
 
===Task 2 Results===
Line 82: Line 92:
  
 
=====Task 2 Overall Results=====
 
=====Task 2 Overall Results=====
−
<csv>qbt/QbtFinalTask2.csv</csv>
+
<csv>2009/qbt/QbtFinalTask2.csv</csv>
  
 
====Task 2 Friedman's Test for Significant Differences====
 
====Task 2 Friedman's Test for Significant Differences====
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
−
<csv>qbt/friedman/QbtTask1aSimple_friedman_tukeyKramerHSD.csv</csv>
 
  
−
[[Image:qbt/friedman/QbtTask1a2Simple_Friedman_Mean_Ranks.png]]
+
Simple Hit/Miss Count:
 +
<csv>2009/qbt/friedman/QbtTask2Simple_friedman_tukeyKramerHSD.csv</csv>
 +
 
 +
[[Image:2009_sqbttask2simple_friedman_mean_ranks.png]]
 +
 
 +
MRR Method:
 +
<csv>2009/qbt/friedman/QbtTask2Mrr_friedman_tukeyKramerHSD.csv</csv>
 +
 
 +
[[Image:2009_sqbttask2mrr_friedman_mean_ranks.png]]
  
 
====Task 2 Summary Results by Query Group====
 
====Task 2 Summary Results by Query Group====
 
Simple Hit/Miss Counting
 
Simple Hit/Miss Counting
−
<csv>qbt/QbtTask2SimpleByGroup.csv</csv>
+
<csv p=2>2009/qbt/QbtTask2SimpleByGroup.csv</csv>
  
 
MRR Method
 
MRR Method
−
<csv>qbt/QbtTask2MrrByGroup.csv</csv>
+
<csv p=2>2009/qbt/QbtTask2MrrByGroup.csv</csv>
−
 
 
  
 
====Task 2 Summary Results by Query ====
 
====Task 2 Summary Results by Query ====
 
Simple Hit/Miss Counting
 
Simple Hit/Miss Counting
−
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask2SimpleByQuery.csv]
+
[https://www.music-ir.org/mirex/results/2009/qbt/QbtTask2SimpleByQuery.csv]
  
 
MRR Method
 
MRR Method
−
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask2MrrByQuery.csv]
+
[https://www.music-ir.org/mirex/results/2009/qbt/QbtTask2MrrByQuery.csv]
  
 
===Runtime Results===
 
===Runtime Results===
  
−
<csv>qbsh.runtime.csv</csv>
+
<csv>2009/qbt/qbtRunTime.csv</csv>
  
 
[[Category: Results]]
 
[[Category: Results]]

Latest revision as of 11:59, 26 July 2010

Introduction

These are the results for the 2008 running of the Query-by-tappingn task. For background information about this task set please refer to the 2009:Query_by_Tapping page.

Task Descriptions

Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:

  • Jang's Dataset Roger Jang's MIR-QBT: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries.
  • Hsiao's Dataset Show Hsiao's QBT_symbolic: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries.


Task 2 Goto Task 2 Results:The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.

General Legend

Team ID

CSJ = Chun-Ta Chen and Jyh-Shing Roger Jang
HAFR = Pierre Hanna, Julien Allali, Pascal Ferraro,Matthias Robine
HL = Shu-Jen Show Hsiao,Tyne Liang

Task 1 Results

Task 1a, Jang's Dataset Results

Task 1a Overall Results
CSJ HAFR HL
Simple Count 0.87 0.88 0.89
MRR 0.78 0.77 0.79
Total count 890 890 890

download these results as csv

Task 1a Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:

TeamID TeamID Lowerbound Mean Upperbound Significance
HL CSJ -0.1863 -0.0184 0.1496 FALSE
HL HAFR -0.1716 -0.0037 0.1643 FALSE
CSJ HAFR -0.1533 0.0147 0.1827 FALSE

download these results as csv

2009 sqbttask1asimple friedman mean ranks.png

MRR Method:

TeamID TeamID Lowerbound Mean Upperbound Significance
HL CSJ -0.2337 -0.0074 0.2190 FALSE
HL HAFR -0.0646 0.1618 0.3881 FALSE
CSJ HAFR -0.0572 0.1691 0.3955 FALSE

download these results as csv

2009 sqbt1atask2mrr friedman mean ranks.png

Task 1a Summary Results by Query Group

Simple Hit/Miss Counting

CSJ HAFR HL
1 1 0.92 0.92
2 1 1 1
3 1 1 1
4 1 0 1
5 1 1 1
6 1 1 0.50
7 1 1 1
8 1 1 1
9 1 1 1
10 1 1 1
11 0.50 1 0.50
12 1 1 1
13 1 1 1
14 1 1 1
15 1 1 1
16 1 1 0.95
17 0.87 0.93 1
18 0.77 0.85 0.69
19 0.38 0.50 1
20 1 1 1
21 1 1 1
22 1 0.50 0.50
23 0 0 0
24 1 1 1
25 1 1 1
26 0 0 1
27 1 0.94 1
28 1 1 1
29 1 1 1
30 0.58 0.75 0.67
31 1 1 1
32 1 1 0.67
33 0.78 0.75 0.94
34 0.92 0.92 0.92
35 1 1 1
36 1 1 1
37 0 1 1
38 1 1 1
39 1 1 1
40 1 1 1
41 0.83 0.75 0.92
42 1 1 1
43 0.80 1 1
44 0.75 0.75 0.50
45 1 1 0.50
46 1 1 1
47 1 1 1
48 1 1 1
49 1 1 1
50 1 1 1
51 0.60 0.80 0.60
52 1 1 1
53 0.80 0.60 0.60
54 1 1 1
55 1 0.98 1
56 1 1 1
57 1 0.80 1
58 1 1 1
59 0.20 0.60 0.20
60 0.43 1 0.71
61 1 1 1
62 1 1 1
63 1 1 1
64 0.50 0.50 0.50
65 1 1 1
66 1 1 0.50
67 1 1 1
68 0.67 0.78 0.61
69 0.82 0.82 0.94
70 1 1 0.75
71 1 0 1
72 0.83 0.92 0.75
73 0.93 0.86 0.93
74 1 1 1
75 0.83 1 0.83
76 1 1 1
77 0 0 1
78 1 0 0
79 1 0.50 0.75
80 1 1 1
81 1 1 1
82 1 1 1
83 1 1 1
84 1 1 1
85 0.86 0.86 0.86
86 1 1 1
87 0.58 0.75 0.75
88 1 1 1
89 1 0.67 1
90 0.50 1 1
91 1 1 1
92 0.25 0.25 0.50
93 1 1 1
94 0.50 0.67 0.67
95 1 1 1
96 1 1 1
97 1 1 1
98 0.50 0.50 0.50
99 0.88 0.88 0.81
100 1 1 1
101 0.83 1 0.83
102 0.88 1 1
103 1 1 1
104 1 1 1
105 1 1 0.96
106 0.73 0.68 0.50
107 0.94 0.94 1
108 0.69 0.56 0.88
109 0.94 0.81 0.94
110 1 1 1
111 1 1 0.78
112 1 1 1
113 1 1 1
114 1 1 1
115 0.80 0.60 0.60
116 0.75 0.75 0.75
117 1 1 1
118 1 1 1
119 0.67 0.67 0.67
120 0.67 0.67 1
121 0.33 0.67 0.33
122 0.33 0.67 0.67
123 0.67 0.33 0.33
124 0.33 0.33 0.33
125 1 0.33 0.67
126 0.67 0.67 1
127 1 1 1
128 0.67 0.67 0.67
129 0.50 0.50 0
130 1 0.50 1
131 1 1 1
132 1 1 1
133 1 1 1
134 1 1 1
135 1 1 1
136 1 1 1

download these results as csv

MRR Method

CSJ HAFR HL
1 0.94 0.88 0.88
2 1 1 1
3 1 1 1
4 1 0 1
5 1 1 1
6 0.78 0.88 0.50
7 1 1 1
8 0.75 0.75 0.75
9 1 0.25 1
10 1 1 1
11 0.50 0.60 0.50
12 1 1 1
13 1 1 0.50
14 1 0.94 0.94
15 1 1 1
16 0.93 0.82 0.35
17 0.76 0.93 1
18 0.71 0.64 0.59
19 0.28 0.30 0.50
20 1 1 1
21 0.56 1 1
22 1 0.50 0.50
23 0 0 0
24 1 0.50 1
25 0.90 0.90 0.89
26 0 0 0.14
27 0.97 0.76 0.88
28 0.83 1 1
29 1 1 1
30 0.52 0.75 0.53
31 1 0.33 1
32 1 1 0.67
33 0.50 0.72 0.91
34 0.92 0.92 0.92
35 1 1 1
36 0.88 0.78 0.88
37 0 1 0.25
38 1 1 1
39 1 1 1
40 0.96 0.96 1
41 0.63 0.58 0.60
42 1 1 1
43 0.80 0.88 0.84
44 0.75 0.63 0.50
45 0.60 0.42 0.17
46 1 0.71 0.92
47 1 0.57 1
48 0.41 1 0.78
49 1 1 0.75
50 0.73 0.44 0.78
51 0.33 0.63 0.27
52 1 1 1
53 0.67 0.60 0.60
54 1 0.50 1
55 0.98 0.96 0.98
56 1 1 1
57 0.90 0.80 1
58 1 1 0.89
59 0.03 0.43 0.20
60 0.43 0.59 0.52
61 1 1 1
62 1 0.96 0.94
63 0.33 0.11 1
64 0.38 0.28 0.38
65 1 1 1
66 1 1 0.50
67 1 1 0.50
68 0.56 0.55 0.56
69 0.77 0.67 0.77
70 1 0.78 0.50
71 1 0 0.50
72 0.69 0.82 0.68
73 0.87 0.82 0.89
74 1 0.80 1
75 0.71 0.85 0.83
76 1 1 1
77 0 0 0.12
78 1 0 0
79 0.65 0.38 0.75
80 0.73 1 1
81 1 1 1
82 1 1 1
83 1 0.88 1
84 1 0.50 1
85 0.63 0.16 0.86
86 1 1 1
87 0.33 0.69 0.45
88 1 1 1
89 0.83 0.42 1
90 0.25 1 1
91 1 1 1
92 0.25 0.13 0.13
93 1 1 1
94 0.42 0.29 0.36
95 0.91 0.94 1
96 1 1 1
97 1 1 0.88
98 0.50 0.50 0.50
99 0.88 0.79 0.70
100 1 1 1
101 0.83 0.57 0.83
102 0.88 1 1
103 1 1 1
104 0.98 1 1
105 0.94 0.94 0.96
106 0.60 0.59 0.48
107 0.73 0.68 0.82
108 0.48 0.46 0.69
109 0.90 0.65 0.89
110 1 0.97 1
111 0.89 0.82 0.78
112 0.78 0.94 0.94
113 1 1 0.90
114 0.80 0.80 0.90
115 0.48 0.31 0.25
116 0.53 0.54 0.63
117 1 1 1
118 0.47 1 1
119 0.38 0.67 0.67
120 0.67 0.67 1
121 0.33 0.40 0.17
122 0.08 0.50 0.39
123 0.38 0.33 0.33
124 0.33 0.33 0.33
125 0.70 0.17 0.67
126 0.23 0.67 0.67
127 1 1 1
128 0.67 0.67 0.67
129 0.50 0.25 0
130 0.58 0.50 1
131 0.67 1 1
132 0.75 1 0.67
133 0.17 0.14 0.20
134 1 1 1
135 1 1 1
136 0.50 1 1

download these results as csv

Task 1a Summary Results by Query

Simple Hit/Miss Counting [1]

MRR Method [2]


Task 1b, Hsiao's Dataset Results

Task 1b Overall Results
CSJ HAFR HL
Simple Count 0.82 0.88 0.90
MRR 0.66 0.70 0.69
Total count 410.00 410.00 410.00

download these results as csv

Task 1b Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:

TeamID TeamID Lowerbound Mean Upperbound Significance
HL HAFR -0.1073 0.0352 0.1776 FALSE
HL CSJ 0.0451 0.1875 0.3299 TRUE
HAFR CSJ 0.0099 0.1523 0.2948 TRUE

download these results as csv

2009 sqbttask1bsimple friedman mean ranks.png

MRR Method:

TeamID TeamID Lowerbound Mean Upperbound Significance
HAFR HL -0.1002 0.1328 0.3659 FALSE
HAFR CSJ -0.0260 0.2070 0.4401 FALSE
HL CSJ -0.1588 0.0742 0.3073 FALSE

download these results as csv

2009 sqbttask1bmrr friedman mean ranks.png

Task 1b Summary Results by Query Group

Simple Hit/Miss Counting

CSJ HAFR HL
1 1 1 1
2 NaN NaN NaN
3 1 1 1
4 1 1 1
5 1 1 1
6 1 1 1
7 1 1 1
8 1 1 1
9 1 1 1
10 0.67 0.67 0.67
11 1 1 1
12 0.75 1 0.50
13 1 1 1
14 1 1 1
15 NaN NaN NaN
16 1 1 1
17 1 1 1
18 1 1 1
19 1 1 1
20 1 1 1
21 0.67 1 1
22 1 1 1
23 1 0.83 0.83
24 0.67 0.67 0.67
25 0 1 1
26 0 0 0
27 1 1 1
28 NaN NaN NaN
29 1 1 1
30 1 1 1
31 NaN NaN NaN
32 1 1 1
33 1 1 1
34 1 1 1
35 NaN NaN NaN
36 0 0 0
37 0.50 0 0.50
38 1 0.86 1
39 1 1 1
40 0 0.50 0.50
41 1 1 1
42 0.67 1 1
43 1 1 1
44 0 1 0.50
45 1 1 1
46 1 1 1
47 1 1 1
48 1 1 1
49 1 0.67 1
50 1 1 1
51 1 1 1
52 1 1 1
53 0.89 0.89 0.89
54 0.33 1 1
55 0.50 1 1
56 1 0.50 1
57 1 1 1
58 NaN NaN NaN
59 1 1 1
60 1 1 1
61 0.75 0.75 1
62 0 0.50 0.50
63 1 1 1
64 NaN NaN NaN
65 1 1 1
66 NaN NaN NaN
67 1 1 1
68 NaN NaN NaN
69 0.40 1 1
70 0.20 1 1
71 NaN NaN NaN
72 1 1 1
73 1 1 0.50
74 NaN NaN NaN
75 1 1 1
76 0.40 1 0.40
77 NaN NaN NaN
78 1 1 1
79 0.50 1 1
80 1 1 0
81 0 1 1
82 0 1 1
83 1 1 1
84 1 1 1
85 0 0 1
86 1 1 1
87 1 1 1
88 0.80 0.60 0.80
89 0.80 1 0.20
90 1 1 1
91 1 1 1
92 1 1 0.80
93 0.50 1 1
94 1 0.75 1
95 0.67 0.67 0.67
96 1 1 1
97 0 0 0
98 1 1 1
99 1 1 1
100 1 1 1
101 1 1 1
102 1 1 1
103 NaN NaN NaN
104 1 1 1
105 1 1 1
106 0 0.33 0
107 NaN NaN NaN
108 1 1 1
109 1 1 1
110 0 1 1
111 1 1 1
112 0.67 1 0.67
113 1 1 1
114 0.50 0.50 0.50
115 NaN NaN NaN
116 0 0 1
117 1 1 1
118 1 1 1
119 0 0 0
120 0.67 0.67 0.67
121 1 0.67 1
122 1 1 1
123 1 1 1
124 1 1 1
125 1 0 1
126 1 1 1
127 1 1 1
128 1 1 1
129 1 1 1
130 0 0 0
131 0 0 1
132 1 1 1
133 0.33 1 0.67
134 0.50 0.50 0.50
135 1 1 1
136 1 1 1
137 1 1 1
138 1 1 1
139 0.60 1 1
140 0 0 1
141 1 1 1
142 1 1 1
143 1 1 1

download these results as csv

MRR Method

CSJ HAFR HL
1 0.56 1 1
2 NaN NaN NaN
3 0.88 0.83 0.83
4 0.50 0.75 0.50
5 1 1 1
6 1 0.50 1
7 1 1 1
8 1 1 1
9 1 1 1
10 0.44 0.50 0.67
11 1 1 1
12 0.75 0.81 0.50
13 1 1 1
14 1 0.42 1
15 NaN NaN NaN
16 1 1 1
17 1 1 1
18 1 1 1
19 1 1 1
20 0.83 1 0.61
21 0.45 1 1
22 1 1 0.88
23 0.88 0.47 0.64
24 0.67 0.67 0.67
25 0 0.75 0.15
26 0 0 0
27 0.50 0.67 0.44
28 NaN NaN NaN
29 1 1 0.75
30 1 1 0.75
31 NaN NaN NaN
32 1 0.50 1
33 0.33 0.56 0.50
34 0.78 0.61 0.58
35 NaN NaN NaN
36 0 0 0
37 0.13 0 0.25
38 0.89 0.26 0.90
39 0.18 1 1
40 0 0.07 0.08
41 1 1 0.50
42 0.50 0.83 0.75
43 0.75 0.75 0.88
44 0 0.42 0.10
45 0.33 0.56 0.22
46 0.78 1 1
47 1 1 0.75
48 1 1 1
49 0.53 0.14 0.28
50 1 1 1
51 0.10 0.13 0.17
52 1 1 1
53 0.89 0.83 0.89
54 0.11 0.37 0.32
55 0.06 1 0.75
56 0.42 0.25 0.67
57 0.80 0.90 0.60
58 NaN NaN NaN
59 1 0.50 0.50
60 1 0.63 1
61 0.50 0.75 0.54
62 0 0.50 0.50
63 1 0.67 0.50
64 NaN NaN NaN
65 0.69 0.49 1
66 NaN NaN NaN
67 0.58 0.20 0.67
68 NaN NaN NaN
69 0.05 0.85 1
70 0.20 0.41 0.90
71 NaN NaN NaN
72 1 1 0.60
73 1 1 0.50
74 NaN NaN NaN
75 1 1 1
76 0.20 0.85 0.30
77 NaN NaN NaN
78 0.67 0.83 0.67
79 0.13 0.58 0.88
80 0.13 0.33 0
81 0 0.35 0.29
82 0 0.50 0.25
83 1 1 1
84 1 0.63 1
85 0 0 0.25
86 1 1 1
87 1 1 1
88 0.67 0.43 0.47
89 0.38 0.69 0.02
90 0.83 0.78 0.53
91 0.88 1 0.88
92 1 1 0.80
93 0.50 1 1
94 0.33 0.16 0.29
95 0.67 0.67 0.67
96 1 1 1
97 0 0 0
98 1 0.75 1
99 1 1 1
100 1 1 1
101 1 1 0.80
102 1 1 1
103 NaN NaN NaN
104 0.33 1 1
105 0.35 0.33 0.40
106 0 0.11 0
107 NaN NaN NaN
108 1 1 1
109 0.33 0.33 0.33
110 0 0.10 0.17
111 0.50 0.50 0.13
112 0.50 0.78 0.20
113 1 1 1
114 0.50 0.17 0.50
115 NaN NaN NaN
116 0 0 0.60
117 0.75 0.75 0.75
118 0.83 0.67 0.33
119 0 0 0
120 0.21 0.50 0.20
121 0.56 0.67 0.71
122 0.65 0.88 0.43
123 0.79 0.78 1
124 0.70 0.70 0.70
125 0.19 0 0.23
126 1 1 1
127 0.50 0.88 0.50
128 0.56 0.78 0.50
129 1 1 1
130 0 0 0
131 0 0 0.29
132 1 0.75 0.63
133 0.03 0.48 0.15
134 0.50 0.06 0.07
135 1 1 0.78
136 0.56 1 1
137 1 1 1
138 0.23 0.38 0.33
139 0.35 0.44 0.80
140 0 0 0.50
141 0.56 0.69 0.42
142 0.50 0.50 0.39
143 1 1 1

download these results as csv

Task 1b Summary Results by Query

Simple Hit/Miss Counting [3]

MRR Method [4]

Task 2 Results

Task 2 Overall Results
CSJ HAFR HL
Simple Count 0.86 0.79 0.84
MRR 0.77 0.66 0.73
Total count 890 890 880

download these results as csv

Task 2 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:

TeamID TeamID Lowerbound Mean Upperbound Significance
CSJ HL -0.1404 0.0515 0.2434 FALSE
CSJ HAFR 0.0103 0.2022 0.3941 TRUE
HL HAFR -0.0412 0.1507 0.3426 FALSE

download these results as csv

2009 sqbttask2simple friedman mean ranks.png

MRR Method:

TeamID TeamID Lowerbound Mean Upperbound Significance
CSJ HL -0.1350 0.1066 0.3482 FALSE
CSJ HAFR 0.1702 0.4118 0.6533 TRUE
HL HAFR 0.0636 0.3051 0.5467 TRUE

download these results as csv

2009 sqbttask2mrr friedman mean ranks.png

Task 2 Summary Results by Query Group

Simple Hit/Miss Counting

CSJ HAFR HL
1 1 0.69 0.77
2 1 1 1
3 1 1 1
4 1 0 1
5 1 1 1
6 1 1 0.75
7 1 1 1
8 1 1 1
9 1 1 1
10 1 1 1
11 0.50 1 1
12 1 1 1
13 1 1 1
14 1 1 1
15 1 1 1
16 1 0.90 0.85
17 0.80 0.93 1
18 0.77 0.62 0.69
19 0.38 0.75 1
20 1 1 1
21 1 1 1
22 1 1 0.50
23 0 0 0
24 1 1 1
25 1 1 1
26 0 0 1
27 1 0.71 0.88
28 1 0.67 1
29 1 1 1
30 0.67 0.67 0.75
31 1 0 1
32 1 1 1
33 0.71 0.59 0.82
34 0.83 0.92 0.92
35 1 1 1
36 1 1 1
37 0 1 1
38 1 1 1
39 1 1 1
40 1 0.86 0.93
41 0.75 0.83 0.92
42 1 1 1
43 0.80 0.90 0.90
44 0.75 0.75 0.50
45 1 1 0.50
46 1 0.83 1
47 1 1 1
48 1 1 1
49 1 1 0.50
50 1 1 1
51 0.60 0.80 0.80
52 1 1 1
53 0.80 0.20 0.60
54 1 1 1
55 1 0.84 0.98
56 1 1 1
57 1 0.80 0.80
58 1 1 1
59 0.20 0.40 0.20
60 0.43 0.57 0.57
61 1 0.82 0.91
62 0.92 0.75 0.83
63 1 0 1
64 0.50 0.50 0.50
65 1 1 1
66 1 1 0.50
67 1 1 1
68 0.56 0.56 0.61
69 0.82 0.64 0.91
70 1 0.75 0.75
71 1 0 1
72 0.83 0.92 0.75
73 0.86 0.71 0.93
74 1 1 1
75 0.83 0.83 0.83
76 1 1 1
77 0 0 1
78 1 0 0
79 1 0.75 0.75
80 1 1 1
81 1 1 1
82 1 1 1
83 1 1 1
84 1 1 1
85 0.86 0.43 0.71
86 1 1 1
87 0.58 0.67 0.67
88 1 1 1
89 1 1 1
90 0.50 1 1
91 1 1 1
92 0.25 0.25 0.25
93 1 1 1
94 0.50 0.83 0.67
95 1 0.88 1
96 1 1 1
97 1 1 1
98 0.50 0.50 0.50
99 0.81 0.88 0.75
100 1 1 1
101 0.83 1 0.83
102 0.88 1 1
103 1 1 1
104 0.96 0.89 0.89
105 1 0.81 0.89
106 0.64 0.73 0.45
107 0.94 0.72 0.89
108 0.69 0.44 0.81
109 0.88 0.81 0.75
110 0.93 0.93 0.87
111 1 1 1
112 1 1 0.88
113 0.86 0.71 0.86
114 1 0.60 0.60
115 0.80 0.60 0.40
116 0.50 0.75 0.75
117 1 1 1
118 1 1 1
119 0.67 0.67 1
120 1 0.67 1
121 1 1 1
122 0.33 0.67 1
123 0.67 0.67 1
124 0.33 0.33 0.50
125 0.67 1 1
126 1 1 1
127 1 1 0.67
128 0.67 0.67 0.67
129 0.50 0.50 0
130 1 0.50 1
131 1 1 1
132 1 1 1
133 1 0 0
134 1 1 1
135 1 1 1
136 1 0 0

download these results as csv

MRR Method

CSJ HAFR HL
1 0.94 0.69 0.77
2 1 1 1
3 1 1 1
4 1 0 1
5 1 0.67 1
6 0.78 0.83 0.75
7 1 1 1
8 0.75 1 0.50
9 1 0.10 1
10 1 1 1
11 0.50 0.67 0.63
12 1 1 1
13 1 1 0.50
14 1 0.89 0.94
15 1 1 1
16 0.91 0.70 0.32
17 0.74 0.79 0.81
18 0.71 0.34 0.59
19 0.28 0.33 0.44
20 1 1 1
21 0.55 1 1
22 1 0.55 0.50
23 0 0 0
24 1 0.50 1
25 0.90 0.93 0.89
26 0 0 0.14
27 0.94 0.63 0.68
28 0.83 0.37 1
29 1 1 1
30 0.60 0.53 0.61
31 1 0 1
32 1 1 0.72
33 0.43 0.56 0.76
34 0.83 0.92 0.92
35 1 1 1
36 0.88 0.58 0.88
37 0 1 0.25
38 1 1 1
39 1 1 1
40 0.96 0.82 0.93
41 0.52 0.47 0.60
42 1 1 1
43 0.73 0.78 0.83
44 0.75 0.75 0.50
45 0.58 0.33 0.17
46 1 0.63 0.92
47 1 1 1
48 0.42 0.43 0.78
49 1 1 0.25
50 0.73 0.51 0.78
51 0.32 0.70 0.47
52 1 1 0.63
53 0.67 0.20 0.60
54 1 0.33 1
55 0.98 0.75 0.98
56 1 1 1
57 1 0.80 0.80
58 1 0.91 0.89
59 0.03 0.09 0.20
60 0.43 0.40 0.50
61 1 0.82 0.91
62 0.92 0.69 0.79
63 0.33 0 1
64 0.38 0.50 0.38
65 1 1 1
66 1 1 0.50
67 1 0.25 0.50
68 0.50 0.44 0.56
69 0.76 0.49 0.75
70 1 0.54 0.50
71 1 0 0.50
72 0.69 0.80 0.61
73 0.86 0.61 0.93
74 1 0.90 1
75 0.71 0.83 0.75
76 1 1 1
77 0 0 0.12
78 1 0 0
79 0.65 0.38 0.75
80 0.73 1 1
81 1 1 1
82 1 1 1
83 1 0.88 1
84 1 0.50 1
85 0.66 0.22 0.62
86 1 1 1
87 0.33 0.50 0.37
88 1 0.75 1
89 0.83 0.58 0.83
90 0.25 0.56 1
91 1 0.87 1
92 0.25 0.06 0.08
93 1 1 1
94 0.42 0.49 0.36
95 0.91 0.70 1
96 1 0.50 1
97 1 0.79 0.88
98 0.50 0.50 0.50
99 0.81 0.79 0.69
100 1 1 1
101 0.83 0.52 0.83
102 0.88 1 1
103 1 0.89 1
104 0.90 0.78 0.85
105 0.97 0.81 0.89
106 0.58 0.58 0.40
107 0.78 0.67 0.66
108 0.47 0.33 0.57
109 0.83 0.62 0.65
110 0.88 0.90 0.87
111 1 0.81 1
112 0.88 1 0.88
113 0.76 0.71 0.64
114 0.80 0.44 0.37
115 0.30 0.35 0.22
116 0.50 0.58 0.35
117 1 0.75 1
118 0.47 1 1
119 0.38 0.50 1
120 0.83 0.67 1
121 0.83 0.83 0.75
122 0.08 0.25 0.25
123 0.40 0.44 0.57
124 0.33 0.17 0.50
125 0.67 0.28 0.75
126 0.48 1 0.75
127 1 1 0.67
128 0.67 0.67 0.67
129 0.50 0.05 0
130 1 0.50 1
131 1 1 1
132 0.75 0.60 1
133 0.14 0 0
134 1 1 1
135 1 1 1
136 0.20 0 0

download these results as csv

Task 2 Summary Results by Query

Simple Hit/Miss Counting [5]

MRR Method [6]

Runtime Results

Participant Task Runtime (min) Machine
CSJ 1a ~30 BEER 2
CSJ 1b 7 BEER 2
CSJ 2 ~35 BEER 2
HAFR 1a 15 BEER 2
HAFR 1b 5 BEER 2
HAFR 2 35 BEER 2
HL 1a ~95 FAST
HL 1b ~40 FAST
HL 2 ~95 FAST

download these results as csv

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2009:Query-by-Tapping_Results&oldid=7418"