2008:Query-by-Singing/Humming Results

From MIREX Wiki

Introduction

These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the 2008:Query by Singing/Humming page.

Task Descriptions

Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. See ESAC Data Homepage for more information about the Essen Collection. The query database consists of 2797 sung queries.

Task 2 Goto Task 2 Results: The second subtask is the same as last year too. In the second subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.

General Legend

Team ID

JL1 = J-S. R. Jang 1
JL2 = J-S. R. Jang 2
JL3 = J-S. R. Jang 3
JL4 = J-S. R. Jang 4
RK = M. Ryynänen, A. Klapuri
LW1 = L. Wang 1
LW2 = L. Wang 2
LW3 = L. Wang 3
WL1 = X. Wu, M. Li 1
WL2 = X. Wu, M. Li 2

Task 1 Results

Task 1 Overall Results
songID JL1 JL2 JL3 JL4 LW1 RK WL1 WL2
Avg. MRR 0.81 0.69 0.82 0.72 0.93 0.93 0.92 0.91

download these results as csv

Task 1 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

TeamID TeamID Lowerbound Mean Upperbound Significance
JL1 JL2 0.2791 1.7708 3.2626 TRUE
JL1 JL3 -2.1063 -0.6146 0.8772 FALSE
JL1 JL4 -0.3147 1.1771 2.6688 FALSE
JL1 LW1 -4.5334 -3.0417 -1.5499 TRUE
JL1 RK -4.3980 -2.9062 -1.4145 TRUE
JL1 WL1 -4.5751 -3.0833 -1.5916 TRUE
JL1 WL2 -3.6272 -2.1354 -0.6437 TRUE
JL2 JL3 -3.8772 -2.3854 -0.8937 TRUE
JL2 JL4 -2.0855 -0.5938 0.8980 FALSE
JL2 LW1 -6.3043 -4.8125 -3.3207 TRUE
JL2 RK -6.1688 -4.6771 -3.1853 TRUE
JL2 WL1 -6.3459 -4.8542 -3.3624 TRUE
JL2 WL2 -5.3980 -3.9063 -2.4145 TRUE
JL3 JL4 0.2999 1.7917 3.2834 TRUE
JL3 LW1 -3.9188 -2.4271 -0.9353 TRUE
JL3 RK -3.7834 -2.2917 -0.7999 TRUE
JL3 WL1 -3.9605 -2.4688 -0.9770 TRUE
JL3 WL2 -3.0126 -1.5208 -0.0291 TRUE
JL4 LW1 -5.7105 -4.2188 -2.7270 TRUE
JL4 RK -5.5751 -4.0833 -2.5916 TRUE
JL4 WL1 -5.7522 -4.2604 -2.7687 TRUE
JL4 WL2 -4.8043 -3.3125 -1.8207 TRUE
LW1 RK -1.3563 0.1354 1.6272 FALSE
LW1 WL1 -1.5334 -0.0417 1.4501 FALSE
LW1 WL2 -0.5855 0.9063 2.3980 FALSE
RK WL1 -1.6688 -0.1771 1.3147 FALSE
RK WL2 -0.7209 0.7708 2.2626 FALSE
WL1 WL2 -0.5438 0.9479 2.4397 FALSE

download these results as csv

2008 qbsh.task1.friedman.small.png

Task 1 Summary Results by Query Group

Query Group JL1 JL2 JL3 JL4 LW1 RK WL1 WL2
1 0.82 0.67 0.94 0.78 1.00 0.93 1.00 1.00
2 0.78 0.64 0.65 0.59 0.90 0.69 0.45 0.44
3 0.75 0.75 0.88 0.77 0.88 1.00 0.88 0.88
4 1.00 0.94 0.85 0.81 1.00 1.00 1.00 0.94
5 0.56 0.22 1.00 0.56 1.00 0.90 1.00 0.92
6 1.00 0.90 1.00 1.00 1.00 0.75 1.00 1.00
7 0.94 0.81 1.00 0.93 1.00 0.89 0.94 0.90
8 0.56 0.63 0.83 0.94 1.00 1.00 1.00 1.00
9 0.88 0.81 1.00 0.88 1.00 1.00 1.00 1.00
10 0.80 1.00 0.90 0.95 1.00 1.00 1.00 1.00
11 0.85 0.75 0.84 0.78 0.95 0.92 0.93 0.92
12 0.85 0.76 0.87 0.86 0.94 0.93 0.93 0.93
13 0.84 0.76 0.86 0.78 0.93 0.95 0.96 0.95
14 0.82 0.64 0.66 0.43 0.90 0.94 0.94 0.92
15 0.77 0.79 0.87 0.87 0.84 0.87 0.83 0.83
16 0.82 0.78 0.86 0.81 0.95 0.87 0.94 0.92
17 0.74 0.48 0.75 0.59 0.96 0.85 0.96 0.94
18 0.87 0.69 0.87 0.79 0.99 0.99 1.00 0.99
19 0.92 0.81 0.92 0.83 0.97 0.95 0.95 0.94
20 0.89 0.78 0.86 0.85 0.99 0.97 0.97 0.96
21 0.65 0.56 0.63 0.56 0.64 0.77 0.66 0.66
22 0.89 0.87 0.90 0.86 0.98 0.97 0.99 0.99
23 0.84 0.68 0.91 0.81 0.95 0.97 0.96 0.94
24 0.76 0.65 0.64 0.64 0.72 0.88 0.79 0.77
25 0.46 0.30 0.64 0.45 0.60 0.78 0.71 0.69
26 0.88 0.74 0.90 0.86 0.95 0.90 0.90 0.88
27 0.84 0.65 0.84 0.70 0.96 0.91 0.97 0.97
28 0.41 0.22 0.40 0.18 0.93 0.67 0.84 0.81
29 0.79 0.73 0.72 0.64 0.96 0.97 0.96 0.94
30 0.83 0.65 0.87 0.65 0.98 0.94 0.94 0.91
31 0.86 0.81 0.70 0.66 0.97 0.99 1.00 0.97
32 0.86 0.68 0.90 0.79 0.98 0.98 0.97 0.95
33 0.70 0.53 0.91 0.71 0.94 0.96 0.72 0.65
34 0.79 0.66 0.87 0.69 0.97 0.93 0.93 0.90
35 0.92 0.75 0.85 0.70 0.97 0.95 0.92 0.89
36 0.85 0.81 0.88 0.71 0.88 1.00 0.96 0.92
37 0.74 0.53 0.80 0.68 0.93 0.90 0.95 0.93
38 0.87 0.87 0.84 0.79 0.93 0.96 0.91 0.93
39 0.81 0.71 0.85 0.76 0.76 0.85 0.89 0.91
40 0.73 0.60 0.82 0.73 0.91 0.91 0.90 0.88
41 0.76 0.69 0.74 0.64 0.94 0.99 0.96 0.96
42 0.84 0.70 0.89 0.73 0.95 0.93 0.95 0.92
43 0.89 0.58 0.88 0.67 0.98 0.94 0.97 0.95
44 0.82 0.78 0.94 0.92 1.00 0.98 0.96 0.96
45 0.82 0.76 0.87 0.82 0.92 0.86 0.93 0.90
46 0.80 0.64 0.85 0.80 0.92 0.99 0.96 0.95
47 0.78 0.65 0.86 0.75 0.93 1.00 0.97 0.97
48 0.80 0.75 0.78 0.70 0.88 0.94 0.94 0.96

download these results as csv

Task 2 Results

In this subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.

Task 2 Overall Results
songID JL1 JL2 JL3 JL4 LW2 LW3 RK WL1 WL2
Avg. MRR 0.32 0.53 0.32 0.59 0.92 0.94 0.89 0.94 0.92

download these results as csv

Task 2 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

TeamID TeamID Lowerbound Mean Upperbound Significance
JL1 JL2 -1.9291 -0.9198 0.0895 FALSE
JL1 JL3 -1.0093 0.0000 1.0093 FALSE
JL1 JL4 -2.5282 -1.5189 -0.5096 TRUE
JL1 LW2 -4.9574 -3.9481 -2.9388 TRUE
JL1 LW3 -5.1650 -4.1557 -3.1463 TRUE
JL1 RK -4.7169 -3.7075 -2.6982 TRUE
JL1 WL1 -5.2027 -4.1934 -3.1841 TRUE
JL1 WL2 -4.8961 -3.8868 -2.8775 TRUE
JL2 JL3 -0.0895 0.9198 1.9291 FALSE
JL2 JL4 -1.6084 -0.5991 0.4103 FALSE
JL2 LW2 -4.0376 -3.0283 -2.0190 TRUE
JL2 LW3 -4.2452 -3.2358 -2.2265 TRUE
JL2 RK -3.7970 -2.7877 -1.7784 TRUE
JL2 WL1 -4.2829 -3.2736 -2.2643 TRUE
JL2 WL2 -3.9763 -2.9670 -1.9577 TRUE
JL3 JL4 -2.5282 -1.5189 -0.5096 TRUE
JL3 LW2 -4.9574 -3.9481 -2.9388 TRUE
JL3 LW3 -5.1650 -4.1557 -3.1463 TRUE
JL3 RK -4.7169 -3.7075 -2.6982 TRUE
JL3 WL1 -5.2027 -4.1934 -3.1841 TRUE
JL3 WL2 -4.8961 -3.8868 -2.8775 TRUE
JL4 LW2 -3.4386 -2.4292 -1.4199 TRUE
JL4 LW3 -3.6461 -2.6368 -1.6275 TRUE
JL4 RK -3.1980 -2.1887 -1.1794 TRUE
JL4 WL1 -3.6838 -2.6745 -1.6652 TRUE
JL4 WL2 -3.3772 -2.3679 -1.3586 TRUE
LW2 LW3 -1.2169 -0.2075 0.8018 FALSE
LW2 RK -0.7687 0.2406 1.2499 FALSE
LW2 WL1 -1.2546 -0.2453 0.7640 FALSE
LW2 WL2 -0.9480 0.0613 1.0706 FALSE
LW3 RK -0.5612 0.4481 1.4574 FALSE
LW3 WL1 -1.0470 -0.0377 0.9716 FALSE
LW3 WL2 -0.7404 0.2689 1.2782 FALSE
RK WL1 -1.4952 -0.4858 0.5235 FALSE
RK WL2 -1.1886 -0.1792 0.8301 FALSE
WL1 WL2 -0.7027 0.3066 1.3159 FALSE

download these results as csv

2008 qbsh.task2.friedman.s.png

Task 2 Summary Results by Query Group

Query Group JL1 JL2 JL3 JL4 LW2 LW3 RK WL1 WL2
1 0.32 0.74 0.28 0.62 1.00 1.00 0.91 0.95 0.93
2 0.60 0.70 0.40 0.61 1.00 1.00 1.00 1.00 1.00
3 0.00 0.00 0.00 0.50 0.11 0.00 1.00 0.25 0.11
4 0.00 0.50 0.25 0.75 0.75 1.00 1.00 1.00 1.00
5 0.00 0.00 0.00 0.00 1.00 1.00 0.00 0.00 0.00
6 0.00 0.56 0.00 0.11 0.67 0.68 0.83 0.69 0.68
7 0.83 0.83 0.68 0.45 1.00 1.00 1.00 1.00 1.00
8 0.04 0.00 0.11 0.05 0.75 1.00 1.00 1.00 1.00
9 0.00 0.60 0.00 0.70 1.00 1.00 1.00 1.00 1.00
10 0.00 0.50 0.00 0.50 1.00 1.00 1.00 1.00 1.00
11 0.00 0.22 0.00 1.00 1.00 1.00 1.00 0.75 0.63
12 0.33 0.11 0.00 0.07 1.00 1.00 1.00 1.00 1.00
13 0.03 0.60 0.23 0.60 1.00 1.00 0.80 1.00 1.00
14 0.00 0.63 0.00 0.56 1.00 1.00 0.50 1.00 1.00
15 0.00 0.17 0.00 0.00 0.67 1.00 0.13 1.00 0.60
16 0.38 0.38 0.31 0.39 1.00 1.00 0.75 1.00 1.00
17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
18 0.00 0.63 0.25 0.17 0.50 0.50 0.50 1.00 1.00
19 0.00 1.00 0.00 0.75 1.00 1.00 1.00 1.00 1.00
20 0.00 0.00 0.00 0.00 1.00 1.00 0.75 0.42 0.33
21 0.16 0.03 0.03 0.60 0.75 1.00 0.63 0.55 0.53
22 0.39 0.58 0.44 0.62 0.74 0.81 1.00 0.92 0.94
23 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
24 0.00 1.00 0.00 1.00 1.00 1.00 1.00 1.00 1.00
25 0.50 0.67 0.50 1.00 1.00 1.00 1.00 1.00 1.00
26 0.42 0.50 0.30 0.60 1.00 1.00 0.92 1.00 1.00
27 0.06 0.75 0.25 0.65 1.00 1.00 1.00 1.00 1.00
28 0.50 0.58 0.33 0.80 1.00 1.00 0.63 1.00 1.00
29 0.33 0.05 1.00 1.00 1.00 1.00 0.25 1.00 1.00
30 1.00 1.00 0.75 0.58 1.00 1.00 1.00 1.00 1.00
31 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
32 0.00 0.50 0.00 0.71 1.00 1.00 0.86 1.00 1.00
33 0.00 0.80 0.00 0.80 0.63 0.67 1.00 0.72 0.66
34 0.00 0.25 0.50 0.00 0.50 0.50 0.58 0.50 0.50
35 0.00 0.00 0.00 0.00 1.00 1.00 1.00 1.00 1.00
36 0.00 0.67 0.00 0.52 0.82 0.80 0.87 0.87 0.80
37 0.00 0.00 0.00 1.00 1.00 1.00 0.11 0.06 0.00
38 0.50 0.55 0.50 1.00 0.63 1.00 1.00 1.00 1.00
39 0.36 0.73 0.43 1.00 1.00 1.00 0.90 1.00 1.00
40 0.54 0.50 1.00 0.50 1.00 1.00 1.00 1.00 1.00
41 0.25 0.57 0.06 0.56 0.78 0.81 0.88 0.88 0.88
42 0.63 0.38 0.59 0.68 1.00 1.00 0.92 1.00 0.88
43 0.48 0.35 0.67 0.54 0.73 0.85 0.81 0.83 0.76
44 0.48 0.68 0.56 0.83 1.00 1.00 1.00 1.00 1.00
45 0.69 0.67 1.00 0.67 1.00 1.00 1.00 1.00 1.00
46 0.00 0.25 0.00 0.25 1.00 1.00 0.29 1.00 1.00
47 0.75 0.63 0.50 0.51 1.00 1.00 1.00 1.00 1.00
48 0.07 0.00 0.00 1.00 1.00 1.00 1.00 1.00 1.00
49 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
50 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
51 0.05 0.25 0.25 0.26 1.00 1.00 1.00 1.00 1.00
52 0.08 0.54 0.25 0.54 1.00 1.00 0.77 1.00 1.00
53 0.33 0.55 0.33 1.00 1.00 1.00 1.00 1.00 1.00
54 0.20 0.66 0.13 0.60 1.00 1.00 1.00 1.00 1.00
55 0.00 0.20 0.00 0.63 1.00 1.00 1.00 1.00 1.00
56 0.50 0.50 0.50 0.50 0.50 0.50 0.58 0.50 0.50
57 0.90 0.90 1.00 1.00 1.00 1.00 1.00 1.00 1.00
58 0.00 1.00 0.00 1.00 1.00 1.00 1.00 1.00 1.00
59 0.00 0.73 0.00 1.00 1.00 1.00 1.00 1.00 1.00
60 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
61 0.00 0.50 0.00 0.08 1.00 1.00 0.00 1.00 1.00
62 0.08 0.67 0.33 0.67 1.00 1.00 1.00 1.00 1.00
63 0.63 0.53 0.53 0.57 1.00 1.00 1.00 1.00 1.00
64 0.00 1.00 0.00 1.00 1.00 1.00 1.00 1.00 1.00
65 0.00 0.08 0.00 0.17 1.00 1.00 1.00 1.00 1.00
66 0.50 0.75 0.50 0.55 1.00 1.00 1.00 1.00 1.00
67 0.00 0.83 0.00 1.00 1.00 1.00 0.67 1.00 1.00
68 1.00 0.54 0.55 0.54 0.75 1.00 0.50 0.67 0.67
69 0.00 0.03 0.00 0.50 1.00 1.00 1.00 1.00 1.00
70 1.00 0.88 0.81 0.79 1.00 1.00 1.00 1.00 1.00
71 0.00 0.37 0.00 0.67 0.75 1.00 1.00 0.42 0.42
72 1.00 0.71 1.00 1.00 1.00 1.00 1.00 1.00 1.00
73 0.75 0.75 0.65 0.75 1.00 1.00 1.00 1.00 1.00
74 1.00 0.54 1.00 1.00 1.00 1.00 1.00 1.00 1.00
75 0.17 0.35 0.33 0.38 1.00 1.00 0.83 1.00 1.00
76 0.00 0.30 0.00 0.75 1.00 1.00 1.00 1.00 0.55
77 1.00 0.80 0.64 0.45 1.00 1.00 1.00 1.00 1.00
78 0.00 0.33 0.07 0.00 1.00 1.00 1.00 1.00 1.00
79 0.50 0.50 0.50 0.50 1.00 1.00 1.00 1.00 1.00
80 0.33 0.73 0.00 0.67 1.00 1.00 1.00 1.00 1.00
81 0.17 0.60 0.17 0.58 0.92 0.92 1.00 1.00 0.92
82 0.00 0.75 0.00 1.00 1.00 1.00 1.00 1.00 1.00
83 0.00 0.05 0.02 0.22 0.36 0.50 1.00 1.00 1.00
84 0.05 0.02 0.00 0.00 0.27 0.34 0.54 0.75 0.75
85 0.50 1.00 0.50 1.00 1.00 1.00 1.00 1.00 1.00
86 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
87 0.00 1.00 0.00 0.45 1.00 1.00 1.00 1.00 1.00
88 0.00 0.00 0.00 0.00 1.00 1.00 0.17 1.00 1.00
89 0.02 0.16 0.43 0.36 0.93 0.93 0.93 0.86 0.86
90 0.00 0.11 0.00 0.60 1.00 1.00 1.00 1.00 1.00
91 0.00 0.25 0.00 0.50 1.00 1.00 1.00 1.00 1.00
92 0.50 0.54 0.50 0.50 1.00 1.00 1.00 1.00 1.00
93 0.25 0.17 0.50 1.00 1.00 1.00 1.00 1.00 1.00
94 0.00 0.00 0.00 0.00 1.00 1.00 1.00 1.00 1.00
95 0.00 1.00 0.00 1.00 1.00 1.00 1.00 1.00 1.00
96 0.00 0.00 1.00 0.05 1.00 1.00 0.00 1.00 1.00
97 0.00 0.00 0.00 0.00 1.00 1.00 1.00 1.00 1.00
98 0.00 0.00 0.00 0.00 1.00 1.00 0.00 1.00 1.00
99 0.55 0.03 0.38 0.15 1.00 1.00 0.63 1.00 0.88
100 0.33 1.00 0.33 1.00 0.68 0.67 1.00 1.00 1.00
101 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
102 0.00 0.00 0.00 0.00 1.00 1.00 1.00 1.00 1.00
103 0.27 0.39 0.26 0.38 0.94 1.00 1.00 1.00 1.00
104 0.00 0.67 0.00 0.67 1.00 1.00 1.00 1.00 1.00
105 1.00 0.00 0.17 0.08 0.00 0.00 0.11 0.14 0.05
106 0.00 0.36 0.00 0.35 1.00 1.00 0.61 1.00 0.69

download these results as csv

Runtime Results

Participant Task Runtime (min'sec") Machine
JL1 1 20'10" FAST
JL2 1 417'36" FAST
JL3 1 54'25" FAST
JL4 1 297'32" FAST2
LW1 1 143'40" FAST2
RK 1 71'52" cluster
WL1 1 31'03" FAST
WL2 1 28'19" FAST
JL1 2 2'08" FAST2
JL2 2 54'05" FAST2
JL3 2 7'32" FAST2
JL4 2 59'23" FAST2
LW2 2 561'42" FAST2
LW3 2 62'39" FAST2
RK 2 19'36" cluster
WL1 2 8'30" FAST
WL2 2 6'03" FAST

download these results as csv

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2008:Query-by-Singing/Humming_Results&oldid=7393"