Difference between revisions of "2007:Query-by-Singing/Humming Results"

From MIREX Wiki
(Task 2 Overall Results)
(Team ID)
 
(11 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
==Introduction==
 
==Introduction==
These are the results for the 2007 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the [[Query by Singing/Humming]] page.  
+
These are the results for the 2007 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the [[2007:Query by Singing/Humming]] page.  
  
 
===Task Descriptions===
 
===Task Descriptions===
  
'''Task 1 [[#Task 1 Results|Goto Task 1 Results]]''': The first subtask is the same as last years. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. The query database consists of 2797 sung queries.  
+
'''Task 1 [[#Task 1 Results|Goto Task 1 Results]]''': The first subtask is the same as last years. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files.  See [http://www.esac-data.org/  ESAC Data Homepage] for more information about the Essen Collection. The query database consists of 2797 sung queries.  
  
 
'''Task 2 [[#Task 2 Results|Goto Task 2 Results]]''': In the second subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.
 
'''Task 2 [[#Task 2 Results|Goto Task 2 Results]]''': In the second subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.
Line 10: Line 10:
 
===General Legend===
 
===General Legend===
 
====Team ID====
 
====Team ID====
'''FH''' = [https://www.music-ir.org/mirex2007/abs/QBSH_ferraro.pdf Pascal Ferraro, Pierre Hanna, Julien Allali, Matthias Robine]<br />
+
'''FH''' = [https://www.music-ir.org/mirex/abstracts/2007/QBSH_ferraro.pdf Pascal Ferraro, Pierre Hanna, Julien Allali, Matthias Robine]<br />
'''CG''' = [https://www.music-ir.org/mirex2007/abs/QBSH_SMS_gomez.pdf Carlos G├│mez, Soraya Abad-Mota, Edna Ruckhaus]<br />
+
'''CG''' = [https://www.music-ir.org/mirex/abstracts/2007/QBSH_SMS_gomez.pdf Carlos Gómez, Soraya Abad-Mota, Edna Ruckhaus]<br />
'''RJ1''' = [https://www.music-ir.org/mirex2007/abs/QBSH_jang.pdf J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 1]<br />
+
'''RJ1''' = [https://www.music-ir.org/mirex/abstracts/2007/QBSH_jang.pdf J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 1]<br />
'''RJ2''' = [https://www.music-ir.org/mirex2007/abs/QBSH_jang.pdf J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 2]<br />
+
'''RJ2''' = [https://www.music-ir.org/mirex/abstracts/2007/QBSH_jang.pdf J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 2]<br />
'''NM''' = [https://www.music-ir.org/mirex2007/abs/QBSH_lemstrom.pdf Kjell Lemström, Niko Mikkilä]<br />
+
'''NM''' = [https://www.music-ir.org/mirex/abstracts/2007/QBSH_lemstrom.pdf Kjell Lemström, Niko Mikkilä]<br />
'''XW1''' = [https://www.music-ir.org/mirex2007/abs/QBSH_wu.pdf Xiao Wu, Ming Li 1]<br />
+
'''XW1''' = [https://www.music-ir.org/mirex/abstracts/2007/QBSH_wu.pdf Xiao Wu, Ming Li 1]<br />
'''XW2''' = [https://www.music-ir.org/mirex2007/abs/QBSH_wu.pdf Xiao Wu, Ming Li 2]<br />
+
'''XW2''' = [https://www.music-ir.org/mirex/abstracts/2007/QBSH_wu.pdf Xiao Wu, Ming Li 2]<br />
'''AU1''' = [https://www.music-ir.org/mirex2007/abs/QBSH_SMS_uitdenbogerd.pdf Alexandra L. Uitdenbogerd 1]<br />
+
'''AU1''' = [https://www.music-ir.org/mirex/abstracts/2007/QBSH_SMS_uitdenbogerd.pdf Alexandra L. Uitdenbogerd 1]<br />
'''AU2''' = [https://www.music-ir.org/mirex2007/abs/QBSH_SMS_uitdenbogerd.pdf Alexandra L. Uitdenbogerd 2]<br />
+
'''AU2''' = [https://www.music-ir.org/mirex/abstracts/2007/QBSH_SMS_uitdenbogerd.pdf Alexandra L. Uitdenbogerd 2]<br />
'''AU3''' = [https://www.music-ir.org/mirex2007/abs/QBSH_SMS_uitdenbogerd.pdf Alexandra L. Uitdenbogerd 3]<br />
+
'''AU3''' = [https://www.music-ir.org/mirex/abstracts/2007/QBSH_SMS_uitdenbogerd.pdf Alexandra L. Uitdenbogerd 3]<br />
  
 
===Task 1 Results===
 
===Task 1 Results===
Line 25: Line 25:
  
 
=====Task 1 Overall Results=====
 
=====Task 1 Overall Results=====
<csv>qbsh07_task_1_overall.csv</csv>
+
<csv>2007/qbsh07_task1_overall.csv</csv>
 
 
  
 
====Task 1 Friedman's Test for Significant Differences====
 
====Task 1 Friedman's Test for Significant Differences====
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
<csv>qbsh07_task1_sum_friedmans.csv</csv>
+
<csv>2007/qbsh07_task1_sum_friedmans.csv</csv>
<csv>qbsh07_task1_detail_friedmans.csv</csv>
+
<csv>2007/qbsh07_task1_detail_friedmans.csv</csv>
[[Image:Qbsh07_task1_friedmans.png]]
+
[[Image:2007_qbsh07_task1_friedmans.png]]
  
 
====Task 1 Summary Results by Query Group====
 
====Task 1 Summary Results by Query Group====
<csv>qbsh07_task1_avg_per_group.csv</csv>
+
<csv>2007/qbsh07_task1_avg_per_group.csv</csv>
  
 
===Task 2 Results===
 
===Task 2 Results===
Line 44: Line 43:
 
=====Team ID=====
 
=====Team ID=====
 
'''FH_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_ferraro.pdf Pascal Ferraro, Pierre Hanna, Julien Allali, Matthias Robine based on XW note transcriber]<br />
 
'''FH_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_ferraro.pdf Pascal Ferraro, Pierre Hanna, Julien Allali, Matthias Robine based on XW note transcriber]<br />
'''CG_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_SMS_gomez.pdf Carlos G├│mez, Soraya Abad-Mota, Edna Ruckhaus based on XW note transcriber]<br />
+
'''CG_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_SMS_gomez.pdf Carlos Gómez, Soraya Abad-Mota, Edna Ruckhaus based on XW note transcriber]<br />
 
'''RJ1_RJ''' = [https://www.music-ir.org/mirex2007/abs/QBSH_jang.pdf J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 1 based on RJ pitch transcriber]<br />
 
'''RJ1_RJ''' = [https://www.music-ir.org/mirex2007/abs/QBSH_jang.pdf J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 1 based on RJ pitch transcriber]<br />
 
'''RJ1_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_jang.pdf J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 1 based on XW pitch transcriber]<br />
 
'''RJ1_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_jang.pdf J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 1 based on XW pitch transcriber]<br />
 
'''RJ2_RJ''' = [https://www.music-ir.org/mirex2007/abs/QBSH_jang.pdf J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 2 based on RJ pitch transcriber]<br />
 
'''RJ2_RJ''' = [https://www.music-ir.org/mirex2007/abs/QBSH_jang.pdf J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 2 based on RJ pitch transcriber]<br />
 
'''RJ2_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_jang.pdf J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 2 based on XW pitch transcriber]<br />
 
'''RJ2_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_jang.pdf J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 2 based on XW pitch transcriber]<br />
'''NM_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_lemstrom.pdf Kjell Lemström, Niko Mikkilä based on XW note transcriber]<br />
+
'''NM_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_lemstrom.pdf Kjell Lemström, Niko Mikkilä based on XW note transcriber]<br />
 
'''XW1_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_wu.pdf Xiao Wu, Ming Li 1 based on XW note transcriber]<br />
 
'''XW1_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_wu.pdf Xiao Wu, Ming Li 1 based on XW note transcriber]<br />
 
'''XW2_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_wu.pdf Xiao Wu, Ming Li 2 based on XW pitch transcriber]<br />
 
'''XW2_XW''' = [https://www.music-ir.org/mirex2007/abs/QBSH_wu.pdf Xiao Wu, Ming Li 2 based on XW pitch transcriber]<br />
  
 
=====Task 2 Overall Results=====
 
=====Task 2 Overall Results=====
<csv>qbsh07_task2_overall.csv</csv>
+
<csv>2007/qbsh07_task2_overall.csv</csv>
  
 
====Task 2 Friedman's Test for Significant Differences====
 
====Task 2 Friedman's Test for Significant Differences====
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
<csv>qbsh07_task2_sum_friedmans.csv</csv>
+
<csv>2007/qbsh07_task2_sum_friedmans.csv</csv>
<csv>qbsh07_task2_detail_friedmans.csv</csv>
+
<csv>2007/qbsh07_task2_detail_friedmans.csv</csv>
[[Image:Qbsh07_task2_friedmans.png]]
+
[[Image:2007_qbsh07_task2_friedmans.png]]
  
 
====Task 2 Summary Results by Query Group====
 
====Task 2 Summary Results by Query Group====
<csv>qbsh07_task2_avg_per_group.csv</csv>
+
<csv>2007/qbsh07_task2_avg_per_group.csv</csv>
  
  
 
[[Category: Results]]
 
[[Category: Results]]

Latest revision as of 11:43, 26 July 2010

Introduction

These are the results for the 2007 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the 2007:Query by Singing/Humming page.

Task Descriptions

Task 1 Goto Task 1 Results: The first subtask is the same as last years. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. See ESAC Data Homepage for more information about the Essen Collection. The query database consists of 2797 sung queries.

Task 2 Goto Task 2 Results: In the second subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.

General Legend

Team ID

FH = Pascal Ferraro, Pierre Hanna, Julien Allali, Matthias Robine
CG = Carlos Gómez, Soraya Abad-Mota, Edna Ruckhaus
RJ1 = J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 1
RJ2 = J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 2
NM = Kjell Lemström, Niko Mikkilä
XW1 = Xiao Wu, Ming Li 1
XW2 = Xiao Wu, Ming Li 2
AU1 = Alexandra L. Uitdenbogerd 1
AU2 = Alexandra L. Uitdenbogerd 2
AU3 = Alexandra L. Uitdenbogerd 3

Task 1 Results

The first subtask is the same as last years. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. The query database consists of 2797 sung queries.

Task 1 Overall Results
Teams AU1 AU2 AU3 CG NM FH RJ1 RJ2 XW1 XW2
Task 1 (MRR) 0.240 0.093 0.110 0.477 0.576 0.355 0.704 0.872 0.909 0.925

download these results as csv

Task 1 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Friedman's ANOVA Table
Source SS df MS Chi-sq Prob>Chi-sq
Columns 3.5829e+003 9 398.0984 393.0438 0
Error 355.1146 423 0.8395
Total 3938 479

download these results as csv

TeamID TeamID Lowerbound Mean Upperbound Significance
AU1 AU2 -0.3039 1.6458 3.5956 FALSE
AU1 AU3 -0.4706 1.4792 3.4289 FALSE
AU1 CG -4.1164 -2.1667 -0.2169 TRUE
AU1 NM -4.6998 -2.75 -0.8002 TRUE
AU1 FH -2.6373 -0.6875 1.2623 FALSE
AU1 RJ1 -5.5435 -3.5938 -1.644 TRUE
AU1 RJ2 -7.106 -5.1563 -3.2065 TRUE
AU1 XW1 -7.5539 -5.6042 -3.6544 TRUE
AU1 XW2 -8.2414 -6.2917 -4.3419 TRUE
AU2 AU3 -2.1164 -0.1667 1.7831 FALSE
AU2 CG -5.7623 -3.8125 -1.8627 TRUE
AU2 NM -6.3456 -4.3958 -2.4461 TRUE
AU2 FH -4.2831 -2.3333 -0.3836 TRUE
AU2 RJ1 -7.1894 -5.2396 -3.2898 TRUE
AU2 RJ2 -8.7519 -6.8021 -4.8523 TRUE
AU2 XW1 -9.1998 -7.25 -5.3002 TRUE
AU2 XW2 -9.8873 -7.9375 -5.9877 TRUE
AU3 CG -5.5956 -3.6458 -1.6961 TRUE
AU3 NM -6.1789 -4.2292 -2.2794 TRUE
AU3 FH -4.1164 -2.1667 -0.2169 TRUE
AU3 RJ1 -7.0227 -5.0729 -3.1231 TRUE
AU3 RJ2 -8.5852 -6.6354 -4.6856 TRUE
AU3 XW1 -9.0331 -7.0833 -5.1336 TRUE
AU3 XW2 -9.7206 -7.7708 -5.8211 TRUE
CG NM -2.5331 -0.5833 1.3664 FALSE
CG FH -0.4706 1.4792 3.4289 FALSE
CG RJ1 -3.3769 -1.4271 0.5227 FALSE
CG RJ2 -4.9394 -2.9896 -1.0398 TRUE
CG XW1 -5.3873 -3.4375 -1.4877 TRUE
CG XW2 -6.0748 -4.125 -2.1752 TRUE
NM FH 0.1127 2.0625 4.0123 TRUE
NM RJ1 -2.7935 -0.8438 1.106 FALSE
NM RJ2 -4.356 -2.4063 -0.4565 TRUE
NM XW1 -4.8039 -2.8542 -0.9044 TRUE
NM XW2 -5.4914 -3.5417 -1.5919 TRUE
FH RJ1 -4.856 -2.9063 -0.9565 TRUE
FH RJ2 -6.4185 -4.4688 -2.519 TRUE
FH XW1 -6.8664 -4.9167 -2.9669 TRUE
FH XW2 -7.5539 -5.6042 -3.6544 TRUE
RJ1 RJ2 -3.5123 -1.5625 0.3873 FALSE
RJ1 XW1 -3.9602 -2.0104 -0.0606 TRUE
RJ1 XW2 -4.6477 -2.6979 -0.7481 TRUE
RJ2 XW1 -2.3977 -0.4479 1.5019 FALSE
RJ2 XW2 -3.0852 -1.1354 0.8144 FALSE
XW1 XW2 -2.6373 -0.6875 1.2623 FALSE

download these results as csv 2007 qbsh07 task1 friedmans.png

Task 1 Summary Results by Query Group

Group ID Count AU1 AU2 AU3 CG NM FH RJ1 RJ2 XW1 XW2
1 9 0.023 0.010 0.006 0.423 0.726 0.347 0.722 0.944 1.000 1.000
2 9 0.059 0.120 0.117 0.598 0.578 0.056 0.778 0.806 0.444 0.455
3 8 0.500 0.250 0.188 1.000 0.700 0.438 0.875 0.875 0.875 0.875
4 9 0.519 0.444 0.472 0.786 0.901 0.467 1.000 1.000 0.944 1.000
5 9 0.333 0.222 0.229 0.309 0.494 0.278 0.083 0.561 0.917 1.000
6 8 0.188 0.000 0.031 0.380 0.563 0.080 0.760 1.000 1.000 1.000
7 9 0.236 0.111 0.111 0.346 0.722 0.287 0.889 0.944 0.897 0.944
8 9 0.160 0.111 0.111 0.401 0.615 0.228 0.800 0.562 1.000 1.000
9 8 0.196 0.375 0.375 0.688 0.608 0.138 0.503 1.000 1.000 1.000
10 10 0.426 0.064 0.032 0.684 0.810 0.660 1.000 0.825 1.000 1.000
11 62 0.349 0.142 0.194 0.541 0.627 0.519 0.685 0.922 0.922 0.929
12 99 0.352 0.119 0.212 0.684 0.701 0.498 0.838 0.926 0.932 0.934
13 105 0.111 0.068 0.094 0.302 0.640 0.343 0.810 0.922 0.946 0.956
14 99 0.378 0.170 0.235 0.643 0.661 0.347 0.740 0.907 0.923 0.940
15 15 0.157 0.101 0.185 0.401 0.561 0.229 0.636 0.794 0.833 0.826
16 101 0.493 0.173 0.226 0.529 0.683 0.568 0.703 0.874 0.924 0.940
17 95 0.099 0.060 0.045 0.224 0.425 0.065 0.410 0.787 0.942 0.961
18 104 0.244 0.173 0.211 0.562 0.657 0.362 0.709 0.913 0.988 0.995
19 104 0.328 0.107 0.131 0.653 0.651 0.440 0.876 0.956 0.938 0.954
20 108 0.140 0.062 0.133 0.318 0.457 0.210 0.753 0.943 0.959 0.972
21 46 0.182 0.022 0.018 0.438 0.433 0.263 0.487 0.649 0.655 0.665
22 108 0.232 0.104 0.160 0.560 0.797 0.407 0.850 0.916 0.987 0.986
23 45 0.394 0.044 0.000 0.676 0.480 0.478 0.703 0.961 0.944 0.956
24 75 0.142 0.029 0.062 0.146 0.347 0.135 0.645 0.821 0.773 0.785
25 77 0.169 0.019 0.015 0.392 0.553 0.318 0.205 0.499 0.685 0.708
26 39 0.372 0.088 0.024 0.487 0.533 0.309 0.768 0.893 0.885 0.897
27 52 0.206 0.019 0.020 0.705 0.429 0.395 0.808 0.971 0.971 0.971
28 55 0.149 0.011 0.010 0.189 0.067 0.225 0.352 0.519 0.809 0.839
29 97 0.195 0.071 0.075 0.256 0.378 0.200 0.824 0.860 0.944 0.958
30 87 0.331 0.187 0.216 0.401 0.640 0.374 0.802 0.933 0.908 0.943
31 90 0.217 0.135 0.143 0.677 0.651 0.387 0.899 0.942 0.969 1.000
32 85 0.246 0.058 0.015 0.545 0.702 0.499 0.431 0.891 0.951 0.975
33 76 0.135 0.018 0.007 0.399 0.423 0.318 0.526 0.781 0.653 0.722
34 89 0.181 0.044 0.041 0.272 0.529 0.393 0.633 0.856 0.900 0.934
35 93 0.111 0.057 0.056 0.340 0.447 0.234 0.715 0.946 0.893 0.916
36 25 0.354 0.141 0.214 0.377 0.811 0.415 0.920 0.920 0.920 0.960
37 39 0.237 0.139 0.130 0.575 0.452 0.431 0.600 0.811 0.932 0.949
38 68 0.189 0.102 0.014 0.756 0.492 0.450 0.883 0.891 0.925 0.912
39 95 0.318 0.088 0.133 0.610 0.677 0.400 0.773 0.864 0.906 0.891
40 52 0.181 0.067 0.103 0.320 0.461 0.292 0.716 0.837 0.877 0.904
41 77 0.209 0.109 0.135 0.310 0.702 0.434 0.694 0.882 0.961 0.956
42 60 0.391 0.133 0.138 0.626 0.685 0.433 0.810 0.929 0.919 0.953
43 73 0.276 0.131 0.172 0.621 0.676 0.386 0.565 0.923 0.948 0.970
44 48 0.340 0.062 0.035 0.567 0.720 0.493 0.739 0.928 0.958 0.958
45 65 0.088 0.066 0.066 0.425 0.662 0.283 0.716 0.878 0.901 0.928
46 49 0.226 0.020 0.005 0.550 0.545 0.459 0.757 0.893 0.949 0.963
47 20 0.115 0.000 0.013 0.491 0.663 0.224 0.821 0.925 0.967 0.967
48 32 0.254 0.110 0.096 0.586 0.394 0.243 0.868 0.880 0.956 0.940

download these results as csv

Task 2 Results

In this subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.

Task 2 Legend

Team ID

FH_XW = Pascal Ferraro, Pierre Hanna, Julien Allali, Matthias Robine based on XW note transcriber
CG_XW = Carlos Gómez, Soraya Abad-Mota, Edna Ruckhaus based on XW note transcriber
RJ1_RJ = J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 1 based on RJ pitch transcriber
RJ1_XW = J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 1 based on XW pitch transcriber
RJ2_RJ = J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 2 based on RJ pitch transcriber
RJ2_XW = J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 2 based on XW pitch transcriber
NM_XW = Kjell Lemström, Niko Mikkilä based on XW note transcriber
XW1_XW = Xiao Wu, Ming Li 1 based on XW note transcriber
XW2_XW = Xiao Wu, Ming Li 2 based on XW pitch transcriber

Task 2 Overall Results
Combinations CG_XW FH_XW NM_XW RJ1_RJ RJ1_XW RJ2_RJ RJ2_XW XW1_XW XW2_XW XW2_RJ
Task II (MRR) 0.715 0.452 0.618 0.345 0.305 0.536 0.379 0.917 0.937 0.883

download these results as csv

Task 2 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Friedman's ANOVA Table
Source SS df MS Chi-sq Prob>Chi-sq
Columns 3.4149e+003 9 379.4329 451.1891 0
Error 3.8056e+003 945 4.0271
Total 7.2205e+003 1059

download these results as csv

ComboID ComboID Lowerbound Mean Upperbound Significance
CG_XW NM_XW -0.3748 0.8208 2.0163 FALSE
CG_XW FH_XW 0.7054 1.9009 3.0965 TRUE
CG_XW RJ1_RJ 1.6582 2.8538 4.0493 TRUE
CG_XW RJ1_XW 1.9365 3.1321 4.3276 TRUE
CG_XW RJ2_RJ 0.4696 1.6651 2.8606 TRUE
CG_XW RJ2_XW 1.3469 2.5425 3.738 TRUE
CG_XW XW1_XW -2.7333 -1.5377 -0.3422 TRUE
CG_XW XW2_XW -2.988 -1.7925 -0.5969 TRUE
CG_XW XW2_RJ -2.5729 -1.3774 -0.1818 TRUE
NM_XW FH_XW -0.1154 1.0802 2.2757 FALSE
NM_XW RJ1_RJ 0.8375 2.033 3.2286 TRUE
NM_XW RJ1_XW 1.1158 2.3113 3.5069 TRUE
NM_XW RJ2_RJ -0.3512 0.8443 2.0399 FALSE
NM_XW RJ2_XW 0.5262 1.7217 2.9172 TRUE
NM_XW XW1_XW -3.554 -2.3585 -1.1629 TRUE
NM_XW XW2_XW -3.8087 -2.6132 -1.4177 TRUE
NM_XW XW2_RJ -3.3937 -2.1981 -1.0026 TRUE
FH_XW RJ1_RJ -0.2427 0.9528 2.1484 FALSE
FH_XW RJ1_XW 0.0356 1.2311 2.4267 TRUE
FH_XW RJ2_RJ -1.4314 -0.2358 0.9597 FALSE
FH_XW RJ2_XW -0.554 0.6415 1.8371 FALSE
FH_XW XW1_XW -4.6342 -3.4387 -2.2431 TRUE
FH_XW XW2_XW -4.8889 -3.6934 -2.4979 TRUE
FH_XW XW2_RJ -4.4738 -3.2783 -2.0828 TRUE
RJ1_RJ RJ1_XW -0.9172 0.2783 1.4738 FALSE
RJ1_RJ RJ2_RJ -2.3842 -1.1887 0.0069 FALSE
RJ1_RJ RJ2_XW -1.5069 -0.3113 0.8842 FALSE
RJ1_RJ XW1_XW -5.5871 -4.3915 -3.196 TRUE
RJ1_RJ XW2_XW -5.8418 -4.6462 -3.4507 TRUE
RJ1_RJ XW2_RJ -5.4267 -4.2311 -3.0356 TRUE
RJ1_XW RJ2_RJ -2.6625 -1.467 -0.2714 TRUE
RJ1_XW RJ2_XW -1.7852 -0.5896 0.6059 FALSE
RJ1_XW XW1_XW -5.8654 -4.6698 -3.4743 TRUE
RJ1_XW XW2_XW -6.1201 -4.9245 -3.729 TRUE
RJ1_XW XW2_RJ -5.705 -4.5094 -3.3139 TRUE
RJ2_RJ RJ2_XW -0.3182 0.8774 2.0729 FALSE
RJ2_RJ XW1_XW -4.3984 -3.2028 -2.0073 TRUE
RJ2_RJ XW2_XW -4.6531 -3.4575 -2.262 TRUE
RJ2_RJ XW2_RJ -4.238 -3.0425 -1.8469 TRUE
RJ2_XW XW1_XW -5.2757 -4.0802 -2.8846 TRUE
RJ2_XW XW2_XW -5.5304 -4.3349 -3.1394 TRUE
RJ2_XW XW2_RJ -5.1154 -3.9198 -2.7243 TRUE
XW1_XW XW2_XW -1.4503 -0.2547 0.9408 FALSE
XW1_XW XW2_RJ -1.0352 0.1604 1.3559 FALSE
XW2_XW XW2_RJ -0.7804 0.4151 1.6106 FALSE

download these results as csv 2007 qbsh07 task2 friedmans.png

Task 2 Summary Results by Query Group

Group ID Count CG_XW NM_XW FH_XW RJ1_RJ RJ1_XW RJ2_RJ RJ2_XW XW1_XW XW2_XW XW2_RJ
24 11 0.735 0.695 0.424 0.318 0.364 0.783 0.263 0.927 0.955 0.864
25 5 0.814 0.667 0.180 0.600 0.295 0.667 0.149 1.000 1.000 1.000
65 1 0.167 0.000 0.000 0.000 0.000 0.000 0.000 0.111 0.250 0.143
75 2 0.545 0.000 0.250 0.000 0.000 0.000 0.000 1.000 1.000 1.000
80 1 0.000 0.000 0.143 0.000 0.000 0.000 0.000 0.000 0.000 1.000
83 6 0.639 0.667 0.678 0.000 0.000 0.667 0.667 0.682 0.690 0.750
89 6 0.861 1.000 0.700 1.000 1.000 1.000 1.000 1.000 1.000 1.000
101 3 1.000 0.030 0.000 0.056 0.048 0.017 0.000 1.000 1.000 1.000
106 5 0.733 0.411 0.200 0.000 0.000 0.233 0.050 1.000 1.000 1.000
108 2 0.750 1.000 0.000 0.000 0.000 0.031 0.071 1.000 1.000 1.000
110 2 0.417 0.063 0.029 0.000 0.000 0.050 0.083 0.625 0.750 1.000
111 1 1.000 1.000 0.500 1.000 1.000 0.000 0.000 1.000 1.000 1.000
114 5 0.458 0.450 0.022 0.000 0.000 0.400 0.300 1.000 1.000 0.800
116 2 0.600 0.500 0.250 0.000 0.000 0.500 0.125 1.000 1.000 1.000
120 2 0.500 0.000 0.000 0.000 0.000 0.500 0.528 0.600 1.000 0.750
128 4 0.400 0.642 0.516 0.156 0.000 0.053 0.287 1.000 1.000 1.000
129 1 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.333
133 2 0.500 1.000 0.500 0.000 0.500 0.600 1.000 1.000 1.000 1.000
139 2 1.000 0.667 0.313 0.000 0.000 1.000 0.625 1.000 1.000 1.000
140 2 0.375 0.042 0.000 0.000 0.000 0.000 0.000 0.333 0.417 0.500
160 2 0.750 0.750 0.500 0.061 0.036 0.250 0.125 0.525 0.550 0.500
162 9 0.790 0.674 0.365 0.444 0.278 0.911 0.590 0.944 0.917 0.787
167 1 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
185 2 1.000 1.000 0.750 0.000 0.000 1.000 0.500 1.000 1.000 1.000
192 2 1.000 1.000 0.533 0.083 0.100 0.500 0.500 1.000 1.000 0.500
193 11 0.740 0.778 0.327 0.365 0.286 0.360 0.291 1.000 1.000 1.000
195 4 0.750 1.000 0.750 0.250 0.250 0.453 0.750 1.000 1.000 1.000
197 4 0.625 0.500 0.125 0.500 0.161 0.750 0.500 1.000 1.000 1.000
210 1 1.000 0.000 0.000 0.250 0.250 0.000 0.000 1.000 1.000 1.000
232 4 1.000 0.633 0.750 1.000 1.000 1.000 1.000 1.000 1.000 1.000
233 1 1.000 1.000 0.000 1.000 1.000 1.000 0.167 1.000 1.000 1.000
245 7 0.714 0.679 0.571 0.000 0.000 0.786 0.378 1.000 1.000 1.000
250 5 0.700 1.000 1.000 0.000 0.000 0.800 0.511 0.662 0.718 0.414
265 2 0.500 0.000 0.571 0.000 0.000 0.500 0.083 0.500 0.500 0.500
271 1 1.000 0.000 0.000 0.000 0.000 0.000 0.000 1.000 1.000 1.000
276 5 1.000 0.247 0.200 0.000 0.000 0.425 0.350 0.800 0.867 0.800
278 1 0.000 0.000 0.091 0.000 0.000 1.000 0.000 0.000 0.063 0.000
279 2 1.000 0.500 0.500 0.500 0.500 0.500 0.500 1.000 1.000 1.000
289 7 0.524 0.310 0.171 0.429 0.333 0.722 0.361 1.000 1.000 1.000
293 2 0.750 1.000 1.000 1.000 1.000 1.000 0.625 1.000 1.000 0.000
306 8 0.604 0.813 0.639 0.250 0.250 0.427 0.046 0.875 0.875 0.875
319 12 0.651 0.521 0.613 0.625 0.508 0.325 0.121 0.875 1.000 0.917
321 9 0.411 0.376 0.175 0.593 0.358 0.458 0.186 0.763 0.833 0.903
362 9 0.750 0.704 0.306 0.556 0.398 0.926 0.648 1.000 1.000 1.000
363 3 0.667 0.381 0.000 1.000 0.833 0.667 0.667 1.000 1.000 1.000
392 4 0.583 0.250 0.250 0.031 0.000 0.268 0.250 1.000 1.000 0.750
399 4 0.781 0.375 0.396 0.313 0.119 0.467 0.018 1.000 1.000 1.000
438 1 1.000 1.000 1.000 1.000 0.071 0.000 0.067 1.000 1.000 1.000
461 1 1.000 1.000 0.000 1.000 1.000 1.000 0.333 1.000 1.000 1.000
468 1 1.000 1.000 0.125 1.000 1.000 1.000 1.000 1.000 1.000 1.000
475 4 0.583 0.750 1.000 0.250 0.125 0.625 0.375 1.000 1.000 1.000
513 4 0.411 0.088 0.250 0.125 0.000 0.290 0.333 1.000 1.000 1.000
514 6 0.679 0.649 0.556 0.333 0.192 1.000 0.408 1.000 1.000 1.000
515 6 0.639 0.667 0.167 0.308 0.431 0.392 0.722 1.000 1.000 1.000
529 4 0.769 0.750 0.500 0.000 0.000 0.000 0.013 1.000 1.000 1.000
533 2 0.500 0.500 0.500 0.500 0.500 0.500 0.500 0.500 0.500 0.000
540 5 0.712 1.000 0.707 0.767 1.000 0.667 0.850 1.000 1.000 0.500
550 1 1.000 1.000 0.083 0.000 0.000 0.067 0.167 1.000 1.000 1.000
561 3 0.511 1.000 0.733 0.000 0.000 1.000 0.778 1.000 1.000 1.000
562 2 0.417 0.110 0.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
568 1 1.000 1.000 1.000 0.000 0.000 0.167 0.500 1.000 1.000 1.000
570 3 0.833 0.667 0.667 0.037 0.067 0.444 0.377 1.000 1.000 1.000
572 2 0.600 0.500 1.000 1.000 0.750 0.750 0.500 1.000 1.000 1.000
573 1 0.125 1.000 1.000 0.000 0.000 0.333 0.250 1.000 1.000 1.000
594 3 1.000 0.500 0.000 0.000 0.000 0.364 0.208 1.000 1.000 1.000
606 4 1.000 0.769 0.750 0.333 0.500 0.542 0.750 1.000 1.000 0.750
628 3 0.667 0.833 0.394 0.000 0.000 1.000 0.167 1.000 1.000 1.000
636 2 0.250 1.000 0.025 1.000 0.288 0.550 0.000 0.667 0.667 1.000
643 4 0.875 0.750 0.125 0.050 0.013 0.500 0.300 1.000 1.000 1.000
647 4 1.000 0.786 0.641 0.875 0.833 0.606 0.750 1.000 1.000 1.000
660 3 0.500 1.000 0.833 0.000 0.000 0.370 0.167 0.421 0.417 0.667
662 3 0.833 0.367 0.667 1.000 1.000 0.567 0.567 1.000 1.000 1.000
685 4 0.750 1.000 1.000 1.000 0.875 0.750 0.354 1.000 1.000 1.000
693 2 1.000 1.000 1.000 1.000 1.000 0.045 0.167 1.000 1.000 1.000
728 3 0.778 0.333 0.208 0.333 0.333 0.333 0.361 1.000 1.000 1.000
730 2 1.000 0.500 0.276 0.000 0.000 0.417 0.050 0.550 1.000 0.500
732 10 0.870 0.900 1.000 1.000 0.950 0.950 0.491 1.000 1.000 1.000
736 1 1.000 0.500 1.000 0.056 1.000 0.111 0.500 1.000 1.000 1.000
747 2 1.000 1.000 0.500 0.500 0.500 1.000 1.000 1.000 1.000 1.000
770 3 1.000 1.000 0.750 0.333 0.333 0.750 0.444 1.000 1.000 1.000
790 6 1.000 0.769 0.690 0.250 0.333 0.696 0.271 0.917 1.000 0.667
794 2 1.000 1.000 1.000 0.000 0.000 1.000 1.000 1.000 1.000 1.000
805 3 0.333 0.333 0.186 0.000 0.000 0.148 0.111 1.000 1.000 0.667
811 4 0.377 0.000 0.046 0.000 0.000 0.000 0.000 0.750 0.750 0.000
839 4 0.750 1.000 0.181 0.500 0.500 1.000 1.000 1.000 1.000 1.000
850 1 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
858 3 1.000 1.000 1.000 0.000 0.000 1.000 0.483 1.000 1.000 1.000
914 1 1.000 1.000 0.333 0.000 0.000 0.000 0.000 1.000 1.000 1.000
915 7 0.929 0.729 0.786 0.143 0.143 0.143 0.286 0.857 0.857 0.905
926 2 1.000 0.500 0.500 0.000 0.000 0.000 0.026 1.000 1.000 1.000
937 2 0.600 0.500 0.083 0.000 0.000 0.250 0.500 1.000 1.000 1.000
941 2 0.500 0.538 1.000 0.500 0.500 0.167 0.071 1.000 1.000 1.000
956 2 1.000 0.071 0.250 0.031 0.100 0.250 0.050 1.000 1.000 1.000
1000 1 0.500 0.500 1.000 0.000 0.000 0.000 1.000 1.000 1.000 1.000
1014 1 1.000 0.500 1.000 0.000 0.000 1.000 1.000 1.000 1.000 1.000
1036 1 0.000 0.000 0.000 0.056 0.071 0.000 0.000 1.000 1.000 1.000
1039 1 1.000 1.000 1.000 0.000 0.000 0.077 0.111 1.000 1.000 1.000
1044 1 1.000 1.000 0.333 0.000 0.000 0.167 0.000 1.000 1.000 1.000
1054 4 0.653 0.271 0.313 0.583 0.750 0.000 0.063 0.875 1.000 1.000
1062 3 1.000 0.750 0.356 0.333 0.333 0.108 0.111 1.000 1.000 1.000
1067 1 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
1068 1 0.250 1.000 1.000 0.000 0.000 0.000 0.000 1.000 1.000 1.000
1070 8 0.592 0.400 0.094 0.375 0.229 0.700 0.455 1.000 1.000 0.750
1082 3 0.833 0.300 0.083 0.000 0.000 0.733 0.667 1.000 1.000 1.000
1083 1 0.200 0.143 0.000 1.000 0.500 0.000 0.000 0.053 0.143 0.500
1094 3 0.500 0.061 0.000 0.000 0.000 0.047 0.048 0.694 1.000 1.000

download these results as csv