Difference between revisions of "2006:QBSH: Query-by-Singing/Humming Results"

From MIREX Wiki
(Friedman Test with Multiple Comparisons Results (p=0.05))
(Introduction)
 
(39 intermediate revisions by 7 users not shown)
Line 1: Line 1:
 
[[Category: Results]]
 
[[Category: Results]]
 
==Introduction==
 
==Introduction==
These are the results for the 2006 running of the QBSH: Query-by-Singing Humming task set. For background information about this task set please refer to the [[QBSH: Query-by-Singing/Humming]] page.
+
These are the results for the 2006 running of the QBSH: Query-by-Singing Humming task set. For background information about this task set please refer to the [[2006:QBSH: Query-by-Singing/Humming]] page.
 +
QBSH task consists of two subtasks. The first subtask is known as '''Known-Item Retrieval'''. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. The query database consists of 2797 sung queries. 
 +
 
 +
The second subtask is called '''Queries as Variations'''. In this subtask, systems take an input from the query database which consists of 2797 sung queries + 48 ground truth files and return a list of 20 songs from the test database which consists of 48 ground truth MIDIs + 2000 Essen MIDI noise files + 2797 sung queries. The precision based on the number of songs within the same ground truth class of the query is calculated over the top 20 returns for each of the 2845 queries.  
  
 
===General Legend===
 
===General Legend===
====Team ID====
+
====Team ID====  
'''AU''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/QBSH_uitdenbogerd.pdf Alexandra Uitdenbogerd]<br />
+
'''AU''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_uitdenbogerd.pdf Alexandra Uitdenbogerd]<br />
'''CS''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/QBSH_sailer.pdf Christian Sailer]<br />  
+
'''CS1''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_sailer.pdf Christian Sailer-ear]<br />
 +
'''CS2''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_sailer.pdf Christian Sailer-midi]<br />
 +
'''CS3''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_sailer.pdf Christian Sailer-warp]<br />  
 
'''FH''' = Pascal Ferraro and Pierre Hanna<br />
 
'''FH''' = Pascal Ferraro and Pierre Hanna<br />
'''NM''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/QBSH_mikkila.pdf Kjell Lemström, Niko Mikkilä, Veli Mäkinen and Esko Ukkonen]<br />
+
'''NM''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_mikkila.pdf Kjell Lemström, Niko Mikkilä, Veli Mäkinen and Esko Ukkonen]<br />
'''RJ''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/QBSH_jang.pdf Jyh-Shing Roger Jang and Nien-Jung Lee]<br />
+
'''RJ''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_jang.pdf Jyh-Shing Roger Jang and Nien-Jung Lee]<br />
'''RL''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/QBSH_tararira.pdf Ernesto Lopez and Martín Rocamora]<br />
+
'''RL''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_tararira.pdf Ernesto López and Martín Rocamora]<br />
'''RT''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/SMS_QBSH_typke.pdf Rainer Typke, Frans Wiering and Remco C. Veltkamp]<br />  
+
'''RT''' = [https://www.music-ir.org/mirex/abstracts/2006/SMS_QBSH_typke.pdf Rainer Typke, Frans Wiering and Remco C. Veltkamp]<br />  
'''XW''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/QBSH_wu.pdf Xiao Wu and Ming Li]<br />
+
'''XW''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_wu.pdf Xiao Wu and Ming Li]<br />
  
 
===Calculating Summary Measures===
 
===Calculating Summary Measures===
Line 19: Line 24:
 
==Overall Summary Results==
 
==Overall Summary Results==
  
INSERT CSVs for overall scores here.
+
<csv>2006/qbsh_overall_sum.csv</csv>
  
==Friedman Test with Multiple Comparisons Results (p=0.05)==
+
===QBSH Task 1 Runtime Data===
The Friedman test was run in MATLAB against the QBSH data over the 48 ground truth song groups.<br />
+
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
+
<csv>2006/qbsh06_task1_runtime.csv</csv>
<csv>qush_friedman.csv</csv>
+
 
<csv>QBSH_sum.csv</csv>
+
===QBSH Task 2 Runtime Data===
  
==Task I: Known-Item Searching Summary Results==
+
<csv>2006/qbsh06_task2_runtime.csv</csv>
  
<csv>qbsh06_mrr_task1_sum.csv</csv>
+
==Task I: "Known-Item Searching" Friedman Test with Multiple Comparisons Results (p=0.05)==
 +
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.<br />
 +
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 +
<csv>2006/qush_friedman.csv</csv>
 +
<csv>2006/QBSH_sum.csv</csv>
  
==Task IIa: Karaoke Collection Summary Results==
+
==Task I: "Known-Item Searching" Summary Results==
 +
MMR data summarized by the 48 ground truth song groups.
 +
<csv>2006/qbsh_task1_sum.csv</csv>
  
<csv>sms06_kar_results2.csv</csv>
+
==Task II: "Queries as Variations" Friedman Test with Multiple Comparisons Results (p=0.05)==
  
==Task IIb: Mixed Polyphonic Collection Summary Results==
+
The Friedman test was run in MATLAB against the QBSH Task 2 mean precision data over the 48 ground truth song groups.<br />
 +
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 +
<csv>2006/qbsh2_friedman.csv</csv>
 +
<csv>2006/qbsh2_sum.csv</csv>
  
<csv>sms06_mix_results2.csv</csv>
+
==Task II: "Queries as Variations" Summary Results==
 +
Mean precision data summarized over the 48 ground truth song groups.
 +
<csv>2006/qbsh06_task2_sum.csv</csv>
  
 
==Raw Scores==
 
==Raw Scores==
The raw data derived from the Evalutron 6000 human evaluations are located on the [[Query-by-Singing/Humming Raw Data]] page.
+
The raw data are located on the [[2006:Query-by-Singing/Humming Raw Data]] page.

Latest revision as of 22:41, 19 December 2011

Introduction

These are the results for the 2006 running of the QBSH: Query-by-Singing Humming task set. For background information about this task set please refer to the 2006:QBSH: Query-by-Singing/Humming page. QBSH task consists of two subtasks. The first subtask is known as Known-Item Retrieval. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. The query database consists of 2797 sung queries.

The second subtask is called Queries as Variations. In this subtask, systems take an input from the query database which consists of 2797 sung queries + 48 ground truth files and return a list of 20 songs from the test database which consists of 48 ground truth MIDIs + 2000 Essen MIDI noise files + 2797 sung queries. The precision based on the number of songs within the same ground truth class of the query is calculated over the top 20 returns for each of the 2845 queries.

General Legend

Team ID

AU = Alexandra Uitdenbogerd
CS1 = Christian Sailer-ear
CS2 = Christian Sailer-midi
CS3 = Christian Sailer-warp
FH = Pascal Ferraro and Pierre Hanna
NM = Kjell Lemström, Niko Mikkilä, Veli Mäkinen and Esko Ukkonen
RJ = Jyh-Shing Roger Jang and Nien-Jung Lee
RL = Ernesto López and Martín Rocamora
RT = Rainer Typke, Frans Wiering and Remco C. Veltkamp
XW = Xiao Wu and Ming Li

Calculating Summary Measures

MRR = Mean Reciprocal Rank. Reciprocal rank is the reciprocal of the rank of the first correctly identified cover for each query (1/rank). These values are averaged for each ground truth group as well as overall.

Overall Summary Results

AU1 AU2 CS1 CS2 CS3 FH NM RJ RL RT1 RT2 XW1 XW2
Task I (MRR) 0.205 0.288 0.568 0.283 0.348 0.218 0.688 0.883 0.800 0.196 0.390 0.926 0.900
Task II (Mean Precision) 0.163 0.238 0.587 0.649 0.415 0.309 0.722 0.926 no entry 0.468 0.401 no entry no entry

download these results as csv

QBSH Task 1 Runtime Data

Team ID Machine Run-time(seconds)
AU1 qbsh1 beer 2 166
AU1 qbsh2 beer 2 140
AU1-2 indexing beer 2 8
CS1 indexing beer 4 3
CS1 query beer 4 56560
CS2 indexing beer 3 3
CS2 query beer 3 608
CS3 indexing beer 3 3
CS3 query beer 3 4618
FH all beer 1 89239
NM query beer 2 8302
RJ query yellow 25637
RL indexing beer 9 6
RL query beer 9 20604
RT indexing beer 4 23442
RT1 query1 beer 4 2034
RT2 query2 beer 4 4629
XW indexing red 63
XW1 searchingfrombeginning red 2502
XW2 searchingfromanyposition red 2817

download these results as csv

QBSH Task 2 Runtime Data

Team ID Machine Run-time(seconds)
AU indexing beer 8 13
AU1 query1 beer 8 251
AU2 query2 beer 8 215
CS-ear indexing beer 0 4
CS-ear query beer 0 12201
CS-midi indexing beer 3 4
CS-midi query beer 3 781
CS-warp indexing beer 6 4
CS-warp query beer 6 4944
FH all beer 1 414307
NM indexing/query beer 2 16691
RJ all yellow 17057
RT indexing beer 4 17296
RT1 query1 beer 4 2112
RT2 query2 beer 4 3738

download these results as csv

Task I: "Known-Item Searching" Friedman Test with Multiple Comparisons Results (p=0.05)

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Friedman's ANOVA Table
Source SS df MS Chi-sq Prob>Chi-sq
Columns 6731.4592 12 560.9549 444.9051 0
Error 2165.0408 576 3.7588
Total 8896.5 636

download these results as csv

TeamID TeamID Lowerbound Mean Upperbound Significance
AU1 AU2 -3.8482 -1.2449 1.3584 FALSE
AU1 CS1 -7.4502 -4.8469 -2.2436 TRUE
AU1 CS2 -3.389 -0.7857 1.8176 FALSE
AU1 CS3 -5.2768 -2.6735 -0.0702 TRUE
AU1 FH -3.1033 -0.5 2.1033 FALSE
AU1 NM -7.8992 -5.2959 -2.6926 TRUE
AU1 RJ -10.2462 -7.6429 -5.0396 FALSE
AU1 RL -9.3992 -6.7959 -4.1926 TRUE
AU1 RT1 -2.1339 0.4694 3.0727 FALSE
AU1 RT2 -5.2155 -2.6122 -0.0089 TRUE
AU1 XW1 -11.6645 -9.0612 -6.4579 TRUE
AU1 XW2 -10.9604 -8.3571 -5.7538 TRUE
AU2 CS1 -6.2053 -3.602 -0.9987 TRUE
AU2 CS2 -2.1441 0.4592 3.0625 FALSE
AU2 CS3 -4.0319 -1.4286 1.1747 FALSE
AU2 FH -1.8584 0.7449 3.3482 FALSE
AU2 NM -6.6543 -4.051 -1.4477 TRUE
AU2 RJ -9.0013 -6.398 -3.7947 TRUE
AU2 RL -8.1543 -5.551 -2.9477 TRUE
AU2 RT1 -0.889 1.7143 4.3176 FALSE
AU2 RT2 -3.9707 -1.3673 1.236 FALSE
AU2 XW1 -10.4196 -7.8163 -5.213 TRUE
AU2 XW2 -9.7155 -7.1122 -4.5089 TRUE
CS1 CS2 1.4579 4.0612 6.6645 TRUE
CS1 CS3 -0.4298 2.1735 4.7768 FALSE
CS1 FH 1.7436 4.3469 6.9502 TRUE
CS1 NM -3.0523 -0.449 2.1543 FALSE
CS1 RJ -5.3992 -2.7959 -0.1926 TRUE
CS1 RL -4.5523 -1.949 0.6543 FALSE
CS1 RT1 2.713 5.3163 7.9196 TRUE
CS1 RT2 -0.3686 2.2347 4.838 FALSE
CS1 XW1 -6.8176 -4.2143 -1.611 TRUE
CS1 XW2 -6.1135 -3.5102 -0.9069 TRUE
CS2 CS3 -4.4911 -1.8878 0.7155 FALSE
CS2 FH -2.3176 0.2857 2.889 FALSE
CS2 NM -7.1135 -4.5102 -1.9069 TRUE
CS2 RJ -9.4604 -6.8571 -4.2538 TRUE
CS2 RL -8.6135 -6.0102 -3.4069 TRUE
CS2 RT1 -1.3482 1.2551 3.8584 FALSE
CS2 RT2 -4.4298 -1.8265 0.7768 FALSE
CS2 XW1 -10.8788 -8.2755 -5.6722 TRUE
CS2 XW2 -10.1747 -7.5714 -4.9681 TRUE
CS3 FH -0.4298 2.1735 4.7768 FALSE
CS3 NM -5.2258 -2.6224 -0.0191 TRUE
CS3 RJ -7.5727 -4.9694 -2.3661 TRUE
CS3 RL -6.7258 -4.1224 -1.5191 TRUE
CS3 RT1 0.5396 3.1429 5.7462 FALSE
CS3 RT2 -2.5421 0.0612 2.6645 FALSE
CS3 XW1 -8.9911 -6.3878 -3.7845 TRUE
CS3 XW2 -8.287 -5.6837 -3.0804 TRUE
FH NM -7.3992 -4.7959 -2.1926 TRUE
FH RJ -9.7462 -7.1429 -4.5396 TRUE
FH RL -8.8992 -6.2959 -3.6926 TRUE
FH RT1 -1.6339 0.9694 3.5727 FALSE
FH RT2 -4.7155 -2.1122 0.4911 FALSE
FH XW1 -11.1645 -8.5612 -5.9579 TRUE
FH XW2 -10.4604 -7.8571 -5.2538 TRUE
NM RJ -4.9502 -2.3469 0.2564 FALSE
NM RL -4.1033 -1.5 1.1033 FALSE
NM RT1 3.162 5.7653 8.3686 TRUE
NM RT2 0.0804 2.6837 5.287 TRUE
NM XW1 -6.3686 -3.7653 -1.162 TRUE
NM XW2 -5.6645 -3.0612 -0.4579 TRUE
RJ RL -1.7564 0.8469 3.4502 FALSE
RJ RT1 5.5089 8.1122 10.7155 FALSE
RJ RT2 2.4273 5.0306 7.6339 FALSE
RJ XW1 -4.0217 -1.4184 1.1849 FALSE
RJ XW2 -3.3176 -0.7143 1.889 FALSE
RL RT1 4.662 7.2653 9.8686 TRUE
RL RT2 1.5804 4.1837 6.787 TRUE
RL XW1 -4.8686 -2.2653 0.338 FALSE
RL XW2 -4.1645 -1.5612 1.0421 FALSE
RT1 RT2 -5.6849 -3.0816 -0.4783 TRUE
RT1 XW1 -12.1339 -9.5306 -6.9273 TRUE
RT1 XW2 -11.4298 -8.8265 -6.2232 TRUE
RT2 XW1 -9.0523 -6.449 -3.8457 TRUE
RT2 XW2 -8.3482 -5.7449 -3.1416 TRUE
XW1 XW2 -1.8992 0.7041 3.3074 FALSE

download these results as csv

Task I: "Known-Item Searching" Summary Results

MMR data summarized by the 48 ground truth song groups.

Group AU1 AU2 CS1 CS2 CS3 FH NM RJ RL RT1 RT2 XW1 XW2
1 0.061 0.080 0.833 0.333 0.377 0.109 0.750 0.889 0.794 0.176 0.390 1.000 1.000
2 0.170 0.167 0.477 0.266 0.418 0.010 0.583 0.611 0.611 0.000 0.356 0.361 0.361
3 0.389 0.507 0.875 0.500 0.750 0.572 0.708 0.875 0.875 0.375 0.450 0.775 0.750
4 0.502 0.675 0.926 0.333 0.778 0.778 0.944 1.000 0.833 0.355 0.679 1.000 1.000
5 0.250 0.333 0.500 0.148 0.389 0.356 0.526 0.509 0.697 0.222 0.283 0.895 0.900
6 0.258 0.188 0.938 0.400 1.000 0.143 0.688 0.875 0.519 0.125 0.221 1.000 1.000
7 0.135 0.244 1.000 0.370 1.000 0.387 0.790 0.889 0.685 0.111 0.222 0.926 0.900
8 0.119 0.134 0.889 0.472 0.815 0.460 0.815 0.634 0.444 0.000 0.236 1.000 1.000
9 0.394 0.324 0.542 0.125 0.375 0.251 0.669 0.750 0.764 0.313 0.250 1.000 1.000
10 0.202 0.406 0.750 0.400 0.900 0.500 0.827 0.800 0.900 0.206 0.556 1.000 1.000
11 0.280 0.288 0.836 0.662 0.823 0.419 0.800 0.922 0.874 0.319 0.364 0.935 0.933
12 0.265 0.438 0.941 0.738 0.765 0.479 0.737 0.890 0.791 0.401 0.460 0.935 0.923
13 0.207 0.105 0.363 0.032 0.033 0.286 0.733 0.898 0.710 0.057 0.196 0.972 0.966
14 0.344 0.476 0.748 0.221 0.291 0.268 0.735 0.929 0.906 0.361 0.597 0.947 0.908
15 0.245 0.213 0.867 0.252 0.722 0.102 0.671 0.819 0.867 0.022 0.208 0.867 0.871
16 0.311 0.580 0.941 0.811 0.895 0.495 0.829 0.805 0.902 0.414 0.618 0.964 0.949
17 0.119 0.118 0.458 0.159 0.255 0.025 0.681 0.832 0.795 0.018 0.254 0.931 0.914
18 0.282 0.432 0.793 0.261 0.314 0.242 0.754 0.877 0.838 0.195 0.664 0.995 0.984
19 0.224 0.501 0.892 0.399 0.415 0.123 0.752 0.947 0.912 0.239 0.655 0.960 0.943
20 0.164 0.150 0.782 0.238 0.288 0.127 0.609 0.950 0.821 0.222 0.253 0.986 0.977
21 0.170 0.184 0.485 0.056 0.436 0.105 0.547 0.668 0.809 0.205 0.331 0.626 0.552
22 0.218 0.355 0.859 0.402 0.459 0.185 0.820 0.953 0.899 0.294 0.370 0.987 0.987
23 0.174 0.340 0.833 0.738 0.639 0.151 0.618 0.978 0.745 0.183 0.371 0.956 0.956
24 0.134 0.179 0.581 0.073 0.289 0.042 0.493 0.814 0.731 0.103 0.213 0.807 0.787
25 0.126 0.188 0.342 0.142 0.224 0.139 0.638 0.573 0.673 0.022 0.220 0.674 0.609
26 0.334 0.425 0.804 0.592 0.380 0.393 0.603 0.936 0.885 0.128 0.479 0.923 0.910
27 0.047 0.198 0.000 0.000 0.000 0.126 0.693 0.962 0.716 0.252 0.207 1.000 0.869
28 0.130 0.165 0.595 0.132 0.197 0.142 0.136 0.645 0.809 0.127 0.221 0.738 0.614
29 0.192 0.182 0.023 0.000 0.000 0.211 0.548 0.870 0.761 0.076 0.238 0.914 0.933
30 0.252 0.449 0.820 0.213 0.465 0.167 0.760 0.946 0.916 0.247 0.468 0.964 0.924
31 0.200 0.342 0.011 0.000 0.000 0.184 0.812 0.969 0.795 0.294 0.512 0.989 0.983
32 0.169 0.293 0.830 0.665 0.664 0.136 0.863 0.925 0.751 0.324 0.493 0.975 0.982
33 0.069 0.159 0.053 0.000 0.000 0.106 0.541 0.845 0.366 0.365 0.210 0.742 0.507
34 0.172 0.173 0.137 0.000 0.000 0.216 0.675 0.909 0.924 0.065 0.572 0.970 0.916
35 0.093 0.168 0.059 0.000 0.000 0.123 0.562 0.971 0.713 0.091 0.243 0.923 0.903
36 0.237 0.350 0.772 0.320 0.428 0.240 0.888 0.885 0.900 0.057 0.353 0.960 0.960
37 0.272 0.123 0.103 0.000 0.026 0.143 0.647 0.870 0.723 0.060 0.316 0.897 0.891
38 0.235 0.225 0.773 0.641 0.471 0.327 0.511 0.906 0.922 0.258 0.536 0.914 0.893
39 0.258 0.275 0.721 0.566 0.505 0.222 0.736 0.868 0.741 0.223 0.503 0.905 0.871
40 0.257 0.262 0.681 0.381 0.305 0.122 0.587 0.851 0.869 0.265 0.507 0.916 0.880
41 0.186 0.380 0.676 0.267 0.510 0.159 0.682 0.904 0.821 0.076 0.378 0.987 0.965
42 0.297 0.324 0.283 0.039 0.033 0.366 0.893 0.950 0.799 0.142 0.309 0.967 0.964
43 0.223 0.346 0.057 0.000 0.007 0.333 0.796 0.961 0.873 0.176 0.379 0.962 0.944
44 0.208 0.293 0.943 0.645 0.771 0.355 0.851 0.899 0.883 0.108 0.435 0.983 0.984
45 0.129 0.154 0.771 0.384 0.658 0.161 0.654 0.861 0.710 0.104 0.183 0.959 0.948
46 0.157 0.250 0.722 0.357 0.521 0.224 0.598 0.895 0.728 0.088 0.257 0.943 0.947
47 0.060 0.077 0.792 0.268 0.419 0.162 0.716 1.000 0.757 0.171 0.236 0.967 0.963
48 0.195 0.347 0.000 0.000 0.000 0.159 0.559 0.878 0.908 0.147 0.384 0.895 0.948

download these results as csv

Task II: "Queries as Variations" Friedman Test with Multiple Comparisons Results (p=0.05)

The Friedman test was run in MATLAB against the QBSH Task 2 mean precision data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Friedman's ANOVA Table
Source SS df MS Chi-sq Prob>Chi-sq
Columns 3104.14 9 344.904 339.62 0
Error 844.36 423 1.996
Total 3948.5 479

download these results as csv

TeamID TeamID Lowerbound Mean Upperbound Significance
AU1 AU2 -3.171118 -1.21875 0.733618 FALSE
AU1 CS1 -7.650284 -5.697917 -3.745549 TRUE
AU1 CS2 -8.296118 -6.34375 -4.391382 TRUE
AU1 CS3 -5.358618 -3.40625 -1.453882 TRUE
AU1 FH -4.358618 -2.40625 -0.453882 TRUE
AU1 NM -8.733618 -6.78125 -4.828882 TRUE
AU1 RJ -10.639868 -8.6875 -6.735132 TRUE
AU1 RT1 -5.827368 -3.875 -1.922632 TRUE
AU1 RT2 -5.410701 -3.458333 -1.505966 TRUE
AU2 CS1 -6.431534 -4.479167 -2.526799 TRUE
AU2 CS2 -7.077368 -5.125 -3.172632 TRUE
AU2 CS3 -4.139868 -2.1875 -0.235132 TRUE
AU2 FH -3.139868 -1.1875 0.764868 FALSE
AU2 NM -7.514868 -5.5625 -3.610132 TRUE
AU2 RJ -9.421118 -7.46875 -5.516382 TRUE
AU2 RT1 -4.608618 -2.65625 -0.703882 TRUE
AU2 RT2 -4.191951 -2.239583 -0.287216 TRUE
CS1 CS2 -2.598201 -0.645833 1.306534 FALSE
CS1 CS3 0.339299 2.291667 4.244034 TRUE
CS1 FH 1.339299 3.291667 5.244034 TRUE
CS1 NM -3.035701 -1.083333 0.869034 FALSE
CS1 RJ -4.941951 -2.989583 -1.037216 TRUE
CS1 RT1 -0.129451 1.822917 3.775284 FALSE
CS1 RT2 0.287216 2.239583 4.191951 TRUE
CS2 CS3 0.985132 2.9375 4.889868 TRUE
CS2 FH 1.985132 3.9375 5.889868 TRUE
CS2 NM -2.389868 -0.4375 1.514868 FALSE
CS2 RJ -4.296118 -2.34375 -0.391382 TRUE
CS2 RT1 0.516382 2.46875 4.421118 TRUE
CS2 RT2 0.933049 2.885417 4.837784 TRUE
CS3 FH -0.952368 1 2.952368 FALSE
CS3 NM -5.327368 -3.375 -1.422632 TRUE
CS3 RJ -7.233618 -5.28125 -3.328882 TRUE
CS3 RT1 -2.421118 -0.46875 1.483618 FALSE
CS3 RT2 -2.004451 -0.052083 1.900284 FALSE
FH NM -6.327368 -4.375 -2.422632 TRUE
FH RJ -8.233618 -6.28125 -4.328882 TRUE
FH RT1 -3.421118 -1.46875 0.483618 FALSE
FH RT2 -3.004451 -1.052083 0.900284 FALSE
NM RJ -3.858618 -1.90625 0.046118 FALSE
NM RT1 0.953882 2.90625 4.858618 TRUE
NM RT2 1.370549 3.322917 5.275284 TRUE
RJ RT1 2.860132 4.8125 6.764868 TRUE
RJ RT2 3.276799 5.229167 7.181534 TRUE
RT1 RT2 -1.535701 0.416667 2.369034 FALSE

download these results as csv

Task II: "Queries as Variations" Summary Results

Mean precision data summarized over the 48 ground truth song groups.

Group AU1 AU2 CS1 CS2 CS3 FH NM RJ RT1 RT2
1 0.12 0.14 0.3 0.23 0.22 0.14 0.46 0.8 0.31 0.21
2 0.17 0.15 0.28 0.33 0.21 0.2 0.38 0.49 0.13 0.19
3 0.173 0.235 0.506 0.469 0.37 0.185 0.383 0.716 0.296 0.185
4 0.31 0.37 0.46 0.48 0.37 0.37 0.73 0.87 0.29 0.23
5 0.18 0.21 0.3 0.3 0.26 0.22 0.35 0.8 0.38 0.29
6 0.173 0.136 0.37 0.309 0.37 0.111 0.247 0.815 0.247 0.198
7 0.17 0.15 0.3 0.23 0.29 0.16 0.55 0.74 0.17 0.21
8 0.15 0.15 0.26 0.33 0.17 0.21 0.53 0.85 0.17 0.17
9 0.21 0.222 0.247 0.259 0.123 0.259 0.642 0.667 0.272 0.259
10 0.149 0.231 0.554 0.471 0.521 0.248 0.496 0.793 0.215 0.231
11 0.156 0.221 0.62 0.659 0.598 0.221 0.535 0.886 0.548 0.31
12 0.226 0.364 0.889 0.849 0.744 0.362 0.722 0.946 0.769 0.375
13 0.21 0.248 0.58 0.73 0.32 0.39 0.753 0.923 0.508 0.542
14 0.279 0.396 0.825 0.79 0.601 0.519 0.874 0.941 0.49 0.684
15 0.109 0.129 0.316 0.223 0.285 0.113 0.309 0.758 0.176 0.133
16 0.267 0.49 0.886 0.855 0.883 0.391 0.773 0.962 0.672 0.621
17 0.13 0.137 0.376 0.531 0.245 0.215 0.703 0.953 0.357 0.303
18 0.24 0.355 0.876 0.85 0.58 0.473 0.89 0.989 0.457 0.558
19 0.24 0.448 0.856 0.874 0.585 0.551 0.92 0.96 0.513 0.66
20 0.151 0.209 0.614 0.7 0.28 0.396 0.932 0.988 0.649 0.526
21 0.102 0.134 0.219 0.353 0.129 0.257 0.653 0.817 0.239 0.37
22 0.212 0.313 0.859 0.835 0.664 0.394 0.854 0.967 0.667 0.373
23 0.117 0.199 0.524 0.609 0.384 0.239 0.436 0.884 0.322 0.209
24 0.113 0.147 0.341 0.555 0.188 0.24 0.786 0.902 0.607 0.457
25 0.144 0.21 0.363 0.641 0.31 0.404 0.8 0.865 0.263 0.402
26 0.125 0.171 0.543 0.553 0.244 0.245 0.668 0.811 0.19 0.308
27 0.088 0.146 0.502 0.519 0.343 0.144 0.374 0.953 0.38 0.214
28 0.085 0.104 0.149 0.294 0.046 0.266 0.655 0.843 0.292 0.402
29 0.128 0.161 0.253 0.398 0.127 0.297 0.746 0.932 0.241 0.419
30 0.194 0.334 0.725 0.677 0.51 0.407 0.83 0.961 0.585 0.478
31 0.173 0.277 0.921 0.841 0.65 0.267 0.752 0.975 0.61 0.303
32 0.134 0.214 0.635 0.77 0.399 0.268 0.816 0.942 0.635 0.363
33 0.102 0.142 0.463 0.606 0.085 0.211 0.718 0.944 0.544 0.249
34 0.162 0.218 0.389 0.667 0.266 0.462 0.902 0.986 0.492 0.718
35 0.105 0.147 0.396 0.495 0.265 0.266 0.79 0.981 0.232 0.423
36 0.092 0.144 0.51 0.462 0.329 0.146 0.46 0.763 0.144 0.217
37 0.141 0.159 0.478 0.499 0.329 0.151 0.35 0.905 0.185 0.153
38 0.153 0.209 0.618 0.686 0.424 0.323 0.737 0.934 0.662 0.497
39 0.18 0.246 0.806 0.878 0.707 0.231 0.748 0.952 0.694 0.357
40 0.139 0.177 0.448 0.597 0.206 0.292 0.925 0.937 0.509 0.547
41 0.131 0.265 0.719 0.679 0.542 0.257 0.608 0.955 0.502 0.343
42 0.162 0.213 0.74 0.726 0.507 0.227 0.643 0.918 0.451 0.198
43 0.17 0.251 0.671 0.758 0.486 0.286 0.754 0.936 0.474 0.324
44 0.12 0.172 0.686 0.617 0.541 0.176 0.584 0.952 0.381 0.264
45 0.111 0.135 0.511 0.592 0.432 0.12 0.486 0.916 0.292 0.149
46 0.097 0.129 0.192 0.319 0.136 0.215 0.466 0.849 0.216 0.222
47 0.067 0.088 0.164 0.2 0.102 0.114 0.224 0.721 0.138 0.086
48 0.097 0.111 0.085 0.176 0.038 0.142 0.523 0.861 0.245 0.153

download these results as csv

Raw Scores

The raw data are located on the 2006:Query-by-Singing/Humming Raw Data page.