Difference between revisions of "2006:QBSH: Query-by-Singing/Humming Results"
(→Task I: Known-Item Searching Summary Results) |
Kahyun Choi (talk | contribs) (→Introduction) |
||
(35 intermediate revisions by 6 users not shown) | |||
Line 1: | Line 1: | ||
[[Category: Results]] | [[Category: Results]] | ||
==Introduction== | ==Introduction== | ||
− | These are the results for the 2006 running of the QBSH: Query-by-Singing Humming task set. For background information about this task set please refer to the [[QBSH: Query-by-Singing/Humming]] page. | + | These are the results for the 2006 running of the QBSH: Query-by-Singing Humming task set. For background information about this task set please refer to the [[2006:QBSH: Query-by-Singing/Humming]] page. |
+ | QBSH task consists of two subtasks. The first subtask is known as '''Known-Item Retrieval'''. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. The query database consists of 2797 sung queries. | ||
+ | |||
+ | The second subtask is called '''Queries as Variations'''. In this subtask, systems take an input from the query database which consists of 2797 sung queries + 48 ground truth files and return a list of 20 songs from the test database which consists of 48 ground truth MIDIs + 2000 Essen MIDI noise files + 2797 sung queries. The precision based on the number of songs within the same ground truth class of the query is calculated over the top 20 returns for each of the 2845 queries. | ||
===General Legend=== | ===General Legend=== | ||
− | ====Team ID==== | + | ====Team ID==== |
− | '''AU''' = [https://www.music-ir.org/ | + | '''AU''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_uitdenbogerd.pdf Alexandra Uitdenbogerd]<br /> |
− | ''' | + | '''CS1''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_sailer.pdf Christian Sailer-ear]<br /> |
+ | '''CS2''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_sailer.pdf Christian Sailer-midi]<br /> | ||
+ | '''CS3''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_sailer.pdf Christian Sailer-warp]<br /> | ||
'''FH''' = Pascal Ferraro and Pierre Hanna<br /> | '''FH''' = Pascal Ferraro and Pierre Hanna<br /> | ||
− | '''NM''' = [https://www.music-ir.org/ | + | '''NM''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_mikkila.pdf Kjell Lemström, Niko Mikkilä, Veli Mäkinen and Esko Ukkonen]<br /> |
− | '''RJ''' = [https://www.music-ir.org/ | + | '''RJ''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_jang.pdf Jyh-Shing Roger Jang and Nien-Jung Lee]<br /> |
− | '''RL''' = [https://www.music-ir.org/ | + | '''RL''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_tararira.pdf Ernesto López and Martín Rocamora]<br /> |
− | '''RT''' = [https://www.music-ir.org/ | + | '''RT''' = [https://www.music-ir.org/mirex/abstracts/2006/SMS_QBSH_typke.pdf Rainer Typke, Frans Wiering and Remco C. Veltkamp]<br /> |
− | '''XW''' = [https://www.music-ir.org/ | + | '''XW''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_wu.pdf Xiao Wu and Ming Li]<br /> |
===Calculating Summary Measures=== | ===Calculating Summary Measures=== | ||
Line 19: | Line 24: | ||
==Overall Summary Results== | ==Overall Summary Results== | ||
− | + | <csv>2006/qbsh_overall_sum.csv</csv> | |
+ | |||
+ | ===QBSH Task 1 Runtime Data=== | ||
+ | |||
+ | <csv>2006/qbsh06_task1_runtime.csv</csv> | ||
+ | |||
+ | ===QBSH Task 2 Runtime Data=== | ||
− | ==Friedman Test with Multiple Comparisons Results (p=0.05)== | + | <csv>2006/qbsh06_task2_runtime.csv</csv> |
− | The Friedman test was run in MATLAB against the QBSH Task 1 | + | |
+ | ==Task I: "Known-Item Searching" Friedman Test with Multiple Comparisons Results (p=0.05)== | ||
+ | The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.<br /> | ||
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); | Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); | ||
− | <csv>qush_friedman.csv</csv> | + | <csv>2006/qush_friedman.csv</csv> |
− | <csv>QBSH_sum.csv</csv> | + | <csv>2006/QBSH_sum.csv</csv> |
− | ==Task I: Known-Item Searching Summary Results== | + | ==Task I: "Known-Item Searching" Summary Results== |
MMR data summarized by the 48 ground truth song groups. | MMR data summarized by the 48 ground truth song groups. | ||
− | <csv>qbsh_task1_sum.csv</csv> | + | <csv>2006/qbsh_task1_sum.csv</csv> |
− | |||
− | |||
− | + | ==Task II: "Queries as Variations" Friedman Test with Multiple Comparisons Results (p=0.05)== | |
− | + | The Friedman test was run in MATLAB against the QBSH Task 2 mean precision data over the 48 ground truth song groups.<br /> | |
+ | Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); | ||
+ | <csv>2006/qbsh2_friedman.csv</csv> | ||
+ | <csv>2006/qbsh2_sum.csv</csv> | ||
− | <csv> | + | ==Task II: "Queries as Variations" Summary Results== |
+ | Mean precision data summarized over the 48 ground truth song groups. | ||
+ | <csv>2006/qbsh06_task2_sum.csv</csv> | ||
==Raw Scores== | ==Raw Scores== | ||
− | The raw data | + | The raw data are located on the [[2006:Query-by-Singing/Humming Raw Data]] page. |
Latest revision as of 22:41, 19 December 2011
Contents
- 1 Introduction
- 2 Overall Summary Results
- 3 Task I: "Known-Item Searching" Friedman Test with Multiple Comparisons Results (p=0.05)
- 4 Task I: "Known-Item Searching" Summary Results
- 5 Task II: "Queries as Variations" Friedman Test with Multiple Comparisons Results (p=0.05)
- 6 Task II: "Queries as Variations" Summary Results
- 7 Raw Scores
Introduction
These are the results for the 2006 running of the QBSH: Query-by-Singing Humming task set. For background information about this task set please refer to the 2006:QBSH: Query-by-Singing/Humming page. QBSH task consists of two subtasks. The first subtask is known as Known-Item Retrieval. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. The query database consists of 2797 sung queries.
The second subtask is called Queries as Variations. In this subtask, systems take an input from the query database which consists of 2797 sung queries + 48 ground truth files and return a list of 20 songs from the test database which consists of 48 ground truth MIDIs + 2000 Essen MIDI noise files + 2797 sung queries. The precision based on the number of songs within the same ground truth class of the query is calculated over the top 20 returns for each of the 2845 queries.
General Legend
Team ID
AU = Alexandra Uitdenbogerd
CS1 = Christian Sailer-ear
CS2 = Christian Sailer-midi
CS3 = Christian Sailer-warp
FH = Pascal Ferraro and Pierre Hanna
NM = Kjell Lemström, Niko Mikkilä, Veli Mäkinen and Esko Ukkonen
RJ = Jyh-Shing Roger Jang and Nien-Jung Lee
RL = Ernesto López and Martín Rocamora
RT = Rainer Typke, Frans Wiering and Remco C. Veltkamp
XW = Xiao Wu and Ming Li
Calculating Summary Measures
MRR = Mean Reciprocal Rank. Reciprocal rank is the reciprocal of the rank of the first correctly identified cover for each query (1/rank). These values are averaged for each ground truth group as well as overall.
Overall Summary Results
AU1 | AU2 | CS1 | CS2 | CS3 | FH | NM | RJ | RL | RT1 | RT2 | XW1 | XW2 | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Task I (MRR) | 0.205 | 0.288 | 0.568 | 0.283 | 0.348 | 0.218 | 0.688 | 0.883 | 0.800 | 0.196 | 0.390 | 0.926 | 0.900 |
Task II (Mean Precision) | 0.163 | 0.238 | 0.587 | 0.649 | 0.415 | 0.309 | 0.722 | 0.926 | no entry | 0.468 | 0.401 | no entry | no entry |
QBSH Task 1 Runtime Data
Team ID | Machine | Run-time(seconds) | |
---|---|---|---|
AU1 | qbsh1 | beer 2 | 166 |
AU1 | qbsh2 | beer 2 | 140 |
AU1-2 | indexing | beer 2 | 8 |
CS1 | indexing | beer 4 | 3 |
CS1 | query | beer 4 | 56560 |
CS2 | indexing | beer 3 | 3 |
CS2 | query | beer 3 | 608 |
CS3 | indexing | beer 3 | 3 |
CS3 | query | beer 3 | 4618 |
FH | all | beer 1 | 89239 |
NM | query | beer 2 | 8302 |
RJ | query | yellow | 25637 |
RL | indexing | beer 9 | 6 |
RL | query | beer 9 | 20604 |
RT | indexing | beer 4 | 23442 |
RT1 | query1 | beer 4 | 2034 |
RT2 | query2 | beer 4 | 4629 |
XW | indexing | red | 63 |
XW1 | searchingfrombeginning | red | 2502 |
XW2 | searchingfromanyposition | red | 2817 |
QBSH Task 2 Runtime Data
Team ID | Machine | Run-time(seconds) | |
---|---|---|---|
AU | indexing | beer 8 | 13 |
AU1 | query1 | beer 8 | 251 |
AU2 | query2 | beer 8 | 215 |
CS-ear | indexing | beer 0 | 4 |
CS-ear | query | beer 0 | 12201 |
CS-midi | indexing | beer 3 | 4 |
CS-midi | query | beer 3 | 781 |
CS-warp | indexing | beer 6 | 4 |
CS-warp | query | beer 6 | 4944 |
FH | all | beer 1 | 414307 |
NM | indexing/query | beer 2 | 16691 |
RJ | all | yellow | 17057 |
RT | indexing | beer 4 | 17296 |
RT1 | query1 | beer 4 | 2112 |
RT2 | query2 | beer 4 | 3738 |
Task I: "Known-Item Searching" Friedman Test with Multiple Comparisons Results (p=0.05)
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Friedman's ANOVA Table | |||||
---|---|---|---|---|---|
Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
Columns | 6731.4592 | 12 | 560.9549 | 444.9051 | 0 |
Error | 2165.0408 | 576 | 3.7588 | ||
Total | 8896.5 | 636 |
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
AU1 | AU2 | -3.8482 | -1.2449 | 1.3584 | FALSE |
AU1 | CS1 | -7.4502 | -4.8469 | -2.2436 | TRUE |
AU1 | CS2 | -3.389 | -0.7857 | 1.8176 | FALSE |
AU1 | CS3 | -5.2768 | -2.6735 | -0.0702 | TRUE |
AU1 | FH | -3.1033 | -0.5 | 2.1033 | FALSE |
AU1 | NM | -7.8992 | -5.2959 | -2.6926 | TRUE |
AU1 | RJ | -10.2462 | -7.6429 | -5.0396 | FALSE |
AU1 | RL | -9.3992 | -6.7959 | -4.1926 | TRUE |
AU1 | RT1 | -2.1339 | 0.4694 | 3.0727 | FALSE |
AU1 | RT2 | -5.2155 | -2.6122 | -0.0089 | TRUE |
AU1 | XW1 | -11.6645 | -9.0612 | -6.4579 | TRUE |
AU1 | XW2 | -10.9604 | -8.3571 | -5.7538 | TRUE |
AU2 | CS1 | -6.2053 | -3.602 | -0.9987 | TRUE |
AU2 | CS2 | -2.1441 | 0.4592 | 3.0625 | FALSE |
AU2 | CS3 | -4.0319 | -1.4286 | 1.1747 | FALSE |
AU2 | FH | -1.8584 | 0.7449 | 3.3482 | FALSE |
AU2 | NM | -6.6543 | -4.051 | -1.4477 | TRUE |
AU2 | RJ | -9.0013 | -6.398 | -3.7947 | TRUE |
AU2 | RL | -8.1543 | -5.551 | -2.9477 | TRUE |
AU2 | RT1 | -0.889 | 1.7143 | 4.3176 | FALSE |
AU2 | RT2 | -3.9707 | -1.3673 | 1.236 | FALSE |
AU2 | XW1 | -10.4196 | -7.8163 | -5.213 | TRUE |
AU2 | XW2 | -9.7155 | -7.1122 | -4.5089 | TRUE |
CS1 | CS2 | 1.4579 | 4.0612 | 6.6645 | TRUE |
CS1 | CS3 | -0.4298 | 2.1735 | 4.7768 | FALSE |
CS1 | FH | 1.7436 | 4.3469 | 6.9502 | TRUE |
CS1 | NM | -3.0523 | -0.449 | 2.1543 | FALSE |
CS1 | RJ | -5.3992 | -2.7959 | -0.1926 | TRUE |
CS1 | RL | -4.5523 | -1.949 | 0.6543 | FALSE |
CS1 | RT1 | 2.713 | 5.3163 | 7.9196 | TRUE |
CS1 | RT2 | -0.3686 | 2.2347 | 4.838 | FALSE |
CS1 | XW1 | -6.8176 | -4.2143 | -1.611 | TRUE |
CS1 | XW2 | -6.1135 | -3.5102 | -0.9069 | TRUE |
CS2 | CS3 | -4.4911 | -1.8878 | 0.7155 | FALSE |
CS2 | FH | -2.3176 | 0.2857 | 2.889 | FALSE |
CS2 | NM | -7.1135 | -4.5102 | -1.9069 | TRUE |
CS2 | RJ | -9.4604 | -6.8571 | -4.2538 | TRUE |
CS2 | RL | -8.6135 | -6.0102 | -3.4069 | TRUE |
CS2 | RT1 | -1.3482 | 1.2551 | 3.8584 | FALSE |
CS2 | RT2 | -4.4298 | -1.8265 | 0.7768 | FALSE |
CS2 | XW1 | -10.8788 | -8.2755 | -5.6722 | TRUE |
CS2 | XW2 | -10.1747 | -7.5714 | -4.9681 | TRUE |
CS3 | FH | -0.4298 | 2.1735 | 4.7768 | FALSE |
CS3 | NM | -5.2258 | -2.6224 | -0.0191 | TRUE |
CS3 | RJ | -7.5727 | -4.9694 | -2.3661 | TRUE |
CS3 | RL | -6.7258 | -4.1224 | -1.5191 | TRUE |
CS3 | RT1 | 0.5396 | 3.1429 | 5.7462 | FALSE |
CS3 | RT2 | -2.5421 | 0.0612 | 2.6645 | FALSE |
CS3 | XW1 | -8.9911 | -6.3878 | -3.7845 | TRUE |
CS3 | XW2 | -8.287 | -5.6837 | -3.0804 | TRUE |
FH | NM | -7.3992 | -4.7959 | -2.1926 | TRUE |
FH | RJ | -9.7462 | -7.1429 | -4.5396 | TRUE |
FH | RL | -8.8992 | -6.2959 | -3.6926 | TRUE |
FH | RT1 | -1.6339 | 0.9694 | 3.5727 | FALSE |
FH | RT2 | -4.7155 | -2.1122 | 0.4911 | FALSE |
FH | XW1 | -11.1645 | -8.5612 | -5.9579 | TRUE |
FH | XW2 | -10.4604 | -7.8571 | -5.2538 | TRUE |
NM | RJ | -4.9502 | -2.3469 | 0.2564 | FALSE |
NM | RL | -4.1033 | -1.5 | 1.1033 | FALSE |
NM | RT1 | 3.162 | 5.7653 | 8.3686 | TRUE |
NM | RT2 | 0.0804 | 2.6837 | 5.287 | TRUE |
NM | XW1 | -6.3686 | -3.7653 | -1.162 | TRUE |
NM | XW2 | -5.6645 | -3.0612 | -0.4579 | TRUE |
RJ | RL | -1.7564 | 0.8469 | 3.4502 | FALSE |
RJ | RT1 | 5.5089 | 8.1122 | 10.7155 | FALSE |
RJ | RT2 | 2.4273 | 5.0306 | 7.6339 | FALSE |
RJ | XW1 | -4.0217 | -1.4184 | 1.1849 | FALSE |
RJ | XW2 | -3.3176 | -0.7143 | 1.889 | FALSE |
RL | RT1 | 4.662 | 7.2653 | 9.8686 | TRUE |
RL | RT2 | 1.5804 | 4.1837 | 6.787 | TRUE |
RL | XW1 | -4.8686 | -2.2653 | 0.338 | FALSE |
RL | XW2 | -4.1645 | -1.5612 | 1.0421 | FALSE |
RT1 | RT2 | -5.6849 | -3.0816 | -0.4783 | TRUE |
RT1 | XW1 | -12.1339 | -9.5306 | -6.9273 | TRUE |
RT1 | XW2 | -11.4298 | -8.8265 | -6.2232 | TRUE |
RT2 | XW1 | -9.0523 | -6.449 | -3.8457 | TRUE |
RT2 | XW2 | -8.3482 | -5.7449 | -3.1416 | TRUE |
XW1 | XW2 | -1.8992 | 0.7041 | 3.3074 | FALSE |
Task I: "Known-Item Searching" Summary Results
MMR data summarized by the 48 ground truth song groups.
Group | AU1 | AU2 | CS1 | CS2 | CS3 | FH | NM | RJ | RL | RT1 | RT2 | XW1 | XW2 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | 0.061 | 0.080 | 0.833 | 0.333 | 0.377 | 0.109 | 0.750 | 0.889 | 0.794 | 0.176 | 0.390 | 1.000 | 1.000 |
2 | 0.170 | 0.167 | 0.477 | 0.266 | 0.418 | 0.010 | 0.583 | 0.611 | 0.611 | 0.000 | 0.356 | 0.361 | 0.361 |
3 | 0.389 | 0.507 | 0.875 | 0.500 | 0.750 | 0.572 | 0.708 | 0.875 | 0.875 | 0.375 | 0.450 | 0.775 | 0.750 |
4 | 0.502 | 0.675 | 0.926 | 0.333 | 0.778 | 0.778 | 0.944 | 1.000 | 0.833 | 0.355 | 0.679 | 1.000 | 1.000 |
5 | 0.250 | 0.333 | 0.500 | 0.148 | 0.389 | 0.356 | 0.526 | 0.509 | 0.697 | 0.222 | 0.283 | 0.895 | 0.900 |
6 | 0.258 | 0.188 | 0.938 | 0.400 | 1.000 | 0.143 | 0.688 | 0.875 | 0.519 | 0.125 | 0.221 | 1.000 | 1.000 |
7 | 0.135 | 0.244 | 1.000 | 0.370 | 1.000 | 0.387 | 0.790 | 0.889 | 0.685 | 0.111 | 0.222 | 0.926 | 0.900 |
8 | 0.119 | 0.134 | 0.889 | 0.472 | 0.815 | 0.460 | 0.815 | 0.634 | 0.444 | 0.000 | 0.236 | 1.000 | 1.000 |
9 | 0.394 | 0.324 | 0.542 | 0.125 | 0.375 | 0.251 | 0.669 | 0.750 | 0.764 | 0.313 | 0.250 | 1.000 | 1.000 |
10 | 0.202 | 0.406 | 0.750 | 0.400 | 0.900 | 0.500 | 0.827 | 0.800 | 0.900 | 0.206 | 0.556 | 1.000 | 1.000 |
11 | 0.280 | 0.288 | 0.836 | 0.662 | 0.823 | 0.419 | 0.800 | 0.922 | 0.874 | 0.319 | 0.364 | 0.935 | 0.933 |
12 | 0.265 | 0.438 | 0.941 | 0.738 | 0.765 | 0.479 | 0.737 | 0.890 | 0.791 | 0.401 | 0.460 | 0.935 | 0.923 |
13 | 0.207 | 0.105 | 0.363 | 0.032 | 0.033 | 0.286 | 0.733 | 0.898 | 0.710 | 0.057 | 0.196 | 0.972 | 0.966 |
14 | 0.344 | 0.476 | 0.748 | 0.221 | 0.291 | 0.268 | 0.735 | 0.929 | 0.906 | 0.361 | 0.597 | 0.947 | 0.908 |
15 | 0.245 | 0.213 | 0.867 | 0.252 | 0.722 | 0.102 | 0.671 | 0.819 | 0.867 | 0.022 | 0.208 | 0.867 | 0.871 |
16 | 0.311 | 0.580 | 0.941 | 0.811 | 0.895 | 0.495 | 0.829 | 0.805 | 0.902 | 0.414 | 0.618 | 0.964 | 0.949 |
17 | 0.119 | 0.118 | 0.458 | 0.159 | 0.255 | 0.025 | 0.681 | 0.832 | 0.795 | 0.018 | 0.254 | 0.931 | 0.914 |
18 | 0.282 | 0.432 | 0.793 | 0.261 | 0.314 | 0.242 | 0.754 | 0.877 | 0.838 | 0.195 | 0.664 | 0.995 | 0.984 |
19 | 0.224 | 0.501 | 0.892 | 0.399 | 0.415 | 0.123 | 0.752 | 0.947 | 0.912 | 0.239 | 0.655 | 0.960 | 0.943 |
20 | 0.164 | 0.150 | 0.782 | 0.238 | 0.288 | 0.127 | 0.609 | 0.950 | 0.821 | 0.222 | 0.253 | 0.986 | 0.977 |
21 | 0.170 | 0.184 | 0.485 | 0.056 | 0.436 | 0.105 | 0.547 | 0.668 | 0.809 | 0.205 | 0.331 | 0.626 | 0.552 |
22 | 0.218 | 0.355 | 0.859 | 0.402 | 0.459 | 0.185 | 0.820 | 0.953 | 0.899 | 0.294 | 0.370 | 0.987 | 0.987 |
23 | 0.174 | 0.340 | 0.833 | 0.738 | 0.639 | 0.151 | 0.618 | 0.978 | 0.745 | 0.183 | 0.371 | 0.956 | 0.956 |
24 | 0.134 | 0.179 | 0.581 | 0.073 | 0.289 | 0.042 | 0.493 | 0.814 | 0.731 | 0.103 | 0.213 | 0.807 | 0.787 |
25 | 0.126 | 0.188 | 0.342 | 0.142 | 0.224 | 0.139 | 0.638 | 0.573 | 0.673 | 0.022 | 0.220 | 0.674 | 0.609 |
26 | 0.334 | 0.425 | 0.804 | 0.592 | 0.380 | 0.393 | 0.603 | 0.936 | 0.885 | 0.128 | 0.479 | 0.923 | 0.910 |
27 | 0.047 | 0.198 | 0.000 | 0.000 | 0.000 | 0.126 | 0.693 | 0.962 | 0.716 | 0.252 | 0.207 | 1.000 | 0.869 |
28 | 0.130 | 0.165 | 0.595 | 0.132 | 0.197 | 0.142 | 0.136 | 0.645 | 0.809 | 0.127 | 0.221 | 0.738 | 0.614 |
29 | 0.192 | 0.182 | 0.023 | 0.000 | 0.000 | 0.211 | 0.548 | 0.870 | 0.761 | 0.076 | 0.238 | 0.914 | 0.933 |
30 | 0.252 | 0.449 | 0.820 | 0.213 | 0.465 | 0.167 | 0.760 | 0.946 | 0.916 | 0.247 | 0.468 | 0.964 | 0.924 |
31 | 0.200 | 0.342 | 0.011 | 0.000 | 0.000 | 0.184 | 0.812 | 0.969 | 0.795 | 0.294 | 0.512 | 0.989 | 0.983 |
32 | 0.169 | 0.293 | 0.830 | 0.665 | 0.664 | 0.136 | 0.863 | 0.925 | 0.751 | 0.324 | 0.493 | 0.975 | 0.982 |
33 | 0.069 | 0.159 | 0.053 | 0.000 | 0.000 | 0.106 | 0.541 | 0.845 | 0.366 | 0.365 | 0.210 | 0.742 | 0.507 |
34 | 0.172 | 0.173 | 0.137 | 0.000 | 0.000 | 0.216 | 0.675 | 0.909 | 0.924 | 0.065 | 0.572 | 0.970 | 0.916 |
35 | 0.093 | 0.168 | 0.059 | 0.000 | 0.000 | 0.123 | 0.562 | 0.971 | 0.713 | 0.091 | 0.243 | 0.923 | 0.903 |
36 | 0.237 | 0.350 | 0.772 | 0.320 | 0.428 | 0.240 | 0.888 | 0.885 | 0.900 | 0.057 | 0.353 | 0.960 | 0.960 |
37 | 0.272 | 0.123 | 0.103 | 0.000 | 0.026 | 0.143 | 0.647 | 0.870 | 0.723 | 0.060 | 0.316 | 0.897 | 0.891 |
38 | 0.235 | 0.225 | 0.773 | 0.641 | 0.471 | 0.327 | 0.511 | 0.906 | 0.922 | 0.258 | 0.536 | 0.914 | 0.893 |
39 | 0.258 | 0.275 | 0.721 | 0.566 | 0.505 | 0.222 | 0.736 | 0.868 | 0.741 | 0.223 | 0.503 | 0.905 | 0.871 |
40 | 0.257 | 0.262 | 0.681 | 0.381 | 0.305 | 0.122 | 0.587 | 0.851 | 0.869 | 0.265 | 0.507 | 0.916 | 0.880 |
41 | 0.186 | 0.380 | 0.676 | 0.267 | 0.510 | 0.159 | 0.682 | 0.904 | 0.821 | 0.076 | 0.378 | 0.987 | 0.965 |
42 | 0.297 | 0.324 | 0.283 | 0.039 | 0.033 | 0.366 | 0.893 | 0.950 | 0.799 | 0.142 | 0.309 | 0.967 | 0.964 |
43 | 0.223 | 0.346 | 0.057 | 0.000 | 0.007 | 0.333 | 0.796 | 0.961 | 0.873 | 0.176 | 0.379 | 0.962 | 0.944 |
44 | 0.208 | 0.293 | 0.943 | 0.645 | 0.771 | 0.355 | 0.851 | 0.899 | 0.883 | 0.108 | 0.435 | 0.983 | 0.984 |
45 | 0.129 | 0.154 | 0.771 | 0.384 | 0.658 | 0.161 | 0.654 | 0.861 | 0.710 | 0.104 | 0.183 | 0.959 | 0.948 |
46 | 0.157 | 0.250 | 0.722 | 0.357 | 0.521 | 0.224 | 0.598 | 0.895 | 0.728 | 0.088 | 0.257 | 0.943 | 0.947 |
47 | 0.060 | 0.077 | 0.792 | 0.268 | 0.419 | 0.162 | 0.716 | 1.000 | 0.757 | 0.171 | 0.236 | 0.967 | 0.963 |
48 | 0.195 | 0.347 | 0.000 | 0.000 | 0.000 | 0.159 | 0.559 | 0.878 | 0.908 | 0.147 | 0.384 | 0.895 | 0.948 |
Task II: "Queries as Variations" Friedman Test with Multiple Comparisons Results (p=0.05)
The Friedman test was run in MATLAB against the QBSH Task 2 mean precision data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Friedman's ANOVA Table | |||||
---|---|---|---|---|---|
Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
Columns | 3104.14 | 9 | 344.904 | 339.62 | 0 |
Error | 844.36 | 423 | 1.996 | ||
Total | 3948.5 | 479 |
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
AU1 | AU2 | -3.171118 | -1.21875 | 0.733618 | FALSE |
AU1 | CS1 | -7.650284 | -5.697917 | -3.745549 | TRUE |
AU1 | CS2 | -8.296118 | -6.34375 | -4.391382 | TRUE |
AU1 | CS3 | -5.358618 | -3.40625 | -1.453882 | TRUE |
AU1 | FH | -4.358618 | -2.40625 | -0.453882 | TRUE |
AU1 | NM | -8.733618 | -6.78125 | -4.828882 | TRUE |
AU1 | RJ | -10.639868 | -8.6875 | -6.735132 | TRUE |
AU1 | RT1 | -5.827368 | -3.875 | -1.922632 | TRUE |
AU1 | RT2 | -5.410701 | -3.458333 | -1.505966 | TRUE |
AU2 | CS1 | -6.431534 | -4.479167 | -2.526799 | TRUE |
AU2 | CS2 | -7.077368 | -5.125 | -3.172632 | TRUE |
AU2 | CS3 | -4.139868 | -2.1875 | -0.235132 | TRUE |
AU2 | FH | -3.139868 | -1.1875 | 0.764868 | FALSE |
AU2 | NM | -7.514868 | -5.5625 | -3.610132 | TRUE |
AU2 | RJ | -9.421118 | -7.46875 | -5.516382 | TRUE |
AU2 | RT1 | -4.608618 | -2.65625 | -0.703882 | TRUE |
AU2 | RT2 | -4.191951 | -2.239583 | -0.287216 | TRUE |
CS1 | CS2 | -2.598201 | -0.645833 | 1.306534 | FALSE |
CS1 | CS3 | 0.339299 | 2.291667 | 4.244034 | TRUE |
CS1 | FH | 1.339299 | 3.291667 | 5.244034 | TRUE |
CS1 | NM | -3.035701 | -1.083333 | 0.869034 | FALSE |
CS1 | RJ | -4.941951 | -2.989583 | -1.037216 | TRUE |
CS1 | RT1 | -0.129451 | 1.822917 | 3.775284 | FALSE |
CS1 | RT2 | 0.287216 | 2.239583 | 4.191951 | TRUE |
CS2 | CS3 | 0.985132 | 2.9375 | 4.889868 | TRUE |
CS2 | FH | 1.985132 | 3.9375 | 5.889868 | TRUE |
CS2 | NM | -2.389868 | -0.4375 | 1.514868 | FALSE |
CS2 | RJ | -4.296118 | -2.34375 | -0.391382 | TRUE |
CS2 | RT1 | 0.516382 | 2.46875 | 4.421118 | TRUE |
CS2 | RT2 | 0.933049 | 2.885417 | 4.837784 | TRUE |
CS3 | FH | -0.952368 | 1 | 2.952368 | FALSE |
CS3 | NM | -5.327368 | -3.375 | -1.422632 | TRUE |
CS3 | RJ | -7.233618 | -5.28125 | -3.328882 | TRUE |
CS3 | RT1 | -2.421118 | -0.46875 | 1.483618 | FALSE |
CS3 | RT2 | -2.004451 | -0.052083 | 1.900284 | FALSE |
FH | NM | -6.327368 | -4.375 | -2.422632 | TRUE |
FH | RJ | -8.233618 | -6.28125 | -4.328882 | TRUE |
FH | RT1 | -3.421118 | -1.46875 | 0.483618 | FALSE |
FH | RT2 | -3.004451 | -1.052083 | 0.900284 | FALSE |
NM | RJ | -3.858618 | -1.90625 | 0.046118 | FALSE |
NM | RT1 | 0.953882 | 2.90625 | 4.858618 | TRUE |
NM | RT2 | 1.370549 | 3.322917 | 5.275284 | TRUE |
RJ | RT1 | 2.860132 | 4.8125 | 6.764868 | TRUE |
RJ | RT2 | 3.276799 | 5.229167 | 7.181534 | TRUE |
RT1 | RT2 | -1.535701 | 0.416667 | 2.369034 | FALSE |
Task II: "Queries as Variations" Summary Results
Mean precision data summarized over the 48 ground truth song groups.
Group | AU1 | AU2 | CS1 | CS2 | CS3 | FH | NM | RJ | RT1 | RT2 |
---|---|---|---|---|---|---|---|---|---|---|
1 | 0.12 | 0.14 | 0.3 | 0.23 | 0.22 | 0.14 | 0.46 | 0.8 | 0.31 | 0.21 |
2 | 0.17 | 0.15 | 0.28 | 0.33 | 0.21 | 0.2 | 0.38 | 0.49 | 0.13 | 0.19 |
3 | 0.173 | 0.235 | 0.506 | 0.469 | 0.37 | 0.185 | 0.383 | 0.716 | 0.296 | 0.185 |
4 | 0.31 | 0.37 | 0.46 | 0.48 | 0.37 | 0.37 | 0.73 | 0.87 | 0.29 | 0.23 |
5 | 0.18 | 0.21 | 0.3 | 0.3 | 0.26 | 0.22 | 0.35 | 0.8 | 0.38 | 0.29 |
6 | 0.173 | 0.136 | 0.37 | 0.309 | 0.37 | 0.111 | 0.247 | 0.815 | 0.247 | 0.198 |
7 | 0.17 | 0.15 | 0.3 | 0.23 | 0.29 | 0.16 | 0.55 | 0.74 | 0.17 | 0.21 |
8 | 0.15 | 0.15 | 0.26 | 0.33 | 0.17 | 0.21 | 0.53 | 0.85 | 0.17 | 0.17 |
9 | 0.21 | 0.222 | 0.247 | 0.259 | 0.123 | 0.259 | 0.642 | 0.667 | 0.272 | 0.259 |
10 | 0.149 | 0.231 | 0.554 | 0.471 | 0.521 | 0.248 | 0.496 | 0.793 | 0.215 | 0.231 |
11 | 0.156 | 0.221 | 0.62 | 0.659 | 0.598 | 0.221 | 0.535 | 0.886 | 0.548 | 0.31 |
12 | 0.226 | 0.364 | 0.889 | 0.849 | 0.744 | 0.362 | 0.722 | 0.946 | 0.769 | 0.375 |
13 | 0.21 | 0.248 | 0.58 | 0.73 | 0.32 | 0.39 | 0.753 | 0.923 | 0.508 | 0.542 |
14 | 0.279 | 0.396 | 0.825 | 0.79 | 0.601 | 0.519 | 0.874 | 0.941 | 0.49 | 0.684 |
15 | 0.109 | 0.129 | 0.316 | 0.223 | 0.285 | 0.113 | 0.309 | 0.758 | 0.176 | 0.133 |
16 | 0.267 | 0.49 | 0.886 | 0.855 | 0.883 | 0.391 | 0.773 | 0.962 | 0.672 | 0.621 |
17 | 0.13 | 0.137 | 0.376 | 0.531 | 0.245 | 0.215 | 0.703 | 0.953 | 0.357 | 0.303 |
18 | 0.24 | 0.355 | 0.876 | 0.85 | 0.58 | 0.473 | 0.89 | 0.989 | 0.457 | 0.558 |
19 | 0.24 | 0.448 | 0.856 | 0.874 | 0.585 | 0.551 | 0.92 | 0.96 | 0.513 | 0.66 |
20 | 0.151 | 0.209 | 0.614 | 0.7 | 0.28 | 0.396 | 0.932 | 0.988 | 0.649 | 0.526 |
21 | 0.102 | 0.134 | 0.219 | 0.353 | 0.129 | 0.257 | 0.653 | 0.817 | 0.239 | 0.37 |
22 | 0.212 | 0.313 | 0.859 | 0.835 | 0.664 | 0.394 | 0.854 | 0.967 | 0.667 | 0.373 |
23 | 0.117 | 0.199 | 0.524 | 0.609 | 0.384 | 0.239 | 0.436 | 0.884 | 0.322 | 0.209 |
24 | 0.113 | 0.147 | 0.341 | 0.555 | 0.188 | 0.24 | 0.786 | 0.902 | 0.607 | 0.457 |
25 | 0.144 | 0.21 | 0.363 | 0.641 | 0.31 | 0.404 | 0.8 | 0.865 | 0.263 | 0.402 |
26 | 0.125 | 0.171 | 0.543 | 0.553 | 0.244 | 0.245 | 0.668 | 0.811 | 0.19 | 0.308 |
27 | 0.088 | 0.146 | 0.502 | 0.519 | 0.343 | 0.144 | 0.374 | 0.953 | 0.38 | 0.214 |
28 | 0.085 | 0.104 | 0.149 | 0.294 | 0.046 | 0.266 | 0.655 | 0.843 | 0.292 | 0.402 |
29 | 0.128 | 0.161 | 0.253 | 0.398 | 0.127 | 0.297 | 0.746 | 0.932 | 0.241 | 0.419 |
30 | 0.194 | 0.334 | 0.725 | 0.677 | 0.51 | 0.407 | 0.83 | 0.961 | 0.585 | 0.478 |
31 | 0.173 | 0.277 | 0.921 | 0.841 | 0.65 | 0.267 | 0.752 | 0.975 | 0.61 | 0.303 |
32 | 0.134 | 0.214 | 0.635 | 0.77 | 0.399 | 0.268 | 0.816 | 0.942 | 0.635 | 0.363 |
33 | 0.102 | 0.142 | 0.463 | 0.606 | 0.085 | 0.211 | 0.718 | 0.944 | 0.544 | 0.249 |
34 | 0.162 | 0.218 | 0.389 | 0.667 | 0.266 | 0.462 | 0.902 | 0.986 | 0.492 | 0.718 |
35 | 0.105 | 0.147 | 0.396 | 0.495 | 0.265 | 0.266 | 0.79 | 0.981 | 0.232 | 0.423 |
36 | 0.092 | 0.144 | 0.51 | 0.462 | 0.329 | 0.146 | 0.46 | 0.763 | 0.144 | 0.217 |
37 | 0.141 | 0.159 | 0.478 | 0.499 | 0.329 | 0.151 | 0.35 | 0.905 | 0.185 | 0.153 |
38 | 0.153 | 0.209 | 0.618 | 0.686 | 0.424 | 0.323 | 0.737 | 0.934 | 0.662 | 0.497 |
39 | 0.18 | 0.246 | 0.806 | 0.878 | 0.707 | 0.231 | 0.748 | 0.952 | 0.694 | 0.357 |
40 | 0.139 | 0.177 | 0.448 | 0.597 | 0.206 | 0.292 | 0.925 | 0.937 | 0.509 | 0.547 |
41 | 0.131 | 0.265 | 0.719 | 0.679 | 0.542 | 0.257 | 0.608 | 0.955 | 0.502 | 0.343 |
42 | 0.162 | 0.213 | 0.74 | 0.726 | 0.507 | 0.227 | 0.643 | 0.918 | 0.451 | 0.198 |
43 | 0.17 | 0.251 | 0.671 | 0.758 | 0.486 | 0.286 | 0.754 | 0.936 | 0.474 | 0.324 |
44 | 0.12 | 0.172 | 0.686 | 0.617 | 0.541 | 0.176 | 0.584 | 0.952 | 0.381 | 0.264 |
45 | 0.111 | 0.135 | 0.511 | 0.592 | 0.432 | 0.12 | 0.486 | 0.916 | 0.292 | 0.149 |
46 | 0.097 | 0.129 | 0.192 | 0.319 | 0.136 | 0.215 | 0.466 | 0.849 | 0.216 | 0.222 |
47 | 0.067 | 0.088 | 0.164 | 0.2 | 0.102 | 0.114 | 0.224 | 0.721 | 0.138 | 0.086 |
48 | 0.097 | 0.111 | 0.085 | 0.176 | 0.038 | 0.142 | 0.523 | 0.861 | 0.245 | 0.153 |
Raw Scores
The raw data are located on the 2006:Query-by-Singing/Humming Raw Data page.