2011:Query-by-Tapping Results
Contents
Introduction
These are the results for the 2011 running of the Query-by-tappingn task. For background information about this task set please refer to the 2011:Query_by_Tapping page.
Task Descriptions
Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:
- Jang's Dataset Roger Jang's MIR-QBT: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries.
- Hsiao's Dataset Show Hsiao's QBT_symbolic: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries.
Task 2 Goto Task 2 Results:The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.
General Legend
Team ID
| Sub code | Submission name | Abstract | Contributors |
|---|---|---|---|
| CJ1 | QBT | ChunTa Chen, Roger Jang |
Task 1 Results
Task 1a, Jang's Dataset Results
Task 1a Overall Results
| CJ1 | |
|---|---|
| Simple Count | 0.747 |
| MRR | 0.599 |
| Total Count | 890 |
Task 1a Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Simple Hit/Miss Count: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask1aSimpleByGroup.friedman.tukeyKramerHSD.csv not found
MRR Method: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask1aMrrByGroup.friedman.tukeyKramerHSD.csv not found
Task 1a Summary Results by Query Group
Simple Hit/Miss Counting file /nema-raid/www/mirex/results/2011/qbt/QbtTask1aSimpleByGroup.csv not found
MRR Method file /nema-raid/www/mirex/results/2011/qbt/QbtTask1aMrrByGroup.csv not found
Task 1a Summary Results by Query
Simple Hit/Miss Counting [1]
MRR Method [2]
Task 1b, Hsiao's Dataset Results
Task 1b Overall Results
| CJ1 | |
|---|---|
| 1 | 0.50 |
| 2 | NaN |
| 3 | 0.50 |
| 4 | 0.42 |
| 5 | 1.00 |
| 6 | 0.50 |
| 7 | 1.00 |
| 8 | 1.00 |
| 9 | 1.00 |
| 10 | 0.33 |
| 11 | 1.00 |
| 12 | 0.12 |
| 13 | 0.75 |
| 14 | 0.00 |
| 15 | NaN |
| 16 | 1.00 |
| 17 | 1.00 |
| 18 | 0.33 |
| 19 | 0.84 |
| 20 | 0.33 |
| 21 | 0.58 |
| 22 | 1.00 |
| 23 | 0.71 |
| 24 | 0.33 |
| 25 | 0.00 |
| 26 | 0.00 |
| 27 | 0.27 |
| 28 | NaN |
| 29 | 1.00 |
| 30 | 1.00 |
| 31 | NaN |
| 32 | 1.00 |
| 33 | 0.25 |
| 34 | 0.21 |
| 35 | NaN |
| 36 | 0.00 |
| 37 | 0.00 |
| 38 | 0.03 |
| 39 | 0.50 |
| 40 | 0.00 |
| 41 | 0.33 |
| 42 | 0.50 |
| 43 | 0.48 |
| 44 | 0.00 |
| 45 | 0.04 |
| 46 | 0.14 |
| 47 | 0.33 |
| 48 | 1.00 |
| 49 | 0.20 |
| 50 | 1.00 |
| 51 | 0.00 |
| 52 | 0.67 |
| 53 | 0.69 |
| 54 | 0.08 |
| 55 | 0.00 |
| 56 | 0.25 |
| 57 | 0.42 |
| 58 | NaN |
| 59 | 1.00 |
| 60 | 0.32 |
| 61 | 0.27 |
| 62 | 0.00 |
| 63 | 0.67 |
| 64 | NaN |
| 65 | 0.42 |
| 66 | NaN |
| 67 | 0.00 |
| 68 | NaN |
| 69 | 0.04 |
| 70 | 0.00 |
| 71 | NaN |
| 72 | 1.00 |
| 73 | 0.50 |
| 74 | NaN |
| 75 | 1.00 |
| 76 | 0.05 |
| 77 | NaN |
| 78 | 0.67 |
| 79 | 0.03 |
| 80 | 0.00 |
| 81 | 0.00 |
| 82 | 0.00 |
| 83 | 1.00 |
| 84 | 0.56 |
| 85 | 0.00 |
| 86 | 1.00 |
| 87 | 1.00 |
| 88 | 0.40 |
| 89 | 0.21 |
| 90 | 0.17 |
| 91 | 0.75 |
| 92 | 1.00 |
| 93 | 0.10 |
| 94 | 0.00 |
| 95 | 0.75 |
| 96 | 1.00 |
| 97 | 0.00 |
| 98 | 0.67 |
| 99 | 0.65 |
| 100 | 0.62 |
| 101 | 0.90 |
| 102 | 1.00 |
| 103 | NaN |
| 104 | 1.00 |
| 105 | 0.71 |
| 106 | 0.00 |
| 107 | NaN |
| 108 | 1.00 |
| 109 | 0.25 |
| 110 | 0.00 |
| 111 | 0.20 |
| 112 | 0.17 |
| 113 | 1.00 |
| 114 | 0.25 |
| 115 | NaN |
| 116 | 0.00 |
| 117 | 1.00 |
| 118 | 0.83 |
| 119 | 0.00 |
| 120 | 0.17 |
| 121 | 0.19 |
| 122 | 0.38 |
| 123 | 0.33 |
| 124 | 0.90 |
| 125 | 0.00 |
| 126 | 1.00 |
| 127 | 0.50 |
| 128 | 0.43 |
| 129 | 1.00 |
| 130 | 0.00 |
| 131 | 0.00 |
| 132 | 1.00 |
| 133 | 0.00 |
| 134 | 0.00 |
| 135 | 0.71 |
| 136 | 0.25 |
| 137 | 1.00 |
| 138 | 0.10 |
| 139 | 0.23 |
| 140 | 0.00 |
| 141 | 0.21 |
| 142 | 0.44 |
| 143 | 1.00 |
Task 1b Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Simple Hit/Miss Count: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask1bSimpleByGroup.friedman.tukeyKramerHSD.csv not found
MRR Method: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask1bMrrByGroup.friedman.tukeyKramerHSD.csv not found
Task 1b Summary Results by Query Group
Simple Hit/Miss Counting file /nema-raid/www/mirex/results/2011/qbt/QbtTask1bSimpleByGroup.csv not found
MRR Method file /nema-raid/www/mirex/results/2011/qbt/QbtTask1bMrrByGroup.csv not found
Task 1b Summary Results by Query
Simple Hit/Miss Counting [3]
MRR Method [4]
Task 2 Results
Task 2 Overall Results
| CJ1 | |
|---|---|
| 1 | 0.846 |
| 2 | 1.000 |
| 3 | 1.000 |
| 4 | 1.000 |
| 5 | 1.000 |
| 6 | 0.271 |
| 7 | 1.000 |
| 8 | 0.750 |
| 9 | 0.250 |
| 10 | 0.200 |
| 11 | 0.625 |
| 12 | 0.750 |
| 13 | 0.250 |
| 14 | 0.875 |
| 15 | 1.000 |
| 16 | 0.334 |
| 17 | 0.689 |
| 18 | 0.641 |
| 19 | 0.250 |
| 20 | 1.000 |
| 21 | 0.250 |
| 22 | 0.750 |
| 23 | 0.000 |
| 24 | 1.000 |
| 25 | 0.871 |
| 26 | 0.000 |
| 27 | 0.394 |
| 28 | 0.833 |
| 29 | 1.000 |
| 30 | 0.600 |
| 31 | 0.143 |
| 32 | 0.778 |
| 33 | 0.202 |
| 34 | 0.767 |
| 35 | 1.000 |
| 36 | 0.400 |
| 37 | 0.000 |
| 38 | 1.000 |
| 39 | 0.571 |
| 40 | 0.869 |
| 41 | 0.583 |
| 42 | 1.000 |
| 43 | 0.700 |
| 44 | 0.563 |
| 45 | 0.050 |
| 46 | 1.000 |
| 47 | 0.500 |
| 48 | 0.333 |
| 49 | 0.625 |
| 50 | 0.500 |
| 51 | 0.222 |
| 52 | 1.000 |
| 53 | 0.367 |
| 54 | 1.000 |
| 55 | 0.509 |
| 56 | 1.000 |
| 57 | 0.867 |
| 58 | 1.000 |
| 59 | 0.000 |
| 60 | 0.124 |
| 61 | 0.955 |
| 62 | 0.917 |
| 63 | 0.100 |
| 64 | 0.286 |
| 65 | 1.000 |
| 66 | 1.000 |
| 67 | 0.000 |
| 68 | 0.435 |
| 69 | 0.611 |
| 70 | 0.500 |
| 71 | 0.000 |
| 72 | 0.667 |
| 73 | 0.810 |
| 74 | 1.000 |
| 75 | 0.833 |
| 76 | 0.875 |
| 77 | 0.000 |
| 78 | 0.000 |
| 79 | 0.375 |
| 80 | 0.667 |
| 81 | 0.167 |
| 82 | 1.000 |
| 83 | 1.000 |
| 84 | 1.000 |
| 85 | 0.376 |
| 86 | 1.000 |
| 87 | 0.308 |
| 88 | 1.000 |
| 89 | 0.733 |
| 90 | 0.000 |
| 91 | 1.000 |
| 92 | 0.125 |
| 93 | 0.583 |
| 94 | 0.250 |
| 95 | 0.893 |
| 96 | 1.000 |
| 97 | 1.000 |
| 98 | 0.500 |
| 99 | 0.750 |
| 100 | 0.625 |
| 101 | 0.064 |
| 102 | 0.875 |
| 103 | 1.000 |
| 104 | 0.866 |
| 105 | 0.873 |
| 106 | 0.443 |
| 107 | 0.437 |
| 108 | 0.438 |
| 109 | 0.735 |
| 110 | 0.867 |
| 111 | 1.000 |
| 112 | 0.588 |
| 113 | 0.493 |
| 114 | 0.373 |
| 115 | 0.400 |
| 116 | 0.500 |
| 117 | 0.375 |
| 118 | 0.000 |
| 119 | 0.333 |
| 120 | 0.667 |
| 121 | 0.667 |
| 122 | 0.000 |
| 123 | 0.333 |
| 124 | 0.333 |
| 125 | 0.120 |
| 126 | 0.000 |
| 127 | 1.000 |
| 128 | 0.667 |
| 129 | 0.500 |
| 130 | 0.500 |
| 131 | 1.000 |
| 132 | 0.500 |
| 133 | 0.100 |
| 134 | 1.000 |
| 135 | 1.000 |
| 136 | 0.000 |
Task 2 Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Simple Hit/Miss Count: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask2SimpleByGroup.friedman.tukeyKramerHSD.csv not found
MRR Method: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask2MrrByGroup.friedman.tukeyKramerHSD.csv not found
Task 2 Summary Results by Query Group
Simple Hit/Miss Counting file /nema-raid/www/mirex/results/2011/qbt/QbtTask2SimpleByGroup.csv not found
MRR Method file /nema-raid/www/mirex/results/2011/qbt/QbtTask2MrrByGroup.csv not found
Task 2 Summary Results by Query
Simple Hit/Miss Counting [5]
MRR Method [6]
Runtime Results
file /nema-raid/www/mirex/results/2011/qbt/qbtRunTime.csv not found
