2011:Query-by-Tapping Results
Contents
Introduction
These are the results for the 2011 running of the Query-by-tappingn task. For background information about this task set please refer to the 2011:Query_by_Tapping page.
Task Descriptions
Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:
- Jang's Dataset Roger Jang's MIR-QBT: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries.
- Hsiao's Dataset Show Hsiao's QBT_symbolic: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries.
Task 2 Goto Task 2 Results:The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.
General Legend
Team ID
| Sub code | Submission name | Abstract | Contributors |
|---|---|---|---|
| CJ1 | QBT | ChunTa Chen, Roger Jang |
Task 1 Results
Task 1a, Jang's Dataset Results
Task 1a Overall Results
| CJ1 | |
|---|---|
| Simple Count | 0.747 |
| MRR | 0.599 |
| Total Count | 890 |
Task 1a Summary Results by Query Group
Simple Hit/Miss Counting file /nema-raid/www/mirex/results/2011/qbt/QbtTask1aSimpleByGroup.csv not found
MRR Method file /nema-raid/www/mirex/results/2011/qbt/QbtTask1aMrrByGroup.csv not found
Task 1a Summary Results by Query
Simple Hit/Miss Counting [1]
MRR Method [2]
Task 1b, Hsiao's Dataset Results
Task 1b Overall Results
| CJ1 | |
|---|---|
| Simple Count | 0.68 |
| MRR | 0.51 |
| Total Count | 410.00 |
Task 1b Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Simple Hit/Miss Count: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask1bSimpleByGroup.friedman.tukeyKramerHSD.csv not found
MRR Method: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask1bMrrByGroup.friedman.tukeyKramerHSD.csv not found
Task 1b Summary Results by Query Group
Simple Hit/Miss Counting file /nema-raid/www/mirex/results/2011/qbt/QbtTask1bSimpleByGroup.csv not found
MRR Method file /nema-raid/www/mirex/results/2011/qbt/QbtTask1bMrrByGroup.csv not found
Task 1b Summary Results by Query
Simple Hit/Miss Counting [3]
MRR Method [4]
Task 2 Results
Task 2 Overall Results
| CJ1 | |
|---|---|
| Simple Count | 0.727 |
| MRR | 0.588 |
| Total Count | 890 |
Task 2 Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Simple Hit/Miss Count: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask2SimpleByGroup.friedman.tukeyKramerHSD.csv not found
MRR Method: file /nema-raid/www/mirex/results/2011/qbt/friedman/QbtTask2MrrByGroup.friedman.tukeyKramerHSD.csv not found
Task 2 Summary Results by Query Group
Simple Hit/Miss Counting file /nema-raid/www/mirex/results/2011/qbt/QbtTask2SimpleByGroup.csv not found
MRR Method file /nema-raid/www/mirex/results/2011/qbt/QbtTask2MrrByGroup.csv not found
Task 2 Summary Results by Query
Simple Hit/Miss Counting [5]
MRR Method [6]
Runtime Results
file /nema-raid/www/mirex/results/2011/qbt/qbtRunTime.csv not found
