Difference between revisions of "2009:Query-by-Tapping Results"
(→Microphone) |
|||
Line 1: | Line 1: | ||
+ | ==Introduction== | ||
+ | These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the [[Query by Singing/Humming]] page. | ||
+ | |||
+ | ===Task Descriptions=== | ||
+ | |||
+ | '''Task 1 [[#Task 1 Results|Goto Task 1 Results]]''': The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used: | ||
+ | |||
+ | * [[Task 1, Jang's dataset Results]] Roger Jang's [http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries. | ||
+ | * [[Task 1, Hsiao's dataset Results]] Show Hsiao's [http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries. | ||
+ | |||
+ | |||
+ | '''Task 2 [[#Task 2 Results|Goto Task 2 Results]]''':The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files. | ||
+ | ===General Legend=== | ||
+ | ====Team ID==== | ||
+ | '''CSJ''' = [[Chun-Ta Chen and Jyh-Shing Roger Jang]] | ||
+ | |||
+ | '''HAFR''' = [[Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine]] | ||
+ | |||
+ | '''HL''' = [[Shu-Jen Show and Hsiao Tyne Liang]] | ||
+ | |||
+ | |||
+ | ===Task 1 Results=== | ||
+ | |||
+ | |||
+ | ===Task 1, Jang's dataset Results=== | ||
+ | |||
+ | =====Task 1 Overall Results===== | ||
+ | <csv>qbsh/qbsh_task1_summary.csv</csv> | ||
+ | |||
+ | ====Task 1 Friedman's Test for Significant Differences==== | ||
+ | The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. | ||
+ | Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); | ||
+ | <csv>qbsh/qbsh.task1.friedman_detailed.csv</csv> | ||
+ | |||
+ | [[Image:Qbsh.task1.friedman.small.png]] | ||
+ | |||
+ | ====Task 1 Summary Results by Query Group==== | ||
+ | <csv>qbsh/qbsh.task1.res.byQueryGroup.csv</csv> | ||
+ | |||
+ | |||
+ | ===Task 1, Hsiao's dataset Results=== | ||
+ | |||
+ | ===Task 2 Results=== | ||
+ | In this subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries. | ||
+ | |||
+ | =====Task 2 Overall Results===== | ||
+ | <csv>qbsh/qbsh.task2.summary.csv</csv> | ||
+ | |||
+ | ====Task 2 Friedman's Test for Significant Differences==== | ||
+ | The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. | ||
+ | Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); | ||
+ | <csv>qbsh/qbsh.task2.friedman_detailed.csv</csv> | ||
+ | |||
+ | [[Image:Qbsh.task2.friedman.s.png]] | ||
+ | |||
+ | ====Task 2 Summary Results by Query Group==== | ||
+ | <csv>qbsh/qbsh_task2_res_byQueryGroup.csv</csv> | ||
+ | |||
+ | ===Runtime Results=== | ||
+ | |||
+ | <csv>qbsh.runtime.csv</csv> | ||
+ | |||
+ | [[Category: Results]] | ||
+ | |||
+ | |||
+ | ==Introduction== | ||
+ | |||
+ | |||
+ | ===Task Descriptions=== | ||
+ | |||
+ | ===General Legend=== | ||
+ | ===Team ID=== | ||
+ | |||
+ | '''CSJ''' = [[Chun-Ta Chen and Jyh-Shing Roger Jang]] | ||
+ | |||
+ | '''HAFR1''' = [[Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine]] | ||
+ | |||
+ | '''HAFR2''' = [[Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine]] | ||
+ | |||
+ | '''HL''' = [[Shu-Jen Show and Hsiao Tyne Liang]] | ||
+ | |||
+ | |||
+ | ====Jang Collection ==== | ||
+ | |||
+ | ====Think IT==== | ||
+ | |||
+ | ====Queries as Variation==== | ||
+ | |||
+ | |||
+ | |||
===General Legend=== | ===General Legend=== | ||
====Team ID==== | ====Team ID==== |
Revision as of 15:07, 11 October 2009
Contents
Introduction
These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the Query by Singing/Humming page.
Task Descriptions
Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:
- Task 1, Jang's dataset Results Roger Jang's MIR-QBT: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries.
- Task 1, Hsiao's dataset Results Show Hsiao's QBT_symbolic: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries.
Task 2 Goto Task 2 Results:The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.
General Legend
Team ID
CSJ = Chun-Ta Chen and Jyh-Shing Roger Jang
HAFR = Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine
HL = Shu-Jen Show and Hsiao Tyne Liang
Task 1 Results
Task 1, Jang's dataset Results
Task 1 Overall Results
file /nema-raid/www/mirex/results/qbsh/qbsh_task1_summary.csv not found
Task 1 Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/qbsh/qbsh.task1.friedman_detailed.csv not found
File:Qbsh.task1.friedman.small.png
Task 1 Summary Results by Query Group
file /nema-raid/www/mirex/results/qbsh/qbsh.task1.res.byQueryGroup.csv not found
Task 1, Hsiao's dataset Results
Task 2 Results
In this subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.
Task 2 Overall Results
file /nema-raid/www/mirex/results/qbsh/qbsh.task2.summary.csv not found
Task 2 Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/qbsh/qbsh.task2.friedman_detailed.csv not found
File:Qbsh.task2.friedman.s.png
Task 2 Summary Results by Query Group
file /nema-raid/www/mirex/results/qbsh/qbsh_task2_res_byQueryGroup.csv not found
Runtime Results
file /nema-raid/www/mirex/results/qbsh.runtime.csv not found
Introduction
Task Descriptions
General Legend
Team ID
CSJ = Chun-Ta Chen and Jyh-Shing Roger Jang
HAFR1 = Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine
HAFR2 = Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine
HL = Shu-Jen Show and Hsiao Tyne Liang
Jang Collection
Think IT
Queries as Variation
General Legend
Team ID
CSJ1 = Chun-Ta Chen and Jyh-Shing Roger Jang
CSJ2 = Chun-Ta Chen and Jyh-Shing Roger Jang
HAFR1 = Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine
HAFR2 = Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine
HL = Shu-Jen Show Hsiao and Tyne Liang