Difference between revisions of "2006:QBSH: Query-by-Singing/Humming Results"
| Line 2: | Line 2: | ||
==Introduction== | ==Introduction== | ||
These are the results for the 2006 running of the QBSH: Query-by-Singing Humming task set. For background information about this task set please refer to the [[QBSH: Query-by-Singing/Humming]] page. | These are the results for the 2006 running of the QBSH: Query-by-Singing Humming task set. For background information about this task set please refer to the [[QBSH: Query-by-Singing/Humming]] page. | ||
| + | QBSH task consists of two sub tasks. First one is Known-Item Retrieval. The system takes a sung query as input and returns a list of songs from the test database. Mean reciprocal rank of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground truth and 2000 noise midi files. The query database consists of 2797 sung queries. | ||
| + | |||
| + | The second task is the Variants Retrieval. The System takes an input from the query database which consists of 2797 sung queries + 48 ground truth files and returns a list of 20 songs. The precision is calculated over the returns for each query. | ||
===General Legend=== | ===General Legend=== | ||
Revision as of 14:00, 2 October 2006
Contents
- 1 Introduction
- 2 Overall Summary Results
- 3 Task I: "Known-Item Searching" Friedman Test with Multiple Comparisons Results (p=0.05)
- 4 Task I: "Known-Item Searching" Summary Results
- 5 Task II: "Queries as Variations" Friedman Test with Multiple Comparisons Results (p=0.05)
- 6 Task II: "Queries as Variations" Summary Results
- 7 Raw Scores
Introduction
These are the results for the 2006 running of the QBSH: Query-by-Singing Humming task set. For background information about this task set please refer to the QBSH: Query-by-Singing/Humming page. QBSH task consists of two sub tasks. First one is Known-Item Retrieval. The system takes a sung query as input and returns a list of songs from the test database. Mean reciprocal rank of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground truth and 2000 noise midi files. The query database consists of 2797 sung queries.
The second task is the Variants Retrieval. The System takes an input from the query database which consists of 2797 sung queries + 48 ground truth files and returns a list of 20 songs. The precision is calculated over the returns for each query.
General Legend
Team ID
AU = Alexandra Uitdenbogerd
CS1 = Christian Sailer-ear
CS2 = Christian Sailer-midi
CS3 = Christian Sailer-warp
FH = Pascal Ferraro and Pierre Hanna
NM = Kjell Lemström, Niko Mikkilä, Veli Mäkinen and Esko Ukkonen
RJ = Jyh-Shing Roger Jang and Nien-Jung Lee
RL = Ernesto Lopez and Martín Rocamora
RT = Rainer Typke, Frans Wiering and Remco C. Veltkamp
XW = Xiao Wu and Ming Li
Calculating Summary Measures
MRR = Mean Reciprocal Rank. Reciprocal rank is the reciprocal of the rank of the first correctly identified cover for each query (1/rank). These values are averaged for each ground truth group as well as overall.
Overall Summary Results
file /nema-raid/www/mirex/results/qbsh_overall_sum.csv not found
QBSH Task 1 Runtime Data
file /nema-raid/www/mirex/results/qbsh06_task1_runtime.csv not found
QBSH Task 2 Runtime Data
file /nema-raid/www/mirex/results/qbsh06_task2_runtime.csv not found
Task I: "Known-Item Searching" Friedman Test with Multiple Comparisons Results (p=0.05)
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
file /nema-raid/www/mirex/results/qush_friedman.csv not found
file /nema-raid/www/mirex/results/QBSH_sum.csv not found
Task I: "Known-Item Searching" Summary Results
MMR data summarized by the 48 ground truth song groups. file /nema-raid/www/mirex/results/qbsh_task1_sum.csv not found
Task II: "Queries as Variations" Friedman Test with Multiple Comparisons Results (p=0.05)
The Friedman test was run in MATLAB against the QBSH Task 2 mean precision data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
file /nema-raid/www/mirex/results/qbsh2_friedman.csv not found
file /nema-raid/www/mirex/results/qbsh2_sum.csv not found
Task II: "Queries as Variations" Summary Results
Mean precision data summarized over the 48 ground truth song groups. file /nema-raid/www/mirex/results/qbsh06_task2_sum.csv not found
Raw Scores
The raw data are located on the Query-by-Singing/Humming Raw Data page.
