2006:QBSH: Query-by-Singing/Humming Results
Contents
- 1 Introduction
- 2 Overall Summary Results
- 3 Task I: "Known-Item Searching" Friedman Test with Multiple Comparisons Results (p=0.05)
- 4 Task I: "Known-Item Searching" Summary Results
- 5 Task II: "Queries as Variations" Friedman Test with Multiple Comparisons Results (p=0.05)
- 6 Task II: "Queries as Variations" Summary Results
- 7 Raw Scores
Introduction
These are the results for the 2006 running of the QBSH: Query-by-Singing Humming task set. For background information about this task set please refer to the QBSH: Query-by-Singing/Humming page. QBSH task consists of two subtasks. The first subtask is known as Known-Item Retrieval. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. The query database consists of 2797 sung queries.
The second subtask is called Queries as Variations. In this subtask, systems take an input from the query database which consists of 2797 sung queries + 48 ground truth files and return a list of 20 songs from the test database which consists of 48 ground truth MIDIs + 2000 Essen MIDI noise files + 2797 sung queries. The precision based on the number of songs within the same ground truth class of the query is calculated over the top 20 returns for each of the 2845 queries.
General Legend
Team ID
AU = Alexandra Uitdenbogerd
CS1 = Christian Sailer-ear
CS2 = Christian Sailer-midi
CS3 = Christian Sailer-warp
FH = Pascal Ferraro and Pierre Hanna
NM = Kjell Lemström, Niko Mikkilä, Veli Mäkinen and Esko Ukkonen
RJ = Jyh-Shing Roger Jang and Nien-Jung Lee
RL = Ernesto Lopez and Martín Rocamora
RT = Rainer Typke, Frans Wiering and Remco C. Veltkamp
XW = Xiao Wu and Ming Li
Calculating Summary Measures
MRR = Mean Reciprocal Rank. Reciprocal rank is the reciprocal of the rank of the first correctly identified cover for each query (1/rank). These values are averaged for each ground truth group as well as overall.
Overall Summary Results
file /nema-raid/www/mirex/results/qbsh_overall_sum.csv not found
QBSH Task 1 Runtime Data
file /nema-raid/www/mirex/results/qbsh06_task1_runtime.csv not found
QBSH Task 2 Runtime Data
file /nema-raid/www/mirex/results/qbsh06_task2_runtime.csv not found
Task I: "Known-Item Searching" Friedman Test with Multiple Comparisons Results (p=0.05)
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
file /nema-raid/www/mirex/results/qush_friedman.csv not found
file /nema-raid/www/mirex/results/QBSH_sum.csv not found
Task I: "Known-Item Searching" Summary Results
MMR data summarized by the 48 ground truth song groups. file /nema-raid/www/mirex/results/qbsh_task1_sum.csv not found
Task II: "Queries as Variations" Friedman Test with Multiple Comparisons Results (p=0.05)
The Friedman test was run in MATLAB against the QBSH Task 2 mean precision data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
file /nema-raid/www/mirex/results/qbsh2_friedman.csv not found
file /nema-raid/www/mirex/results/qbsh2_sum.csv not found
Task II: "Queries as Variations" Summary Results
Mean precision data summarized over the 48 ground truth song groups. file /nema-raid/www/mirex/results/qbsh06_task2_sum.csv not found
Raw Scores
The raw data are located on the Query-by-Singing/Humming Raw Data page.
