2006:QBSH: Query-by-Singing/Humming Results

Introduction

These are the results for the 2006 running of the QBSH: Query-by-Singing Humming task set. For background information about this task set please refer to the QBSH: Query-by-Singing/Humming page.

General Legend

Team ID

AU = Alexandra Uitdenbogerd
CS = Christian Sailer
FH = Pascal Ferraro and Pierre Hanna
NM = Kjell Lemström, Niko Mikkilä, Veli Mäkinen and Esko Ukkonen
RJ = Jyh-Shing Roger Jang and Nien-Jung Lee
RL = Ernesto Lopez and Martín Rocamora
RT = Rainer Typke, Frans Wiering and Remco C. Veltkamp
XW = Xiao Wu and Ming Li

Calculating Summary Measures

MRR = Mean Reciprocal Rank. Reciprocal rank is the reciprocal of the rank of the first correctly identified cover for each query (1/rank). These values are averaged for each ground truth group as well as overall.

Overall Summary Results

INSERT CSVs for overall scores here.

Friedman Test with Multiple Comparisons Results (p=0.05)

The Friedman test was run in MATLAB against the QBSH data over the 30 song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/qush_friedman.csv not found file /nema-raid/www/mirex/results/QBSH_sum.csv not found

Task I: Known-Item Searching Summary Results

file /nema-raid/www/mirex/results/qbsh06_mrr_task1_sum.csv not found

Task IIa: Karaoke Collection Summary Results

file /nema-raid/www/mirex/results/sms06_kar_results2.csv not found

Task IIb: Mixed Polyphonic Collection Summary Results

file /nema-raid/www/mirex/results/sms06_mix_results2.csv not found

Raw Scores

The raw data derived from the Evalutron 6000 human evaluations are located on the Query-by-Singing/Humming Raw Data page.