Difference between revisions of "2009:Query-by-Tapping Results"

From MIREX Wiki
Line 23: Line 23:
  
  
−
===Task 1, Jang's dataset Results===
+
===Task 1a, Jang's Dataset Results===
  
−
=====Task 1 Overall Results=====
+
=====Task 1a Overall Results=====
−
<csv>qbsh/qbsh_task1_summary.csv</csv>
+
<csv>qbt/QbtFinalTask1a.csv</csv>
  
−
====Task 1 Friedman's Test for Significant Differences====
+
====Task 1a Friedman's Test for Significant Differences====
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Line 35: Line 35:
 
[[Image:Qbsh.task1.friedman.small.png]]
 
[[Image:Qbsh.task1.friedman.small.png]]
  
−
====Task 1 Summary Results by Query Group====
+
====Task 1a Summary Results by Query Group====
−
<csv>qbsh/qbsh.task1.res.byQueryGroup.csv</csv>
+
Simple Hit/Miss Counting
 +
<csv>qbt/QbtTask1aSimpleByGroup.csv</csv>
  
 +
MRR Method
 +
<csv>qbt/QbtTask1aMrrByGroup.csv</csv>
  
−
===Task 1, Hsiao's dataset Results===
 
  
−
===Task 2 Results===
+
====Task 1a Summary Results by Query ====
−
In this subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.
+
Simple Hit/Miss Counting
 +
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask1aSimpleByQuery.csv]
  
−
=====Task 2 Overall Results=====
+
MRR Method
−
<csv>qbsh/qbsh.task2.summary.csv</csv>
+
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask1aMrrByQuery.csv]
  
−
====Task 2 Friedman's Test for Significant Differences====
 
−
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
−
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
−
<csv>qbsh/qbsh.task2.friedman_detailed.csv</csv>
 
  
−
[[Image:Qbsh.task2.friedman.s.png]]
+
===Task 1b, Hsiao's Dataset Results===
  
−
====Task 2 Summary Results by Query Group====
+
=====Task 1b Overall Results=====
−
<csv>qbsh/qbsh_task2_res_byQueryGroup.csv</csv>
+
<csv>qbt/QbtFinalTask1b.csv</csv>
  
−
===Runtime Results===
+
====Task 1b Friedman's Test for Significant Differences====
 +
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 +
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 +
<csv>qbsh/qbsh.task1.friedman_detailed.csv</csv>
  
−
<csv>qbsh.runtime.csv</csv>
+
[[Image:Qbsh.task1.friedman.small.png]]
  
−
[[Category: Results]]
+
====Task 1b Summary Results by Query Group====
 +
Simple Hit/Miss Counting
 +
<csv>qbt/QbtTask1bSimpleByGroup.csv</csv>
  
 +
MRR Method
 +
<csv>qbt/QbtTask1bMrrByGroup.csv</csv>
  
−
==Introduction==
 
  
 +
====Task 1b Summary Results by Query ====
 +
Simple Hit/Miss Counting
 +
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask1bSimpleByQuery.csv]
  
−
===Task Descriptions===
+
MRR Method
 +
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask1bMrrByQuery.csv]
  
−
===General Legend===
+
===Task 2 Results===
−
===Team ID===
 
  
−
'''CSJ''' = [[Chun-Ta Chen and Jyh-Shing Roger Jang]]
 
  
−
'''HAFR1''' = [[Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine]]
+
=====Task 2 Overall Results=====
 +
<csv>qbt/QbtFinalTask2.csv</csv>
  
−
'''HAFR2''' = [[Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine]]
+
====Task 2 Friedman's Test for Significant Differences====
 +
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 +
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 +
<csv>qbsh/qbsh.task1.friedman_detailed.csv</csv>
  
−
'''HL''' = [[Shu-Jen Show and Hsiao Tyne Liang]]
+
[[Image:Qbsh.task1.friedman.small.png]]
  
 +
====Task 2 Summary Results by Query Group====
 +
Simple Hit/Miss Counting
 +
<csv>qbt/QbtTask2SimpleByGroup.csv</csv>
  
−
====Jang Collection ====
+
MRR Method
 +
<csv>qbt/QbtTask2MrrByGroup.csv</csv>
  
−
====Think IT====
 
  
−
====Queries as Variation====
+
====Task 2 Summary Results by Query ====
 +
Simple Hit/Miss Counting
 +
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask2SimpleByQuery.csv]
  
 +
MRR Method
 +
[https://www.music-ir.org/mirex/2009/results/qbt/QbtTask2MrrByQuery.csv]
  
 +
===Runtime Results===
  
−
===General Legend===
+
<csv>qbsh.runtime.csv</csv>
−
====Team ID====
 
  
−
'''CSJ1''' = [[Chun-Ta Chen and Jyh-Shing Roger Jang]]
+
[[Category: Results]]
−
 
 
−
'''CSJ2''' = [[Chun-Ta Chen and Jyh-Shing Roger Jang]]
 
−
 
 
−
'''HAFR1''' = [[Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine]]
 
−
 
 
−
'''HAFR2''' = [[Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine]]
 
−
 
 
−
'''HL''' = [[Shu-Jen Show Hsiao and Tyne Liang]]
 
−
 
 
−
 
 
−
 
 
−
====Keyboard Collction====
 
−
 
 
−
====Microphone====
 

Revision as of 19:24, 12 October 2009

Introduction

These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the Query by Singing/Humming page.

Task Descriptions

Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:


Task 2 Goto Task 2 Results:The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.

General Legend

Team ID

CSJ = Chun-Ta Chen and Jyh-Shing Roger Jang

HAFR = Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine

HL = Shu-Jen Show and Hsiao Tyne Liang


Task 1 Results

Task 1a, Jang's Dataset Results

Task 1a Overall Results

file /nema-raid/www/mirex/results/qbt/QbtFinalTask1a.csv not found

Task 1a Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/qbsh/qbsh.task1.friedman_detailed.csv not found

File:Qbsh.task1.friedman.small.png

Task 1a Summary Results by Query Group

Simple Hit/Miss Counting file /nema-raid/www/mirex/results/qbt/QbtTask1aSimpleByGroup.csv not found

MRR Method file /nema-raid/www/mirex/results/qbt/QbtTask1aMrrByGroup.csv not found


Task 1a Summary Results by Query

Simple Hit/Miss Counting [1]

MRR Method [2]


Task 1b, Hsiao's Dataset Results

Task 1b Overall Results

file /nema-raid/www/mirex/results/qbt/QbtFinalTask1b.csv not found

Task 1b Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/qbsh/qbsh.task1.friedman_detailed.csv not found

File:Qbsh.task1.friedman.small.png

Task 1b Summary Results by Query Group

Simple Hit/Miss Counting file /nema-raid/www/mirex/results/qbt/QbtTask1bSimpleByGroup.csv not found

MRR Method file /nema-raid/www/mirex/results/qbt/QbtTask1bMrrByGroup.csv not found


Task 1b Summary Results by Query

Simple Hit/Miss Counting [3]

MRR Method [4]

Task 2 Results

Task 2 Overall Results

file /nema-raid/www/mirex/results/qbt/QbtFinalTask2.csv not found

Task 2 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/qbsh/qbsh.task1.friedman_detailed.csv not found

File:Qbsh.task1.friedman.small.png

Task 2 Summary Results by Query Group

Simple Hit/Miss Counting file /nema-raid/www/mirex/results/qbt/QbtTask2SimpleByGroup.csv not found

MRR Method file /nema-raid/www/mirex/results/qbt/QbtTask2MrrByGroup.csv not found


Task 2 Summary Results by Query

Simple Hit/Miss Counting [5]

MRR Method [6]

Runtime Results

file /nema-raid/www/mirex/results/qbsh.runtime.csv not found

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2009:Query-by-Tapping_Results&oldid=5179"