2010:Query-by-Tapping Results - Revision history

AndreasEhmann at 20:47, 3 August 2010

2010-08-03T20:47:24Z

Gzhu1 at 19:25, 3 August 2010

2010-08-03T19:25:39Z

Gzhu1: /* Team ID */

2010-08-02T18:56:35Z

‎Team ID

Gzhu1 at 17:15, 2 August 2010

2010-08-02T17:15:56Z

Gzhu1: qbt, result

2010-07-28T23:12:36Z

qbt, result

New page

==Introduction==
These are the results for the 2010 running of the Query-by-tappingn task. For background information about this task set please refer to the [[2010:Query_by_Tapping]] page.

===Task Descriptions===

'''Task 1 [[#Task 1 Results|Goto Task 1 Results]]''': The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:

* [[#Task 1a, Jang's Dataset Results|Jang's Dataset]] Roger Jang's [http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries.
* [[#Task 1b, Hsiao's Dataset Results|Hsiao's Dataset]] Show Hsiao's [http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries.

'''Task 2 [[#Task 2 Results|Goto Task 2 Results]]''':The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.

===General Legend===
====Team ID====
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: #efefef;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! HRFA1
| QBT simbals || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/HRFA1.pdf PDF] || [http://dept-info.labri.fr/~hanna Pierre Hanna], [http://simbals.labri.fr Matthias Robine], [http://simbals.labri.fr Pascal Ferraro], [http://simbals.labri.fr Julien Allali]
|-
! ML4
| QBT-M&O || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/ML4.pdf PDF] || [http://www-sipl.technion.ac.il/ Merabi Mirel], [http://www-sipl.technion.ac.il/ Oron Levy]
|}
===Task 1 Results===

===Task 1a, Jang's Dataset Results===

=====Task 1a Overall Results=====
<csv>2010/qbt/QbtFinalTask1a.csv</csv>

====Task 1a Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:
<csv>2010/qbt/friedman/QbtTask1aSimple_friedman_tukeyKramerHSD.csv</csv>

[[Image:2010_sqbttask1asimple_friedman_mean_ranks.png]]

MRR Method:
<csv>2010/qbt/friedman/Qbt1aTask2Mrr_friedman_tukeyKramerHSD.csv</csv>

[[Image:2010_sqbt1atask2mrr_friedman_mean_ranks.png]]

====Task 1a Summary Results by Query Group====
Simple Hit/Miss Counting
<csv p=2>2010/qbt/QbtTask1aSimpleByGroup.csv</csv>

MRR Method
<csv p=2>2010/qbt/QbtTask1aMrrByGroup.csv</csv>

====Task 1a Summary Results by Query ====
Simple Hit/Miss Counting
[https://www.music-ir.org/mirex/results/2010/qbt/QbtTask1aSimpleByQuery.csv]

MRR Method
[https://www.music-ir.org/mirex/results/2010/qbt/QbtTask1aMrrByQuery.csv]

===Task 1b, Hsiao's Dataset Results===

=====Task 1b Overall Results=====
<csv p=2>2010/qbt/QbtFinalTask1b.csv</csv>

====Task 1b Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:
<csv>2010/qbt/friedman/QbtTask1bSimple_friedman_tukeyKramerHSD.csv</csv>

[[Image:2010_sqbttask1bsimple_friedman_mean_ranks.png]]

MRR Method:
<csv>2010/qbt/friedman/QbtTask1bMrr_friedman_tukeyKramerHSD.csv</csv>

[[Image:2010_sqbttask1bmrr_friedman_mean_ranks.png ]]

====Task 1b Summary Results by Query Group====
Simple Hit/Miss Counting
<csv p=2>2010/qbt/QbtTask1bSimpleByGroup.csv</csv>

MRR Method
<csv p=2>2010/qbt/QbtTask1bMrrByGroup.csv</csv>

====Task 1b Summary Results by Query ====
Simple Hit/Miss Counting
[https://www.music-ir.org/mirex/results/2010/qbt/QbtTask1bSimpleByQuery.csv]

MRR Method
[https://www.music-ir.org/mirex/results/2010/qbt/QbtTask1bMrrByQuery.csv]

===Task 2 Results===

=====Task 2 Overall Results=====
<csv>2010/qbt/QbtFinalTask2.csv</csv>

====Task 2 Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:
<csv>2010/qbt/friedman/QbtTask2Simple_friedman_tukeyKramerHSD.csv</csv>

[[Image:2010_sqbttask2simple_friedman_mean_ranks.png]]

MRR Method:
<csv>2010/qbt/friedman/QbtTask2Mrr_friedman_tukeyKramerHSD.csv</csv>

[[Image:2010_sqbttask2mrr_friedman_mean_ranks.png]]

====Task 2 Summary Results by Query Group====
Simple Hit/Miss Counting
<csv p=2>2010/qbt/QbtTask2SimpleByGroup.csv</csv>

MRR Method
<csv p=2>2010/qbt/QbtTask2MrrByGroup.csv</csv>

====Task 2 Summary Results by Query ====
Simple Hit/Miss Counting
[https://www.music-ir.org/mirex/results/2010/qbt/QbtTask2SimpleByQuery.csv]

MRR Method
[https://www.music-ir.org/mirex/results/2010/qbt/QbtTask2MrrByQuery.csv]

===Runtime Results===

<csv>2010/qbt/qbtRunTime.csv</csv>

[[Category: Results]]

@@ Line 46: / Line 46: @@
 MRR Method:
-<csv>2010/qbt/friedman/Qbt1aTask2MrrByGroup.friedman.tukeyKramerHSD.csv</csv>
+<csv>2010/qbt/friedman/QbtTask1aMrrByGroup.friedman.tukeyKramerHSD.csv</csv>
 [[Image:2010QbtTask1aMrrByGroup.friedman.tukeyKramerHSD.png|500px]]

@@ Line 15: / Line 15: @@
 ====Team ID====
 {| border="1" cellspacing="0" style="text-align: left; width: 800px;"
-	|- style="background: #efefef;"
+	|- style="background: yellow;"
 	! width="80" | Sub code
 	! width="200" | Submission name

@@ Line 24: / Line 24: @@
 	| QBT simbals ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/HRFA1.pdf PDF] || [http://dept-info.labri.fr/~hanna Pierre Hanna], [http://simbals.labri.fr Matthias Robine], [http://simbals.labri.fr Pascal Ferraro], [http://simbals.labri.fr Julien Allali]
 	|-
-	! ML4
+	! ML
 	| QBT-M&O ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/ML4.pdf PDF] || [http://www-sipl.technion.ac.il/ Merabi Mirel], [http://www-sipl.technion.ac.il/ Oron Levy]
 	|}
 ===Task 1 Results===

@@ Line 40: / Line 40: @@
 Simple Hit/Miss Count:
-<csv>2010/qbt/friedman/QbtTask1aSimple_friedman_tukeyKramerHSD.csv</csv>
+<csv>2010/qbt/friedman/QbtTask1aSimpleByGroup.friedman.tukeyKramerHSD.csv</csv>
-[[Image:2010_sqbttask1asimple_friedman_mean_ranks.png]]
+[[Image:2010QbtTask1aSimpleByGroup.friedman.tukeyKramerHSD.png|500px]]
 MRR Method:
-<csv>2010/qbt/friedman/Qbt1aTask2Mrr_friedman_tukeyKramerHSD.csv</csv>
+<csv>2010/qbt/friedman/Qbt1aTask2MrrByGroup.friedman.tukeyKramerHSD.csv</csv>
-[[Image:2010_sqbt1atask2mrr_friedman_mean_ranks.png]]
+[[Image:2010QbtTask1aMrrByGroup.friedman.tukeyKramerHSD.png|500px]]
 ====Task 1a Summary Results by Query Group====
@@ Line 74: / Line 74: @@
 Simple Hit/Miss Count:
-<csv>2010/qbt/friedman/QbtTask1bSimple_friedman_tukeyKramerHSD.csv</csv>
+<csv>2010/qbt/friedman/QbtTask1bSimpleByGroup.friedman.tukeyKramerHSD.csv</csv>
-[[Image:2010_sqbttask1bsimple_friedman_mean_ranks.png]]
+[[Image:2010QbtTask1bSimpleByGroup.friedman.tukeyKramerHSD.png|500px]]
 MRR Method:
-<csv>2010/qbt/friedman/QbtTask1bMrr_friedman_tukeyKramerHSD.csv</csv>
+<csv>2010/qbt/friedman/QbtTask1bMrrByGroup.friedman.tukeyKramerHSD.csv</csv>
-[[Image:2010_sqbttask1bmrr_friedman_mean_ranks.png ]]
+[[Image:2010QbtTask1bMrrByGroup.friedman.tukeyKramerHSD.png|500px ]]
 ====Task 1b Summary Results by Query Group====
@@ Line 108: / Line 108: @@
 Simple Hit/Miss Count:
-<csv>2010/qbt/friedman/QbtTask2Simple_friedman_tukeyKramerHSD.csv</csv>
+<csv>2010/qbt/friedman/QbtTask2SimpleByGroup.friedman.tukeyKramerHSD.csv</csv>
-[[Image:2010_sqbttask2simple_friedman_mean_ranks.png]]
+[[Image:2010QbtTask2SimpleByGroup.friedman.tukeyKramerHSD.png|500px]]
 MRR Method:
-<csv>2010/qbt/friedman/QbtTask2Mrr_friedman_tukeyKramerHSD.csv</csv>
+<csv>2010/qbt/friedman/QbtTask2MrrByGroup.friedman.tukeyKramerHSD.csv</csv>
-[[Image:2010_sqbttask2mrr_friedman_mean_ranks.png]]
+[[Image:2010QbtTask2MrrByGroup.friedman.tukeyKramerHSD.png|500px]]
 ====Task 2 Summary Results by Query Group====