Difference between revisions of "2009:Query-by-Singing/Humming Results"

From MIREX Wiki
(→‎Team ID)
Line 6: Line 6:
 
'''Task 1 [[#Task 1 Results|Goto Task 1 Results]]''': The first subtask is the same as last year. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:
 
'''Task 1 [[#Task 1 Results|Goto Task 1 Results]]''': The first subtask is the same as last year. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:
  
−
*[[#Task 1, Jang's dataset Results]] Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBSH-corpus.rar MIR-QBSH corpus] with 48 songs as ground truth + 2000 Essen Collection MIDI noise files. See [http://www.esac-data.org/  ESAC Data Homepage] for more information about the Essen Collection. The queries consists of 4431 humming. All queries are from the beginning of references  
+
*[[#Task 1a, Jang's dataset Results|Jang's Dataset Results]] Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBSH-corpus.rar MIR-QBSH corpus] with 48 songs as ground truth + 2000 Essen Collection MIDI noise files. See [http://www.esac-data.org/  ESAC Data Homepage] for more information about the Essen Collection. The queries consists of 4431 humming. All queries are from the beginning of references  
−
*[[#Task 1, ThinkIT's dataset Results]] [http://mirlab.org/dataSet/public/IOACAS_QBH_Coprus1.rar IOACAS corpus 1] data set with 106 songs as ground truth + 2000 Essen Collection MIDI noise files. See [http://www.esac-data.org/  ESAC Data Homepage] for more information about the Essen Collection. The queries consists of 355 humming. There are no "singing from beginning" gurantee.
+
*[[#Task 1b, ThinkIT's dataset Results|ThinkIT's Dataset Results]] [http://mirlab.org/dataSet/public/IOACAS_QBH_Coprus1.rar IOACAS corpus 1] data set with 106 songs as ground truth + 2000 Essen Collection MIDI noise files. See [http://www.esac-data.org/  ESAC Data Homepage] for more information about the Essen Collection. The queries consists of 355 humming. There are no "singing from beginning" gurantee.
−
 
+
*[[#Task 1c, ThinkIT's dataset Results|ThinkIT's Dataset Results]] [http://mirlab.org/dataSet/public/IOACAS_QBH_Coprus2.rar IOACAS corpus 2] data set with 192 songs as ground truth + 2000 Essen Collection MIDI noise files. See [http://www.esac-data.org/  ESAC Data Homepage] for more information about the Essen Collection. The queries consists of 404 humming. There are no "singing from beginning" gurantee.
  
 
'''Task 2 [[#Task 2 Results|Goto Task 2 Results]]''': The second subtask is the query against other humming. In the second subtask, Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBSH-corpus.rar MIR-QBSH corpus] has been divided into two groups (2040 as queries and 2391 as database). The query is performed against the other humming database and the top 10 closed are returned. The score is simple count of how many returns belong to the same ground truth song.   
 
'''Task 2 [[#Task 2 Results|Goto Task 2 Results]]''': The second subtask is the query against other humming. In the second subtask, Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBSH-corpus.rar MIR-QBSH corpus] has been divided into two groups (2040 as queries and 2391 as database). The query is performed against the other humming database and the top 10 closed are returned. The score is simple count of how many returns belong to the same ground truth song.   
Line 21: Line 21:
  
  
−
===Task 1, Jang's dataset Results===
+
====Task 1a, Jang's dataset Results====
  
−
=====Task 1 Overall Results=====
+
=====Task 1a Overall Results=====
−
<csv>qbsh/qbsh_task1_summary.csv</csv>
+
<csv>qbsh/QbshFinalTask1a.csv</csv>
  
−
====Task 1 Friedman's Test for Significant Differences====
+
=====Task 1a Friedman's Test for Significant Differences=====
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Line 33: Line 33:
 
[[Image:Qbsh.task1.friedman.small.png]]
 
[[Image:Qbsh.task1.friedman.small.png]]
  
−
====Task 1 Summary Results by Query Group====
+
=====Task 1a Summary Results by Query Group=====
−
<csv>qbsh/qbsh.task1.res.byQueryGroup.csv</csv>
+
Simple Hit/Miss Count
 +
<csv>qbsh/QbshTask1aSimpleByGroup.csv</csv>
 +
 
 +
MRR Method
 +
<csv>qbsh/QbshTask1aMrrByGroup.csv</csv>
  
 +
====Task 1a Summary Results by Query ====
 +
Simple Hit/Miss Counting
 +
[https://www.music-ir.org/mirex/2009/results/qbsh/QbshTask1aSimpleByQuery.csv]
  
−
===Task 1, ThinkIT's dataset Results===
+
MRR Method
 +
[https://www.music-ir.org/mirex/2009/results/qbsh/QbshTask1aMrrByQuery.csv]
  
−
===Task 2 Results===
+
====Task 1b, ThinkIT's dataset Results====
−
In this subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.
 
  
−
=====Task 2 Overall Results=====
+
=====Task 1b Overall Results=====
−
<csv>qbsh/qbsh.task2.summary.csv</csv>
+
<csv>qbsh/QbshFinalTask1b.csv</csv>
  
−
====Task 2 Friedman's Test for Significant Differences====
+
=====Task 1b Friedman's Test for Significant Differences=====
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
−
<csv>qbsh/qbsh.task2.friedman_detailed.csv</csv>
+
<csv>qbsh/qbsh.task1.friedman_detailed.csv</csv>
 +
 
 +
[[Image:Qbsh.task1.friedman.small.png]]
  
−
[[Image:Qbsh.task2.friedman.s.png]]
+
=====Task 1b Summary Results by Query Group=====
 +
Simple Hit/Miss Count
 +
<csv>qbsh/QbshTask1bSimpleByGroup.csv</csv>
 +
 
 +
MRR Method
 +
<csv>qbsh/QbshTask1bMrrByGroup.csv</csv>
 +
 
 +
====Task 1b Summary Results by Query ====
 +
Simple Hit/Miss Counting
 +
[https://www.music-ir.org/mirex/2009/results/qbsh/QbshTask1bSimpleByQuery.csv]
 +
 
 +
MRR Method
 +
[https://www.music-ir.org/mirex/2009/results/qbsh/QbshTask1bMrrByQuery.csv]
 +
 
 +
====Task 1c, IOACAS2's dataset Results====
 +
 
 +
=====Task 1c Overall Results=====
 +
<csv>qbsh/QbshFinalTask1c.csv</csv>
 +
 
 +
=====Task 1c Friedman's Test for Significant Differences=====
 +
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 +
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 +
<csv>qbsh/qbsh.task1.friedman_detailed.csv</csv>
  
−
====Task 2 Summary Results by Query Group====
+
[[Image:Qbsh.task1.friedman.small.png]]
−
<csv>qbsh/qbsh_task2_res_byQueryGroup.csv</csv>
 
  
−
===Runtime Results===
+
=====Task 1c Summary Results by Query Group=====
 +
Simple Hit/Miss Count
 +
<csv>qbsh/QbshTask1cSimpleByGroup.csv</csv>
  
−
<csv>qbsh.runtime.csv</csv>
+
MRR Method
 +
<csv>qbsh/QbshTask1cMrrByGroup.csv</csv>
  
−
[[Category: Results]]
+
====Task 1c Summary Results by Query ====
 +
Simple Hit/Miss Counting
 +
[https://www.music-ir.org/mirex/2009/results/qbsh/QbshTask1cSimpleByQuery.csv]
  
 +
MRR Method
 +
[https://www.music-ir.org/mirex/2009/results/qbsh/QbshTask1cMrrByQuery.csv]
  
−
==Introduction==
+
===Task 2 Results===
  
 +
=====Task 2 Overall Results=====
 +
<csv>qbsh/QbshFinalTask2.csv</csv>
  
−
===Task Descriptions===
+
=====Task 2 Friedman's Test for Significant Differences=====
 +
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.
 +
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 +
<csv>qbsh/qbsh.task1.friedman_detailed.csv</csv>
  
−
===General Legend===
+
[[Image:Qbsh.task1.friedman.small.png]]
−
===Team ID===
 
  
−
'''CSJ''' = [[Chun-Ta Chen and Jyh-Shing Roger Jang]]<br />
+
=====Task 2 Summary Results by Query Group=====
−
'''HAFR1''' = [[Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine]]<br />
+
<csv>qbsh/QbshTask2Group.csv</csv>
−
'''HAFR2''' = [[Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine]]<br />
 
−
'''HL''' = [[Shu-Jen Show and Hsiao Tyne Liang]]<br />
 
  
 +
====Task 2 Summary Results by Query ====
 +
[https://www.music-ir.org/mirex/2009/results/qbsh/QbshTask2Query.csv]
  
−
====Jang Collection ====
+
===Runtime Results===
  
−
====Think IT====
+
<csv>qbsh.runtime.csv</csv>
  
−
====Queries as Variation====
+
[[Category: Results]]

Revision as of 19:56, 12 October 2009

Introduction

These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the Query by Singing/Humming page.

Task Descriptions

Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:

Task 2 Goto Task 2 Results: The second subtask is the query against other humming. In the second subtask, Roger Jang's MIR-QBSH corpus has been divided into two groups (2040 as queries and 2391 as database). The query is performed against the other humming database and the top 10 closed are returned. The score is simple count of how many returns belong to the same ground truth song.

General Legend

Team ID

CSJ1 = Chun-Ta Chen and Jyh-Shing Roger Jang matched by beginning of the sond
CSJ2 = Chun-Ta Chen and Jyh-Shing Roger Jang matched by anywhere of the song
HAFR = Pierre Hanna, Julien Allali, Pascal Ferraro and Matthias Robine

Task 1 Results

Task 1a, Jang's dataset Results

Task 1a Overall Results

file /nema-raid/www/mirex/results/qbsh/QbshFinalTask1a.csv not found

Task 1a Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/qbsh/qbsh.task1.friedman_detailed.csv not found

File:Qbsh.task1.friedman.small.png

Task 1a Summary Results by Query Group

Simple Hit/Miss Count file /nema-raid/www/mirex/results/qbsh/QbshTask1aSimpleByGroup.csv not found

MRR Method file /nema-raid/www/mirex/results/qbsh/QbshTask1aMrrByGroup.csv not found

Task 1a Summary Results by Query

Simple Hit/Miss Counting [1]

MRR Method [2]

Task 1b, ThinkIT's dataset Results

Task 1b Overall Results

file /nema-raid/www/mirex/results/qbsh/QbshFinalTask1b.csv not found

Task 1b Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/qbsh/qbsh.task1.friedman_detailed.csv not found

File:Qbsh.task1.friedman.small.png

Task 1b Summary Results by Query Group

Simple Hit/Miss Count file /nema-raid/www/mirex/results/qbsh/QbshTask1bSimpleByGroup.csv not found

MRR Method file /nema-raid/www/mirex/results/qbsh/QbshTask1bMrrByGroup.csv not found

Task 1b Summary Results by Query

Simple Hit/Miss Counting [3]

MRR Method [4]

Task 1c, IOACAS2's dataset Results

Task 1c Overall Results

file /nema-raid/www/mirex/results/qbsh/QbshFinalTask1c.csv not found

Task 1c Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/qbsh/qbsh.task1.friedman_detailed.csv not found

File:Qbsh.task1.friedman.small.png

Task 1c Summary Results by Query Group

Simple Hit/Miss Count file /nema-raid/www/mirex/results/qbsh/QbshTask1cSimpleByGroup.csv not found

MRR Method file /nema-raid/www/mirex/results/qbsh/QbshTask1cMrrByGroup.csv not found

Task 1c Summary Results by Query

Simple Hit/Miss Counting [5]

MRR Method [6]

Task 2 Results

Task 2 Overall Results

file /nema-raid/www/mirex/results/qbsh/QbshFinalTask2.csv not found

Task 2 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/qbsh/qbsh.task1.friedman_detailed.csv not found

File:Qbsh.task1.friedman.small.png

Task 2 Summary Results by Query Group

file /nema-raid/www/mirex/results/qbsh/QbshTask2Group.csv not found

Task 2 Summary Results by Query

[7]

Runtime Results

file /nema-raid/www/mirex/results/qbsh.runtime.csv not found

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2009:Query-by-Singing/Humming_Results&oldid=5163"