Difference between revisions of "2006:QBSH: Query-by-Singing/Humming"

From MIREX Wiki
(Task Descriptions)
Line 5: Line 5:
 
There will be '''TWO''' QBSH tasks (thus TWO '''index''' buildings and TWO '''test''' runs):
 
There will be '''TWO''' QBSH tasks (thus TWO '''index''' buildings and TWO '''test''' runs):
  
===TASK1:===  
+
===TASK1: Known-Item Retrieval Task===  
  
Based on Prof. Jang's original idea to test for the ability to find the "ground truth" needle in a collection "haystack".
+
Based on Prof. Jang's original idea to test for the ability to find the "ground-truth" needle in a collection "haystack".
  
* '''Test database''': 48 ground truth MIDIs + ~2000 Essen MIDI noise files
+
* '''Test database''': 48 ground-truth MIDIs + ~2000 Essen MIDI noise files
 
* '''Queries:''' 2719 sung queries [1]
 
* '''Queries:''' 2719 sung queries [1]
* '''Evaluation''': Mean Reciprocal rank over top X returns
+
* '''Evaluation''': Mean Reciprocal rank, over top X returns, of the "know-item" ground-truth file for each sung query
  
  
===TASK2:===
+
===TASK2: Variants Retrieval Task===
  
 
Based on Prof. Downie's idea that queries are variants of ground truth.
 
Based on Prof. Downie's idea that queries are variants of ground truth.
Line 20: Line 20:
 
* '''Test database''': 48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + 2719 sung queries [1]
 
* '''Test database''': 48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + 2719 sung queries [1]
 
* '''Queries''': 2719 sung queries + 48 ground truth MIDI files
 
* '''Queries''': 2719 sung queries + 48 ground truth MIDI files
* '''Evaluation''': Classic precision and recall over X top returns  
+
* '''Evaluation''': Classic precision and recall over X top returns
  
 +
[[User:Admin|Admin]]
  
 
==Moderators==
 
==Moderators==

Revision as of 13:29, 21 July 2006

Overview

The QBSH: Query-by-Singing/Humming is a new task for MIREX 2006. This goal of the QBSH task is the evaluation of MIR systems that take as query input queries sung or hummed by real-world users. Prof. J.-S. Roger Jang has created a set of 2719 recorded query files representing requests for 48 different music pieces.

Task Descriptions

There will be TWO QBSH tasks (thus TWO index buildings and TWO test runs):

TASK1: Known-Item Retrieval Task

Based on Prof. Jang's original idea to test for the ability to find the "ground-truth" needle in a collection "haystack".

  • Test database: 48 ground-truth MIDIs + ~2000 Essen MIDI noise files
  • Queries: 2719 sung queries [1]
  • Evaluation: Mean Reciprocal rank, over top X returns, of the "know-item" ground-truth file for each sung query


TASK2: Variants Retrieval Task

Based on Prof. Downie's idea that queries are variants of ground truth.

  • Test database: 48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + 2719 sung queries [1]
  • Queries: 2719 sung queries + 48 ground truth MIDI files
  • Evaluation: Classic precision and recall over X top returns

Admin

Moderators

  • J.-S. Roger Jang
  • J. Stephen Downie
  • Rainer Typke