Difference between revisions of "2006:QBSH: Query-by-Singing/Humming"
From MIREX Wiki
(→Task Descriptions) |
|||
| Line 5: | Line 5: | ||
There will be '''TWO''' QBSH tasks (thus TWO '''index''' buildings and TWO '''test''' runs): | There will be '''TWO''' QBSH tasks (thus TWO '''index''' buildings and TWO '''test''' runs): | ||
| − | ===TASK1:=== | + | ===TASK1: Known-Item Retrieval Task=== |
| − | Based on Prof. Jang's original idea to test for the ability to find the "ground truth" needle in a collection "haystack". | + | Based on Prof. Jang's original idea to test for the ability to find the "ground-truth" needle in a collection "haystack". |
| − | * '''Test database''': 48 ground truth MIDIs + ~2000 Essen MIDI noise files | + | * '''Test database''': 48 ground-truth MIDIs + ~2000 Essen MIDI noise files |
* '''Queries:''' 2719 sung queries [1] | * '''Queries:''' 2719 sung queries [1] | ||
| − | * '''Evaluation''': Mean Reciprocal rank over top X returns | + | * '''Evaluation''': Mean Reciprocal rank, over top X returns, of the "know-item" ground-truth file for each sung query |
| − | ===TASK2:=== | + | ===TASK2: Variants Retrieval Task=== |
Based on Prof. Downie's idea that queries are variants of ground truth. | Based on Prof. Downie's idea that queries are variants of ground truth. | ||
| Line 20: | Line 20: | ||
* '''Test database''': 48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + 2719 sung queries [1] | * '''Test database''': 48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + 2719 sung queries [1] | ||
* '''Queries''': 2719 sung queries + 48 ground truth MIDI files | * '''Queries''': 2719 sung queries + 48 ground truth MIDI files | ||
| − | * '''Evaluation''': Classic precision and recall over X top returns | + | * '''Evaluation''': Classic precision and recall over X top returns |
| + | [[User:Admin|Admin]] | ||
==Moderators== | ==Moderators== | ||
Revision as of 12:29, 21 July 2006
Contents
Overview
The QBSH: Query-by-Singing/Humming is a new task for MIREX 2006. This goal of the QBSH task is the evaluation of MIR systems that take as query input queries sung or hummed by real-world users. Prof. J.-S. Roger Jang has created a set of 2719 recorded query files representing requests for 48 different music pieces.
Task Descriptions
There will be TWO QBSH tasks (thus TWO index buildings and TWO test runs):
TASK1: Known-Item Retrieval Task
Based on Prof. Jang's original idea to test for the ability to find the "ground-truth" needle in a collection "haystack".
- Test database: 48 ground-truth MIDIs + ~2000 Essen MIDI noise files
- Queries: 2719 sung queries [1]
- Evaluation: Mean Reciprocal rank, over top X returns, of the "know-item" ground-truth file for each sung query
TASK2: Variants Retrieval Task
Based on Prof. Downie's idea that queries are variants of ground truth.
- Test database: 48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + 2719 sung queries [1]
- Queries: 2719 sung queries + 48 ground truth MIDI files
- Evaluation: Classic precision and recall over X top returns
Moderators
- J.-S. Roger Jang
- J. Stephen Downie
- Rainer Typke
