Difference between revisions of "2006:QBSH: Query-by-Singing/Humming"
(→TASK1: Known-Item Retrieval Task) |
(→Task Descriptions) |
||
| Line 3: | Line 3: | ||
==Task Descriptions== | ==Task Descriptions== | ||
| − | There will be '''TWO''' QBSH tasks (thus TWO '''index''' buildings and TWO '''test''' runs): | + | There will be '''TWO''' QBSH sub-tasks (thus TWO '''index''' buildings and TWO '''test''' runs): |
===TASK1: Known-Item Retrieval Task=== | ===TASK1: Known-Item Retrieval Task=== | ||
Revision as of 12:43, 21 July 2006
Contents
Overview
The QBSH: Query-by-Singing/Humming task comprises a new set of two closely related sub-tasks for MIREX 2006. The goal of the QBSH task is the evaluation of MIR systems that take as query input queries sung or hummed by real-world users. Prof. J.-S. Roger Jang has created a set of 2719 recorded query files representing requests for 48 different music pieces.
Task Descriptions
There will be TWO QBSH sub-tasks (thus TWO index buildings and TWO test runs):
TASK1: Known-Item Retrieval Task
Based on Prof. Jang's original idea to test for the ability to find the "ground-truth" needle in a collection "haystack".
- Test database: 48 ground-truth MIDIs + ~2000 Essen Collection MIDI noise files
- Queries: 2719 sung queries [1]
- Evaluation: Mean Reciprocal rank, over top X returns, of the "know-item" ground-truth file for each sung query
[1] The sung queries are being represented by both pitch vector (.pv) and MIDI (.mid) files transcribing the pitch vectors. Participants will be able to choose whether to use the pitch vectors or MIDI files.
TASK2: Variants Retrieval Task
Based on Prof. Downie's idea that queries are variants of ground-truth.
- Test database: 48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + 2719 sung queries [1]
- Queries: 2719 sung queries + 48 ground truth MIDI files
- Evaluation: Classic precision and recall over X top returns
J. Stephen Downie 13:30, 21 July 2006 (CDT)
Moderators
- J.-S. Roger Jang
- J. Stephen Downie
- Rainer Typke
