Difference between revisions of "2006:QBSH: Query-by-Singing/Humming"
From MIREX Wiki
| Line 3: | Line 3: | ||
==Task Descriptions== | ==Task Descriptions== | ||
| − | + | There will be *TWO* QBSH tasks (thus TWO *index* buildings and TWO *test* runs): | |
| − | TASK1: | + | ===TASK1:=== |
| − | + | Based on Prof. Jang's original idea to test for the ability to find the "ground truth" needle in a collection "haystack". | |
| − | |||
| − | |||
| + | * Test database: 48 ground truth MIDIs + ~2000 Essen MIDI noise files | ||
| + | * Queries: 2719 sung queries [1] | ||
| + | * Evaluation: Mean Reciprocal rank over top X returns | ||
| − | |||
| − | + | ===TASK2:=== | |
| − | + | ||
| − | + | Based on Prof. Downie's idea that queries are variants of ground truth. | |
| + | |||
| + | * Test database--48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + 2719 sung queries [1] | ||
| + | * Queries: 2719 sung queries + 48 ground truth MIDI files | ||
| + | * Evaluation: Classic precision and recall over X top returns | ||
Revision as of 07:41, 21 July 2006
Overview
The QBSH: Query-by-Singing/Humming is a new task for MIREX 2006. This goal of the QBSH task is the evaluation of MIR systems that take as query input queries sung or hummed by real-world users. Prof. J.-S. Roger Jang has created a set of 2719 recorded query files representing requests for 48 different music pieces.
Task Descriptions
There will be *TWO* QBSH tasks (thus TWO *index* buildings and TWO *test* runs):
TASK1:
Based on Prof. Jang's original idea to test for the ability to find the "ground truth" needle in a collection "haystack".
- Test database: 48 ground truth MIDIs + ~2000 Essen MIDI noise files
- Queries: 2719 sung queries [1]
- Evaluation: Mean Reciprocal rank over top X returns
TASK2:
Based on Prof. Downie's idea that queries are variants of ground truth.
- Test database--48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + 2719 sung queries [1]
- Queries: 2719 sung queries + 48 ground truth MIDI files
- Evaluation: Classic precision and recall over X top returns
Moderators
J.-S. Roger Jang J. Stephen Downie Rainer Typke
