2006:QBSH: Query-by-Singing/Humming

Overview

The QBSH: Query-by-Singing/Humming task comprises a new set of two closely related sub-tasks for MIREX 2006. The goal of the QBSH task is the evaluation of MIR systems that take as query input queries sung or hummed by real-world users. Prof. J.-S. Roger Jang has created a set of 2719 recorded query files representing requests for 48 different music pieces.

Discussion Forum

In an effort to keep this page nice and clean so participants can understand what they need to do to prepare their submissions, I have created a QBSH Discussion Page. Please use the Discussion Page for questions, comments, suggestions and debates concerning the details of the QBSH Tasks.

Task Descriptions

There will be TWO QBSH sub-tasks (thus TWO index buildings and TWO test runs):

TASK1: Known-Item Retrieval Task

Based on Prof. Jang's original idea to test for the ability to find the "ground-truth" needle in a collection "haystack".

• Test database: 48 ground-truth MIDIs + ~2000 Essen Collection MIDI noise files
• Queries: 2719 sung queries [1]
• Evaluation: Mean Reciprocal rank, over top X returns, of the "know-item" ground-truth file for each sung query

[1] The sung queries are being represented by audio (.wav), pitch vector (.pv) and MIDI (.mid) files transcribing the pitch vectors. Participants will be able to choose whether to use the audio, pitch vector or MIDI files for querying.

TASK2: Variants Retrieval Task

Based on Prof. Downie's idea that queries are variants of ground-truth.

• Test database: 48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + 2719 sung queries [1]
• Queries: 2719 sung queries + 48 ground truth MIDI files
• Evaluation: Classic precision and recall over X top returns

J. Stephen Downie 13:30, 21 July 2006 (CDT)

Participants

Alexandra Uitdenbogerd

Moderators

• J.-S. Roger Jang
• J. Stephen Downie
• Rainer Typke