2006:QBSH: Query-by-Singing/Humming

Overview

The QBSH: Query-by-Singing/Humming task comprises a new set of two closely related sub-tasks for MIREX 2006. The goal of the QBSH task is the evaluation of MIR systems that take as query input queries sung or hummed by real-world users. Prof. J.-S. Roger Jang has created a set of 2719 recorded query files representing requests for 48 different music pieces.

Task Descriptions

There will be TWO QBSH tasks (thus TWO index buildings and TWO test runs):

TASK1: Known-Item Retrieval Task

Based on Prof. Jang's original idea to test for the ability to find the "ground-truth" needle in a collection "haystack".

• Test database: 48 ground-truth MIDIs + ~2000 Essen MIDI noise files
• Queries: 2719 sung queries [1]
• Evaluation: Mean Reciprocal rank, over top X returns, of the "know-item" ground-truth file for each sung query

[1] The sung queries are being represented by both pitch vector (.pv) and MIDI (.mid) files transcribing the pitch vectors. Participants will be able to choose whether to use the pitch vectors or MIDI files.

TASK2: Variants Retrieval Task

Based on Prof. Downie's idea that queries are variants of ground-truth.

• Test database: 48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + 2719 sung queries [1]
• Queries: 2719 sung queries + 48 ground truth MIDI files
• Evaluation: Classic precision and recall over X top returns

J. Stephen Downie 13:30, 21 July 2006 (CDT)

Moderators

• J.-S. Roger Jang
• J. Stephen Downie
• Rainer Typke