Difference between revisions of "2006:QBSH: Query-by-Singing/Humming"
(→TASK2: Variants Retrieval Task) |
|||
| (16 intermediate revisions by 6 users not shown) | |||
| Line 1: | Line 1: | ||
| + | ==Results== | ||
| + | Results are on [[2006:QBSH:_Query-by-Singing/Humming Results | QBSH: Query-by-Singing/Humming ]] page. | ||
| + | |||
==Overview== | ==Overview== | ||
| − | The QBSH: Query-by-Singing/Humming | + | The QBSH: Query-by-Singing/Humming task comprises a new set of two closely related sub-tasks for MIREX 2006. The goal of the QBSH task is the evaluation of MIR systems that take as query input queries sung or hummed by real-world users. Prof. J.-S. Roger Jang has created a set of 2797 recorded query files representing requests for 48 different music pieces. |
| + | |||
| + | ==Discussion Forum== | ||
| + | In an effort to keep this page nice and clean so participants can understand what they need to do to prepare their submissions, I have created a [[2006:QBSH Discussion Page]]. Please use the Discussion Page for questions, comments, suggestions and debates concerning the details of the QBSH Tasks. | ||
==Task Descriptions== | ==Task Descriptions== | ||
| − | There will be '''TWO''' QBSH tasks (thus TWO '''index''' buildings and TWO '''test''' runs): | + | There will be '''TWO''' QBSH sub-tasks (thus TWO '''index''' buildings and TWO '''test''' runs): |
===TASK1: Known-Item Retrieval Task=== | ===TASK1: Known-Item Retrieval Task=== | ||
| Line 9: | Line 15: | ||
Based on Prof. Jang's original idea to test for the ability to find the "ground-truth" needle in a collection "haystack". | Based on Prof. Jang's original idea to test for the ability to find the "ground-truth" needle in a collection "haystack". | ||
| − | * '''Test database''': 48 ground-truth MIDIs + ~2000 Essen MIDI noise files | + | * '''Test database''': 48 ground-truth MIDIs + ~2000 Essen Collection MIDI noise files |
| − | * '''Queries:''' | + | * '''Queries:''' 2797 sung queries [1] |
| − | * '''Evaluation''': Mean Reciprocal rank, over top | + | * '''Evaluation''': Mean Reciprocal rank, over top 10 returns, of the "know-item" ground-truth file for each sung query |
| + | [1] The sung queries are being represented by audio (.wav), pitch vector (.pv) and MIDI (.mid) files transcribing the pitch vectors. Participants will be able to choose whether to use the audio, pitch vector or MIDI files for querying. | ||
===TASK2: Variants Retrieval Task=== | ===TASK2: Variants Retrieval Task=== | ||
| Line 18: | Line 25: | ||
Based on Prof. Downie's idea that queries are variants of ground-truth. | Based on Prof. Downie's idea that queries are variants of ground-truth. | ||
| − | * '''Test database''': 48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + | + | * '''Test database''': 48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + 2797 sung queries [1] |
| − | * '''Queries''': | + | * '''Queries''': 2797 sung queries + 48 ground truth MIDI files |
* '''Evaluation''': Classic precision and recall over X top returns | * '''Evaluation''': Classic precision and recall over X top returns | ||
[[User:Admin|J. Stephen Downie]] 13:30, 21 July 2006 (CDT) | [[User:Admin|J. Stephen Downie]] 13:30, 21 July 2006 (CDT) | ||
| + | |||
| + | |||
==Moderators== | ==Moderators== | ||
* J.-S. Roger Jang | * J.-S. Roger Jang | ||
* J. Stephen Downie | * J. Stephen Downie | ||
| − | * Rainer Typke | + | * Rainer Typke begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting |
Latest revision as of 12:51, 29 July 2010
Contents
Results
Results are on QBSH: Query-by-Singing/Humming page.
Overview
The QBSH: Query-by-Singing/Humming task comprises a new set of two closely related sub-tasks for MIREX 2006. The goal of the QBSH task is the evaluation of MIR systems that take as query input queries sung or hummed by real-world users. Prof. J.-S. Roger Jang has created a set of 2797 recorded query files representing requests for 48 different music pieces.
Discussion Forum
In an effort to keep this page nice and clean so participants can understand what they need to do to prepare their submissions, I have created a 2006:QBSH Discussion Page. Please use the Discussion Page for questions, comments, suggestions and debates concerning the details of the QBSH Tasks.
Task Descriptions
There will be TWO QBSH sub-tasks (thus TWO index buildings and TWO test runs):
TASK1: Known-Item Retrieval Task
Based on Prof. Jang's original idea to test for the ability to find the "ground-truth" needle in a collection "haystack".
- Test database: 48 ground-truth MIDIs + ~2000 Essen Collection MIDI noise files
- Queries: 2797 sung queries [1]
- Evaluation: Mean Reciprocal rank, over top 10 returns, of the "know-item" ground-truth file for each sung query
[1] The sung queries are being represented by audio (.wav), pitch vector (.pv) and MIDI (.mid) files transcribing the pitch vectors. Participants will be able to choose whether to use the audio, pitch vector or MIDI files for querying.
TASK2: Variants Retrieval Task
Based on Prof. Downie's idea that queries are variants of ground-truth.
- Test database: 48 ground truth MIDIs + ~ 2000 Essen MIDI noise files + 2797 sung queries [1]
- Queries: 2797 sung queries + 48 ground truth MIDI files
- Evaluation: Classic precision and recall over X top returns
J. Stephen Downie 13:30, 21 July 2006 (CDT)
Moderators
- J.-S. Roger Jang
- J. Stephen Downie
- Rainer Typke begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting
