MIREX Wiki - User contributions [en]

2015:Task Captains

2015-04-14T16:36:25Z

Blair Kaneshiro:

Like ISMIR 2014, we are prepared to improve the distribution of tasks for the upcoming MIREX 2015. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead one or more tasks, please add your name in the "Captains" column.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

{| class="wikitable" style="margin-left: 20px"
!ID !! Task !! Captain(s)
|-
|abt
|[[2015:Audio Beat Tracking]]
|Sebastian Böck
|-
|ace
|[[2015:Audio Chord Estimation]]
|Johan Pauwels
|-
|act
|[[2015:Audio Classification (Train/Test) Tasks]]
|IMIRSEL
|-
|acs
|[[2015:Audio Cover Song Identification]]
|
|-
|ade
|[[2015:Audio Downbeat Estimation]]
|Florian Krebs, Sebastian Böck
|-
|akd
|[[2015:Audio Key Detection]]
|Johan Pauwels
|-
|ame
|[[2015:Audio Melody Extraction]]
|KETI
|-
|ams
|[[2015:Audio Music Similarity and Retrieval]]
|IMIRSEL
|-
|aod
|[[2015:Audio Onset Detection]]
|Sebastian Böck
|-
|ate
|[[2015:Audio Tempo Estimation]]
|Aggelos Gkiokas
|-
|atg
|[[2015:Audio Tag Classification]]
|
|-
|mf0
|[[2015:Multiple Fundamental Frequency Estimation & Tracking]]
|
|-
|qbsh
|[[2015:Query by Singing/Humming]]
|KETI
|-
|qbt
|[[2015:Query by Tapping]]
| CCRMA
|-
|scofo
|[[2015:Real-time Audio to Score Alignment (a.k.a Score Following)]]
|
|-
|sms
|[[2015:Symbolic Melodic Similarity]]
|
|-
|struct
|[[2015:Structural Segmentation]]
|IMIRSEL
|-
|drts
|[[2015:Discovery of Repeated Themes & Sections]]
|Tom Collins
|-
|afp
|[[2015:Audio_Fingerprinting]]
|Chung-Che Wang
|-
|svs
|[[2015:Singing_Voice_Separation]]
|Tak-Shing Chan, Yi-Hsuan Yang, Li Su
|-
|kgc
|[[2015:Audio K-POP Genre Classification]]
|IMIRSEL
|-
|kmc
|[[2015:Audio K-POP Mood Classification]]
|IMIRSEL
|}

2014:Query by Tapping

2014-10-08T18:31:55Z

Blair Kaneshiro:

== Overview ==
The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Submission deadline is September 16.

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

=== New for 2014 ===
* Added QBT-Extended dataset
* New subtask combining rhythm and pitch contours (QBT-Extended only)
* Minor modifications to command format - moving more toward a training/test paradigm, accommodates hidden test datasets
* Please make sure your list of candidates is ranked, as we will be assigning variable points for Top-10, 5, and 1 matches.

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <code><dbMidi.list></code> is the input list of database midi files named as <code>uniq_key.mid</code>. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <code><dir_workspace_root></code>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list_train> [dir_workspace_root]

Where <code><dbMidi_list></code> is a list of the MIDI files in the database to match against (see Step 0), and <code><query_file_list_train></code> maps each query to its associated ground truth. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <code><query_file_list_train></code> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are <code>.onset</code> and <code>.y_onset</code> files (for subtask 3), then the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same <code>.onset</code> file extension as previous symbolic input query datasets.

An <code>.onset</code> file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an <code>.onset</code> file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding <code>.onset</code> file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list_test> <result_file> [dir_workspace_root]

Where <code><query_file_list_test></code> is a single-column text file of input queries (the <code>.onset</code> files only, not the <code>.mid</code> files), and <code><result_file></code> is the filename where your script should store results. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

<code>query_file_list_test</code> thus has the following format:

qbtQuery/query_00001.onset
qbtQuery/query_00002.onset
qbtQuery/query_00003.onset
...

<code><result_file></code> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <code><result_file></code> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, qbt at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-08-28T05:22:08Z

Blair Kaneshiro: /* Step 0: Indexing the MIDIs collection */

== Overview ==
The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Submission deadline is September 16.

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

=== New for 2014 ===
* Added QBT-Extended dataset
* New subtask combining rhythm and pitch contours (QBT-Extended only)
* Minor modifications to command format - moving more toward a training/test paradigm, accommodates hidden test datasets
* Please make sure your list of candidates is ranked, as we will be assigning variable points for Top-10, 5, and 1 matches.

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <code><dbMidi.list></code> is the input list of database midi files named as <code>uniq_key.mid</code>. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <code><dir_workspace_root></code>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <code><dbMidi_list></code> is a list of the MIDI files in the database to match against (see Step 0), and <code><query_file_list></code> maps each query to its associated ground truth. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are <code>.onset</code> and <code>.y_onset</code> files (for subtask 3), then the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same <code>.onset</code> file extension as previous symbolic input query datasets.

An <code>.onset</code> file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an <code>.onset</code> file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding <code>.onset</code> file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <result_file> [dir_workspace_root]

Where <code><query_file_list></code> are input queries, and <code><result_file></code> is the filename where your script should store results. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

<code><result_file></code> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <code><result_file></code> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, qbt at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-08-28T05:20:37Z

Blair Kaneshiro: /* Step 2: Testing */

== Overview ==
The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Submission deadline is September 16.

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

=== New for 2014 ===
* Added QBT-Extended dataset
* New subtask combining rhythm and pitch contours (QBT-Extended only)
* Minor modifications to command format - moving more toward a training/test paradigm, accommodates hidden test datasets
* Please make sure your list of candidates is ranked, as we will be assigning variable points for Top-10, 5, and 1 matches.

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <code><dir_workspace_root></code>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <code><dbMidi_list></code> is a list of the MIDI files in the database to match against (see Step 0), and <code><query_file_list></code> maps each query to its associated ground truth. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are <code>.onset</code> and <code>.y_onset</code> files (for subtask 3), then the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same <code>.onset</code> file extension as previous symbolic input query datasets.

An <code>.onset</code> file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an <code>.onset</code> file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding <code>.onset</code> file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <result_file> [dir_workspace_root]

Where <code><query_file_list></code> are input queries, and <code><result_file></code> is the filename where your script should store results. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

<code><result_file></code> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <code><result_file></code> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, qbt at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-08-28T02:12:31Z

Blair Kaneshiro: /* Onset files format */

== Overview ==
The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Submission deadline is September 16.

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

=== New for 2014 ===
* Added QBT-Extended dataset
* New subtask combining rhythm and pitch contours (QBT-Extended only)
* Minor modifications to command format - moving more toward a training/test paradigm, accommodates hidden test datasets
* Please make sure your list of candidates is ranked, as we will be assigning variable points for Top-10, 5, and 1 matches.

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <code><dir_workspace_root></code>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <code><dbMidi_list></code> is a list of the MIDI files in the database to match against (see Step 0), and <code><query_file_list></code> maps each query to its associated ground truth. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are <code>.onset</code> and <code>.y_onset</code> files (for subtask 3), then the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same <code>.onset</code> file extension as previous symbolic input query datasets.

An <code>.onset</code> file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an <code>.onset</code> file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding <code>.onset</code> file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <resultFile> [dir_workspace_root]

Where <code><query_file_list></code> are input queries, and <code><resultFile></code> is the filename where your script should store results. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

<code><resultFile></code> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <code><resultFile></code> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, qbt at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-08-28T02:11:41Z

Blair Kaneshiro: /* Step 2: Testing */

== Overview ==
The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Submission deadline is September 16.

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

=== New for 2014 ===
* Added QBT-Extended dataset
* New subtask combining rhythm and pitch contours (QBT-Extended only)
* Minor modifications to command format - moving more toward a training/test paradigm, accommodates hidden test datasets
* Please make sure your list of candidates is ranked, as we will be assigning variable points for Top-10, 5, and 1 matches.

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <code><dir_workspace_root></code>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <code><dbMidi_list></code> is a list of the MIDI files in the database to match against (see Step 0), and <code><query_file_list></code> maps each query to its associated ground truth. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are <code>.onset</code> and <code>.y_onset</code> files (for subtask 3), then the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same <code>.onset</code> file extension as previous symbolic input query datasets.

An <code>.onset</code> file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding <code>.onset</code> file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <resultFile> [dir_workspace_root]

Where <code><query_file_list></code> are input queries, and <code><resultFile></code> is the filename where your script should store results. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

<code><resultFile></code> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <code><resultFile></code> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, qbt at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-08-28T02:09:53Z

Blair Kaneshiro: /* Step 1: Training */

== Overview ==
The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Submission deadline is September 16.

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

=== New for 2014 ===
* Added QBT-Extended dataset
* New subtask combining rhythm and pitch contours (QBT-Extended only)
* Minor modifications to command format - moving more toward a training/test paradigm, accommodates hidden test datasets
* Please make sure your list of candidates is ranked, as we will be assigning variable points for Top-10, 5, and 1 matches.

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <code><dir_workspace_root></code>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <code><dbMidi_list></code> is a list of the MIDI files in the database to match against (see Step 0), and <code><query_file_list></code> maps each query to its associated ground truth. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are <code>.onset</code> and <code>.y_onset</code> files (for subtask 3), then the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same <code>.onset</code> file extension as previous symbolic input query datasets.

An <code>.onset</code> file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding <code>.onset</code> file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, qbt at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-08-28T02:05:28Z

Blair Kaneshiro: /* Step 0: Indexing the MIDIs collection */

== Overview ==
The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Submission deadline is September 16.

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

=== New for 2014 ===
* Added QBT-Extended dataset
* New subtask combining rhythm and pitch contours (QBT-Extended only)
* Minor modifications to command format - moving more toward a training/test paradigm, accommodates hidden test datasets
* Please make sure your list of candidates is ranked, as we will be assigning variable points for Top-10, 5, and 1 matches.

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <code><dir_workspace_root></code>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are .onset and .y_onset files (for subtask 3), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, qbt at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-17T22:11:23Z

Blair Kaneshiro: /* Overview */

== Overview ==
The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Submission deadline is September 16.

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

=== New for 2014 ===
* Added QBT-Extended dataset
* New subtask combining rhythm and pitch contours (QBT-Extended only)
* Minor modifications to command format - moving more toward a training/test paradigm, accommodates hidden test datasets
* Please make sure your list of candidates is ranked, as we will be assigning variable points for Top-10, 5, and 1 matches.

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are .onset and .y_onset files (for subtask 3), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, qbt at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-17T22:11:08Z

Blair Kaneshiro: /* Discussions for 2014 */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Submission deadline is September 16.

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

=== New for 2014 ===
* Added QBT-Extended dataset
* New subtask combining rhythm and pitch contours (QBT-Extended only)
* Minor modifications to command format - moving more toward a training/test paradigm, accommodates hidden test datasets
* Please make sure your list of candidates is ranked, as we will be assigning variable points for Top-10, 5, and 1 matches.

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are .onset and .y_onset files (for subtask 3), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, qbt at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-15T00:25:49Z

Blair Kaneshiro: /* Potential Participants */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are .onset and .y_onset files (for subtask 3), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, qbt at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-15T00:24:00Z

Blair Kaneshiro: /* Per-task input specification */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are .onset and .y_onset files (for subtask 3), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-15T00:23:01Z

Blair Kaneshiro: /* Per-task input specification */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are .onset and .y_onset files (for subtask 3), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset-files-format|Onset files format]]


===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-15T00:22:37Z

Blair Kaneshiro: /* Per-task input specification */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset-files-format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are .onset and .y_onset files (for subtask 3), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset-files-format|Onset files format]]


===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-15T00:19:39Z

Blair Kaneshiro: /* Step 2: Testing */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset-files-format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are .onset and .y_onset files (for subtask 3), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset-files-format|Onset files format]]


===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-15T00:19:02Z

Blair Kaneshiro: /* Per-task input specification */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset-files-format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are .onset and .y_onset files (for subtask 3), then the format of <query_file_list> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset-files-format|Onset files format]]


===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-15T00:12:44Z

Blair Kaneshiro: /* Step 2: Testing */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtasks 1 and 3), then the format of <query_file_list> is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset-files-format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...


===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-15T00:12:26Z

Blair Kaneshiro: /* Step 1: Training */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtasks 1 and 3), then the format of <query_file_list> is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset-files-format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...


===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtProgram <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-15T00:10:29Z

Blair Kaneshiro: /* Step 1: Training */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtProgram <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtasks 1 and 3), then the format of <query_file_list> is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset-files-format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...


===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtProgram <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-15T00:09:12Z

Blair Kaneshiro: /* Step 1: Training */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtProgram <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtasks 1 and 3), then the format of <query_file_list> is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset-files-format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtProgram <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-15T00:04:51Z

Blair Kaneshiro: /* Per-task input specification */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtProgram <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtasks 1 and 3), then the format of <query_file_list> is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

An .onset file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an .onset file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding .onset file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same .onset file extension as previous symbolic input query datasets.

=== Step 2: Testing ===
The command format should be like this:

qbtProgram <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-14T23:49:16Z

Blair Kaneshiro: /* Step 1: Test the query files */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Training ===

The command format should be like this:

qbtProgram <dbMidi_list> <query_file_list> [dir_workspace_root]

Where <dbMidi_list> is a list of the MIDI files in the database to match against (see Step 0), and <query_file_list> maps each query to its associated ground truth. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <query_file_list> is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

(Please refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)

If the input query files are wave files (for subtask 2), the the format of <query_file_list> is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are 2-dimensional onset files (for subtask 3), then the format of <query_file_list> is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

=== Step 2: Testing ===
The command format should be like this:

qbtProgram <query_file_list> <resultFile> [dir_workspace_root]

Where <query_file_list> are input queries, and <resultFile> is the filename where your script should store results. You can use [dir_workspace_root] to store any temporary indexing/database structures. (You can omit [dir_workspace_root] if you do not need it at all.)

<resultFile> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <resultFile> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-14T23:22:54Z

Blair Kaneshiro: /* Test the query files */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Step 1: Test the query files ===

The command format should be like this:

qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%

You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

(Please refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)

If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-14T23:22:33Z

Blair Kaneshiro: /* Step 0: Indexing the MIDIs collection */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <dbMidi.list> is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <dir_workspace_root>.

=== Test the query files ===
The command format should be like this:

qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%

You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

(Please refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)

If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-14T23:20:34Z

Blair Kaneshiro: /* Step 0: Indexing the MIDIs collection */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into [dir_workspace_root].

=== Test the query files ===
The command format should be like this:

qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%

You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

(Please refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)

If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-14T23:19:08Z

Blair Kaneshiro: /* Step 0: Indexing the MIDIs collection */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing %dbMidi.list% %dir_workspace_root%

where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into %dir_workspace_root%.

=== Test the query files ===
The command format should be like this:

qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%

You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

(Please refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)

If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-14T23:18:36Z

Blair Kaneshiro: /* Indexing the MIDIs collection */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the formats below. Note that this step is not required unless you want to index or preprocess the MIDI database.

Command format should look like this:

indexing %dbMidi.list% %dir_workspace_root%

where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into %dir_workspace_root%.

=== Test the query files ===
The command format should be like this:

qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%

You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

(Please refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)

If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-14T23:15:53Z

Blair Kaneshiro: /* Indexing the MIDIs collection */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the formats below. Note that this step is not required unless you want to index or preprocess the MIDI database.

Command format should look like this:

indexing %dbMidi.list% %dir_workspace_root%

where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into %dir_workspace_root%.

=== Test the query files ===
The command format should be like this:

qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%

You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

(Please refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)

If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-01T22:42:10Z

Blair Kaneshiro: /* Discussions for 2014 */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==
CCRMA is very excited to be hosting the QBT task this year!

Any questions or suggestions can be added directly here, or you can send us an email, qbt | at | ccrma dot stanford dot edu

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Indexing the MIDIs collection ===
Command format should look like this:

indexing %dbMidi.list% %dir_workspace_root%

where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into %dir_workspace_root%. (Note that this step is not required unless you want to index or preprocess the midi database.)

=== Test the query files ===
The command format should be like this:

qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%

You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

(Please refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)

If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-01T20:57:47Z

Blair Kaneshiro: /* References */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Indexing the MIDIs collection ===
Command format should look like this:

indexing %dbMidi.list% %dir_workspace_root%

where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into %dir_workspace_root%. (Note that this step is not required unless you want to index or preprocess the midi database.)

=== Test the query files ===
The command format should be like this:

qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%

You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

(Please refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)

If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-01T03:47:58Z

Blair Kaneshiro: /* References */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Indexing the MIDIs collection ===
Command format should look like this:

indexing %dbMidi.list% %dir_workspace_root%

where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into %dir_workspace_root%. (Note that this step is not required unless you want to index or preprocess the midi database.)

=== Test the query files ===
The command format should be like this:

qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%

You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

(Please refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)

If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-01T03:46:37Z

Blair Kaneshiro: /* References */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Indexing the MIDIs collection ===
Command format should look like this:

indexing %dbMidi.list% %dir_workspace_root%

where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into %dir_workspace_root%. (Note that this step is not required unless you want to index or preprocess the midi database.)

=== Test the query files ===
The command format should be like this:

qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%

You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

(Please refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)

If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-01T03:36:53Z

Blair Kaneshiro: /* References */

== Overview ==
The text of this section is copied from the 2013 page. Please add your comments and discussions for 2014.

The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2014 ==

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Indexing the MIDIs collection ===
Command format should look like this:

indexing %dbMidi.list% %dir_workspace_root%

where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into %dir_workspace_root%. (Note that this step is not required unless you want to index or preprocess the midi database.)

=== Test the query files ===
The command format should be like this:

qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%

You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this:

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

(Please refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)

If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this:

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==
Jorge Herrera, Hyung-Suk Kim, and Blair Kaneshiro, CCRMA, jorgeh at ccrma dot stanford dot edu

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBand - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Hsiao SJS, Liant T, and Ke HR (2008). A simple algorithm for rhythm similarity comparison. MIREX.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2014:Query by Tapping

2014-07-01T03:23:03Z

Blair Kaneshiro:

2014:Query by Tapping

2014-05-08T19:24:27Z

Blair Kaneshiro: /* Test the query files */

2014:Query by Tapping

2014-05-08T19:24:03Z

Blair Kaneshiro:

2014:Query by Tapping

2014-05-08T19:22:34Z

Blair Kaneshiro: /* Potential Participants */

2014:Query by Tapping

2014-05-08T19:18:15Z

Blair Kaneshiro: /* Subtask 3: QBT-Extended with symbolic input (under construction) */

2014:Query by Tapping

2014-05-08T19:17:36Z

Blair Kaneshiro: /* Overview */

2014:Query by Tapping

2014-05-08T19:16:35Z

Blair Kaneshiro: /* Subtask 3: QBT-Extended with symbolic input (under construction) */

2014:Query by Tapping

2014-05-08T19:08:50Z

Blair Kaneshiro: /* Subtask 3: QBT-Extended with symbolic input (under construction) */

2014:Query by Tapping

2014-05-08T19:05:27Z

Blair Kaneshiro: /* Task description */

2014:Query by Tapping

2014-05-08T19:02:55Z

Blair Kaneshiro: /* Task description */

2014:Query by Tapping

2014-05-08T18:59:20Z

Blair Kaneshiro: /* Overview */

2014:Query by Tapping

2014-05-08T18:58:57Z

Blair Kaneshiro: /* Overview */

2014:Query by Tapping

2014-05-08T18:58:37Z

Blair Kaneshiro: /* Task description */

2014:Query by Tapping

2014-05-08T18:58:00Z

Blair Kaneshiro: /* Subtask 1: QBT with symbolic input */

2014:Query by Tapping

2014-05-08T18:57:47Z

Blair Kaneshiro: /* Subtask 1: QBT with symbolic input */

2014:Query by Tapping

2014-05-08T18:57:07Z

Blair Kaneshiro: /* Subtask 1: QBT with symbolic input */

2014:Query by Tapping

2014-05-08T18:55:56Z

Blair Kaneshiro: /* Overview */