2010:Query by Singing/Humming - Revision history

MertBay: /* Participants */

2010-05-26T21:25:32Z

‎Participants

MertBay: /* Submission Format */

2010-05-26T21:25:05Z

‎Submission Format

MertBay: /* Command Format II: Integrated Version */

2010-05-26T21:24:35Z

‎Command Format II: Integrated Version

MertBay: /* Command Format I: Breakdown Version */

2010-05-26T21:24:15Z

‎Command Format I: Breakdown Version

MertBay: /* Description */

2010-05-26T21:23:41Z

‎Description

MertBay: /* Task Description */

2010-05-26T21:23:28Z

‎Task Description

MertBay: /* Description */

2010-05-26T21:22:14Z

‎Description

MertBay: /* Corpora (Queries and Database) */

2010-05-26T21:21:39Z

‎Corpora (Queries and Database)

MertBay: /* Discussions for 2010 */

2010-05-26T21:21:23Z

‎Discussions for 2010

MertBay: /* Discussions from 2009 */

2010-05-26T21:21:14Z

‎Discussions from 2009

@@ Line 115: / Line 115: @@
 You can use %dir_workspace_root% to store any temporary indexing/database structures. The result file should have the same format as mentioned previously. (For task 2, %dbMidi.list% is in fact a list of wav files in the database to be retrieved.)
-== Participants ==
+=== Packaging submissions ===
-If you think there is a slight chance that you might want to participate, please add your name and e-mail address to this list
+All submissions should be statically linked to all libraries (the presence of
-Tzu-chun Yeh, Yi-Fan Fang (Tzu-chun.Yeh at mirlab dot org)
+All submissions should include a README file including the following the

← Older revision		Revision as of 21:25, 26 May 2010
Line 51:		Line 51:
	== Submission Format ==		== Submission Format ==

		+	=== Breakdown Version ===
	The following was based on the suggestion by Xiao Wu last year with some modifications.		The following was based on the suggestion by Xiao Wu last year with some modifications.

@@ Line 107: / Line 107: @@
   ...
-== Command Format II: Integrated Version ==
+=== Integrated Version ===
 If you want to pack everything together, the command format should be much simpler:

@@ Line 49: / Line 49: @@
 If there are not enough user-contributed corpora, then we shall adopt "no hidden dataset" policy for QBSH task as usual.
-== Command Format I: Breakdown Version ==
+== Submission Format ==
 The following was based on the suggestion by Xiao Wu last year with some modifications.

@@ Line 10: / Line 10: @@
 * [[2007:Query_by_Singing/Humming]]
 * [[2006:QBSH:_Query-by-Singing/Humming]]
 == Data ==

← Older revision		Revision as of 21:23, 26 May 2010
Line 30:		Line 30:

	If there are not enough user-contributed corpora, then we shall adopt "no hidden dataset" policy for QBSH task as usual.		If there are not enough user-contributed corpora, then we shall adopt "no hidden dataset" policy for QBSH task as usual.
−
−	~~== Task Description ==~~
−
−	~~=== Subtask 1: Classic QBSH evaluation ===~~
−	~~This is the classic QBSH problem where we need to find the ground-truth midi from a user's singing or humming.~~
−	* '''Queries''': human singing/humming snippets (.wav). Queries are from Roger Jang's corpus and ThinkIT corpus.
−	* '''Database''': ground-truth and noise MIDI files(which are monophonic). Comprised of 48+106 Roger Jang's and ThinkIT's ground-truth along with a cleaned version of Essen Database(2000+ MIDIs which are used last year)
−	* '''Output''': top-10 candidate list.
−	* '''Evaluation''': Top-10 hit rate (1 point is scored for a hit in the top 10 and 0 is scored otherwise).
−
−	~~=== Subtask 2: Variants QBSH evaluation ===~~
−	This is based on Prof. Downie's idea that queries are variants of "ground-truth" midi. In fact, this becomes more important since user-contributed singing/humming is an important part of the song database to be searched, as evidenced by the QBSH search service at [http://www.midomi.com/ www.midomi.com].
−	* '''Queries''': human singing/humming snippets (.wav). Queries are from Roger Jang's corpus and ThinkIT corpus.
−	* '''Database''': human singing/humming snippets (.wav) from all available corpora (excluding the query input being searched).
−	* '''Output''': top-10 candidate list.
−	* '''Evaluation''': Top-10 hit rate (1 point is scored for a hit in the top 10 and 0 is scored otherwise).
−
−	To make algorithms able to share intermediate steps, participants are encouraged to submit separate tracker and matcher modules instead of integrated ones, which is according to Rainer Typke's suggestion. So trackers and matchers from different submissions could work together with the same pre-defined interface and thus for us it's possible to find the best combination.

	== Command Format I: Breakdown Version ==		== Command Format I: Breakdown Version ==

← Older revision		Revision as of 21:22, 26 May 2010
Line 11:		Line 11:
	* [[2006:QBSH:_Query-by-Singing/Humming]]		* [[2006:QBSH:_Query-by-Singing/Humming]]

		+	== Data ==
		+	Currently we have 3 publicly available corpora for QBSH:
		+
		+	* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBSH-corpus.rar MIR-QBSH corpus] which is comprised of 4431 queries along with 48 ground-truth MIDI files. All queries are from the beginning of references. Manually labeled pitch for each recording is available.
		+
		+	* [http://mirlab.org/dataSet/public/IOACAS_QBH_Coprus1.rar IOACAS corpus 1] comprised of 355 queries and 106 monophonic ground-truth MIDI files (with MIDI 0 or 1 format). There are no "singing from beginning" gurantee.
		+
		+	* [http://mirlab.org/dataSet/public/IOACAS_QBH_Coprus2.rar IOACAS corpus 2] comprised of 404 queries and 192 monophonic ground-truth MIDI files. There are no "singing from beginning" gurantee.
		+
		+	The noise MIDI will be the 5000+ Essen collection(can be accessed from http://www.esac-data.org/).
		+
		+	To build a large test set which can reflect real-world queries, it is suggested that every participant makes a contribution to the evaluation corpus. Sometimes this is hard in practice. So we shall adopt "no hidden dataset" policy if there are not enough user-contributed copora.

	== Evaluation Corpus Contribution ==		== Evaluation Corpus Contribution ==

← Older revision		Revision as of 21:21, 26 May 2010
Line 11:		Line 11:
	* [[2006:QBSH:_Query-by-Singing/Humming]]		* [[2006:QBSH:_Query-by-Singing/Humming]]

−
−	~~== Corpora (Queries and Database) ==~~
−	~~Currently we have 3 publicly available corpora for QBSH:~~
−
−	* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBSH-corpus.rar MIR-QBSH corpus] which is comprised of 4431 queries along with 48 ground-truth MIDI files. All queries are from the beginning of references. Manually labeled pitch for each recording is available.
−
−	* [http://mirlab.org/dataSet/public/IOACAS_QBH_Coprus1.rar IOACAS corpus 1] comprised of 355 queries and 106 monophonic ground-truth MIDI files (with MIDI 0 or 1 format). There are no "singing from beginning" gurantee.
−
−	* [http://mirlab.org/dataSet/public/IOACAS_QBH_Coprus2.rar IOACAS corpus 2] comprised of 404 queries and 192 monophonic ground-truth MIDI files. There are no "singing from beginning" gurantee.
−
−	~~The noise MIDI will be the 5000+ Essen collection(can be accessed from http://www.esac-data.org/).~~
−
−	To build a large test set which can reflect real-world queries, it is suggested that every participant makes a contribution to the evaluation corpus. Sometimes this is hard in practice. So we shall adopt "no hidden dataset" policy if there are not enough user-contributed copora.

	== Evaluation Corpus Contribution ==		== Evaluation Corpus Contribution ==

← Older revision		Revision as of 21:21, 26 May 2010
Line 17:		Line 17:

	Please feel free to edit this page.		Please feel free to edit this page.
−
−	~~== Discussions from 2009 ==~~
−
−	~~=== Roger Jang's Comments 04/09/2009 ===~~
−	~~I would like to suggest to extend the submission deadline to Sept 15. There are two reasons for this:~~
−	* I have prepare another QBSH dataset which is bigger and more balanced. But I need some time to tidy things up during the weekend. I wish to make it available for the QBSH task this year.
−	* I propose two subtasks which are the same as what we had during MIREX 2006. Please see "Task Description" for details.

	== Corpora (Queries and Database) ==		== Corpora (Queries and Database) ==