2010:Audio Music Similarity and Retrieval - Revision history

Kriswest at 01:17, 19 June 2010

2010-06-19T01:17:07Z

Kriswest: Linking AMS eval software

2010-06-19T01:15:45Z

Linking AMS eval software

Heywhoah: Added link to 2009 page and results.

2010-06-05T13:22:06Z

Added link to 2009 page and results.

Kriswest: /* Submission opening date */

2010-06-05T09:13:01Z

‎Submission opening date

Jdownie: /* Task specific mailing list */

2010-06-04T01:27:34Z

‎Task specific mailing list

Xiao: /* Human Evaluation */

2010-06-03T16:40:19Z

‎Human Evaluation

Xiao: /* Human Evaluation */

2010-06-03T16:40:07Z

‎Human Evaluation

Xiao: /* Human Evaluation */

2010-06-03T16:39:34Z

‎Human Evaluation

Xiao: /* Description */

2010-06-03T16:35:34Z

‎Description

Xiao: /* Human Evaluation */

2010-06-03T16:28:09Z

‎Human Evaluation

@@ Line 215: / Line 215: @@
 * (pseudo-)objective statistical evaluation of distance matrices by comparing query metadata to the metadata of the top N results retrieved. Supports artist, album, genre and artist-filtered genre (where results form the same artist as query are skipped). Additionally, the number tracks never returned as results for all possible queries (orphans) and the largest hub (track similar to the most other tracks) are measured. Finally, the number of cases where the triangular inequality holds.
 * preparation and post processing of results for the IMIRSEL Evalutron 6k  human evaluation interface.
 == Submission opening date ==

← Older revision		Revision as of 01:15, 19 June 2010
Line 205:		Line 205:
	A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.		A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.

		+
		+	== AMS evaluation software ==
		+	The legacy software for performing various AMS related function is available [https://www.music-ir.org/mirex/results/2010/AMS_tools.zip here], [https://www.music-ir.org/mirex/results/2010/AMS_TOOLS_README%20.txt README file]. It maybe used to benchmark systems prior to submission and to check distance matrix file formats.
		+
		+	This tool set supports the following functions:
		+	* the import of collection metadata from a delimited text file (e.g. TAB or CSV)
		+	* the selection of a stratified random list of queries from the collection (i.e. an equal number of queries are chosen for each class of a particular metadata field, such as genre).
		+	* the generation of results from distance matrices based on a list of pre-chosen queries.
		+	* (pseudo-)objective statistical evaluation of distance matrices by comparing query metadata to the metadata of the top N results retrieved. Supports artist, album, genre and artist-filtered genre (where results form the same artist as query are skipped). Additionally, the number tracks never returned as results for all possible queries (orphans) and the largest hub (track similar to the most other tracks) are measured. Finally, the number of cases where the triangular inequality holds.
		+	* preparation and post processing of results for the IMIRSEL Evalutron 6k human evaluation interface.

	== Submission opening date ==		== Submission opening date ==

@@ Line 4: / Line 4: @@
 This page presents the Audio Music Similarity Evaluation, including the submission rules and formats. Additionally background information can be found here that should help explain some of the reasoning behind the approach taken in the evaluation. The intention of the Music Audio Search track is to evaluate music similarity searches (A music search engine that takes a single song as a query aka Query-by-example), not playlist generation or music recommendation.
-The Audio Music Similarity and Retrieval task has been run in MIREX 2007 and 2006.
+The Audio Music Similarity and Retrieval task has been run in MIREX 2009, 2007, and 2006.
 [[2007:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2007]] || [[2007:Audio_Music_Similarity_and_Retrieval_Results|Results]]

@@ Line 205: / Line 205: @@
 == Submission opening date ==
 == Submission closing date ==
 TBA

@@ Line 12: / Line 12: @@
 === Task specific mailing list ===
-A specific mailing list is provided for the discussion of this task and related tasks ( [[2010:Audio Classification (Test/Train) tasks]], [[2010:Audio_Cover_Song_Identification]], [[2010:Audio_Tag_Classification]], [[2010:Audio_Music_Similarity_and_Retrieval]]) at: [https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com00 https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com00]. If you wish to participate in any of these tasks please sign up to this mailing listas discussion of the task format and evaluation should be conducted there.
+In the past we have use a specific mailing list for the discussion of this task and related tasks (e.g., [[2010:Audio Classification (Train/Test) Tasks]], [[2010:Audio Cover Song Identification]], [[2010:Audio Tag Classification]], [[2010:Audio Music Similarity and Retrieval]]). This year, however, we are asking that all discussions take place on the MIREX  [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.
 == Data ==

@@ Line 56: / Line 56: @@
 * The songs by the same artist as the query will be filtered out of each result list (artist-filtering) to avoid colouring an evaluators judgement (a cover song or song by the same artist in a result list is likely to reduce the relative ranking of other similar but independent songs - use of songs by the same artist may allow over-fitting to affect the results)
 * It will be possible for researchers to use this data for other types of system comparisons after MIREX 2010 results have been finalized.
-* Human evaluation to be designed and led by IMIRSEL following a similar format to that used at MIREX 2006 (see [[2006:Evalutron6000_Issues|Evalutron Issues in MIREX 2006]].
+* Human evaluation to be designed and led by IMIRSEL following a similar format to that used at MIREX 2006 (see: [[2006:Evalutron6000_Issues|Evalutron Issues in MIREX 2006]]).
 * Human evaluators will be drawn from the participating labs (and any volunteers from IMIRSEL or on the MIREX lists)