2010:Symbolic Music Similarity and Retrieval - Revision history

CameronNonAdmin: /* Data */

2010-12-16T21:25:44Z

‎Data

2010-05-26T23:02:49Z

‎Description

2010-05-26T23:02:21Z

‎Evaluation

2010-05-26T22:58:58Z

‎Submission Format

2010-05-26T22:57:05Z

2010-05-26T22:56:15Z

‎Submission Format

2010-05-26T22:55:58Z

‎Submission Format

2010-05-26T22:55:19Z

‎Submission Format

2010-05-26T22:54:35Z

‎Submission Format

2010-05-26T22:50:51Z

‎Evaluation

← Older revision		Revision as of 21:25, 16 December 2010
Line 23:		Line 23:
	* 5,274 tunes belonging to the Essen folksong collection. The tunes are in standard MIDI file format. [http://www.ldc.usb.ve/~cgomez/essen.tar.gz Download] (< 1 MB)		* 5,274 tunes belonging to the Essen folksong collection. The tunes are in standard MIDI file format. [http://www.ldc.usb.ve/~cgomez/essen.tar.gz Download] (< 1 MB)

		+	* [http://www.essaymill.com termpapers]

	==Evaluation ==		==Evaluation ==

@@ Line 5: / Line 5: @@
 Monophonic to monophonic. Both the query and the documents in the collection will be monophonic.
-Each system will be  given a query and returned the 10 most melodically similar songs from those taken from the Essen Collection (5274 pieces in the MIDI format; see [http://www.esac-data.org/  ESAC Data Homepage] for more information). For each query, we made four classes of error-mutations, thus the set comprises the following query classes:
+Each system will be  given a query and returned the 10 most melodically similar songs from those taken from the Essen Collection (5274 pieces in the MIDI format; see [http://www.esac-data.org/  ESAC Data Homepage] for more information). For of the 6 queries, we made four classes of error-mutations, thus the set comprises the following query classes:
 * 0. No errors

@@ Line 26: / Line 26: @@
 ==Evaluation ==
-The same method for building the ground truth as last year will be used. This method has the advantage that no ground truth needs to be built in advance. After the algorithms have been submitted, their results are pooled for every query, and human evaluators are asked to judge the relevance of the matches for some queries.
+The same method for building the ground truth as in the previous iterations in 2006 and 2007 will be used. This method has the advantage that no ground truth needs to be built in advance. After the algorithms have been submitted, their results are pooled for every query, and human evaluators are asked to judge the relevance of the matches for some queries.
 For each query (and its 4 mutations), the returned results (candidates) from all systems will  then grouped together (query set) for evaluation by the human graders. The graders will  provide with only heard perfect version against which to evaluate the candidates and did not know whether the candidates came from a perfect or mutated query. Each query/candidate set was evaluated by 1 individual grader. Using the Evalutron 6000 system, the graders gave each query/candidate pair two types of scores. Graders will be  asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 10).

@@ Line 38: / Line 38: @@
 - the name of a directory containing about 5,000 MIDI files containing monophonic folk songs and <br/>
 - the name of one MIDI file containing a monophonic query.
 The program will be called once for each query.

@@ Line 54: / Line 54: @@
   query2.mid,song5454.mid,song423.mid,song454.mid ...
 ...
-=== Packaging submissions ===
+All submissions should include a README file including the following the information:
-* Command line calling format for all executables and an example formatted set of commands
+* Command line calling format for all executables including examples
 * Number of threads/cores used or whether this should be specified on the command line
 * Expected memory footprint
 * Expected runtime
-* Any required environments (and versions), e.g. python, java, bash, matlab.
+* Approximately how much scratch disk space will the submission need to store any feature/cache files?

@@ Line 47: / Line 47: @@
 E.g.
   query1.mid song242.mid song213.mid song1242.mid ...
-  query1.mid song5454.mid song423.mid song454.mid ...
+  query2.mid song5454.mid song423.mid song454.mid ...
 ...
 E.g.
   query1.mid,song242.mid,song213.mid,song1242.mid ...
-  query1.mid,song5454.mid,song423.mid,song454.mid ...
+  query2.mid,song5454.mid,song423.mid,song454.mid ...
 ...

← Older revision		Revision as of 22:54, 26 May 2010
Line 44:		Line 44:

	A list of the names of the 10 most similar matching MIDI files, ordered by melodic similarity. Write the file name in separate lines, without empty lines in between.		A list of the names of the 10 most similar matching MIDI files, ordered by melodic similarity. Write the file name in separate lines, without empty lines in between.
		+
		+	E.g.
		+	query1.mid song242.mid song213.mid song1242.mid ...
		+	query1.mid song5454.mid song423.mid song454.mid
		+

	=== Packaging submissions ===		=== Packaging submissions ===

← Older revision		Revision as of 22:50, 26 May 2010
Line 27:		Line 27:

	The same method for building the ground truth as last year will be used. This method has the advantage that no ground truth needs to be built in advance. After the algorithms have been submitted, their results are pooled for every query, and human evaluators are asked to judge the relevance of the matches for some queries.		The same method for building the ground truth as last year will be used. This method has the advantage that no ground truth needs to be built in advance. After the algorithms have been submitted, their results are pooled for every query, and human evaluators are asked to judge the relevance of the matches for some queries.
		+
		+	For each query (and its 4 mutations), the returned results (candidates) from all systems will then grouped together (query set) for evaluation by the human graders. The graders will provide with only heard perfect version against which to evaluate the candidates and did not know whether the candidates came from a perfect or mutated query. Each query/candidate set was evaluated by 1 individual grader. Using the Evalutron 6000 system, the graders gave each query/candidate pair two types of scores. Graders will be asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 10).
		+
	== Submission Format ==		== Submission Format ==