2010:Symbolic Melodic Similarity - Revision history

Xiao: /* Evaluation */

2010-07-14T18:26:52Z

‎Evaluation

MertBay: /* Submission Format */

2010-06-25T19:37:05Z

‎Submission Format

MertBay: /* Submission Format */

2010-06-25T19:36:25Z

‎Submission Format

Kris West: /* Command line calling format */

2010-06-25T09:08:49Z

‎Command line calling format

Kris West: Modifying task calling formats and IO

2010-06-25T01:48:13Z

Modifying task calling formats and IO

Kriswest: /* Submission opening date */

2010-06-05T09:13:24Z

‎Submission opening date

Jdownie: /* Packaging submissions */

2010-06-04T03:50:30Z

‎Packaging submissions

Xiao: /* Output */

2010-05-29T04:57:58Z

‎Output

Kriswest: Light reorganisation and copy changes

2010-05-28T17:13:26Z

Light reorganisation and copy changes

MertBay: moved 2010:Symbolic Melodic Similarity and Retrieval to 2010:Symbolic Melodic Similarity: Consistency with past name convention

2010-05-26T23:53:01Z

moved 2010:Symbolic Melodic Similarity and Retrieval to 2010:Symbolic Melodic Similarity: Consistency with past name convention

@@ Line 19: / Line 19: @@
 ==Evaluation ==
-The 2010 SMS task replicates the 2007 task. After the algorithms have been submitted, their results will be pooled for every query, and human evaluators, using the Evalutron 6000 system, will asked to judge the relevance of the matches to the queries.
+The 2010 SMS task replicates the 2007 task. After the algorithms have been submitted, their results will be pooled for every query, and human evaluators, using the Evalutron 6000 system, will asked to judge the relevance of the matches to the queries.[[2010:Evalutron6000_Walkthrough |2010 Evalutron6000 Walkthrough Page]].
 For each query (and its four mutations), the returned results (candidates) from all systems will be anonymously grouped together (query set) for evaluation by the human graders. The graders will be provided with only the "base" perfect version against which to evaluate the candidates and thus did not know whether the candidates came from a perfect or mutated query. We expect that each query/candidate set will be evaluated by one individual grader. Using the Evalutron 6000 system, the graders will give each query/candidate pair two types of scores. Graders will be asked to provide one "BROAD" categorical score with three categories: NS,SS,VS as explained below, and one "FINE" score (in the range from 0 to 10).

@@ Line 31: / Line 31: @@
 === Output ===
-A list of the names of the 10 most similar matching MIDI files, ordered by melodic similarity output to the stdout.
+A list of the names of the 10 most similar matching MIDI files, ordered by melodic similarity output to the stdout. Please use stderr for the log and error messages.
 E.g.

@@ Line 27: / Line 27: @@
 == Submission Format ==
-=== I/O formats ===
+=== Input ===
-In this section the input and output files used in this task are described as are the command line calling format requirements for submissions.
+A path to the folder containing midi files.
-==== Collection and Query list files ====
+E.g.
-UTF8 text list file containing paths to the files to be used as the collection or as queries (one path per line).
+ song242.mid song213.mid song1242.mid ...
-e.g.
+Where
-=== Output file ===
+/path/indexFileOrDirectory/ - The path to the index file or folder that will be used in the query step.
 E.g.
-  query1.mid song242.mid song213.mid song1242.mid ...
+  doQuery.sh -numThreads 4(optional) /path/to/scratch/folder(optional) /path/to/indexFileOrFolder/ /path/to/query.mid
- query2.mid song5454.mid song423.mid song454.mid ...
- ...
+/path/to/indexFileOrFolder/ - Path to the index file or folder created in the first step.
-=== Command line calling format ===
+==== 1 Step Calling Format ====
-Parameters:<br/>
+If your program does not perform indexing.
 E.g.
-   doSMS.sh -numThreads 4 /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/queryFile.txt /path/to/outputResultsFile.txt
+ doSMS.sh -numThreads 4(optional) /path/to/scratch/folder /path/to/folder/withMIDIfiles/ /path/to/query.mid
-OR (if writing to standard out)
-   doSMS.sh -numThreads 4 /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/queryFile.txt > /path/to/outputResultsFile.txt
+The program will be called once for each query.
 === Packaging submissions ===

@@ Line 58: / Line 58: @@
 E.g.
     doSMS.sh -numThreads 4 /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/queryFile.txt /path/to/outputResultsFile.txt
 The program will be called only once to index the collection and perform the queries.
 === Packaging submissions ===

@@ Line 27: / Line 27: @@
 == Submission Format ==
-Parameters:<br/>
+==== Collection and Query list files ====
-- the name of a directory containing about 5,000 MIDI files containing monophonic folk songs and <br/>
+UTF8 text list file containing paths to the files to be used as the collection or as queries (one path per line).
-E.g.
+e.g.
- myAlgo.sh /path/to/folder/withMIDIfile/ /path/to/query.mid
+  /aDirectory/collectionFolder/b002342.wav
-The program will be called once for each query.
+E.g.
-=== Output ===
+=== Command line calling format ===
-A list of the names of the 10 most similar matching MIDI files, ordered by melodic similarity. Write the file name in separate lines, without empty lines in between.
+E.g.
 === Packaging submissions ===
 * All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).
 * Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
@@ Line 72: / Line 82: @@
 Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.
-A hard limit of 24 hours will be imposed on feature extraction times.
+A hard limit of 48 hours total runtime will be imposed on each submission. Submissions exceeding this runtime may not receive a result.

@@ Line 78: / Line 78: @@
 == Submission opening date ==
 == Submission closing date ==
 TBA

@@ Line 54: / Line 54: @@
 * All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).
-* Be sure to follow the [[2006:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
+* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
 * Be sure to follow the [[MIREX 2010 Submission Instructions]]
@@ Line 67: / Line 67: @@
 * Any special notice regarding to running your algorithm
 Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.
 == Time and hardware limits ==

@@ Line 1: / Line 1: @@
 == Description ==
 The goal of SMS is to retrieve the most similar items from a collection of symbolic pieces, given a symbolic query, and rank them by melodic similarity. There will be only 1 task this year which comprises a set of six "base" monophonic MIDI queries to be matched against a monophonic MIDI collection.
 Each system will be given a query and is asked to return the 10 most melodically similar songs from those taken from the Essen Collection (5274 pieces in the MIDI format; see [http://www.esac-data.org/  ESAC Data Homepage] for more information). For each of the six "base" queries, we have created four classes of error-mutations, thus the query set comprises the following query classes:
-* 0. No errors (i.e., "base")
+# No errors (i.e., "base")
-* 1. One note deleted
+# One note deleted
-* 2. One note inserted
+# One note inserted
-* 3. One interval enlarged
+# One interval enlarged
-* 4. One interval compressed
+# One interval compressed
 == Data ==
 * 5,274 tunes belonging to the Essen folksong collection. The tunes are in standard MIDI file format. [http://www.ldc.usb.ve/~cgomez/essen.tar.gz Download] (< 1 MB)
@@ Line 31: / Line 28: @@
-'''Input'''
+=== Input ===
 Parameters:<br/>
@@ Line 44: / Line 41: @@
 The program will be called once for each query.
-'''Expected output'''
 A list of the names of the 10 most similar matching MIDI files, ordered by melodic similarity. Write the file name in separate lines, without empty lines in between.
@@ Line 51: / Line 49: @@
   query1.mid song242.mid song213.mid song1242.mid ...
   query2.mid song5454.mid song423.mid song454.mid ...
-...
+ ...
 E.g.
   query1.mid,song242.mid,song213.mid,song1242.mid ...
   query2.mid,song5454.mid,song423.mid,song454.mid ...
-...
+ ...
 === Packaging submissions ===
@@ Line 75: / Line 75: @@
 Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.
-=== Time and hardware limits ===
 Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.
@@ Line 82: / Line 83: @@
 A hard limit of 48 hours will be imposed on the 3 training/classification cycles, leading to a total runtime limit of 72 hours for each submission.
 TBA
 TBA

← Older revision	Revision as of 23:53, 26 May 2010
(No difference)