2006:Score Following Proposal - Revision history

IMIRSELBot: Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

2010-05-13T18:05:45Z

Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

Xiao: /* Proposers */

2006-10-06T22:52:56Z

‎Proposers

Arshiacont: /* Reference Database */

2006-09-25T09:01:23Z

‎Reference Database

Arshiacont: /* Reference Database */

2006-09-25T08:39:48Z

‎Reference Database

Arshiacont: /* Evaluation metrics */

2006-09-25T08:32:53Z

‎Evaluation metrics

Diemo: /* Reference alignment */

2006-08-10T13:14:40Z

‎Reference alignment

Diemo: Remarks

2006-08-10T13:12:48Z

Remarks

Diemo: /* File formats */

2006-08-10T10:15:45Z

‎File formats

Diemo: /* Reference alignment */

2006-08-07T14:36:52Z

‎Reference alignment

Diemo: /* Suggested calling formats for submitted algorithms */

2006-08-07T14:34:34Z

‎Suggested calling formats for submitted algorithms

@@ Line 1: / Line 1: @@
 ==Results==
-Results are on [[Score Following Results ]] page.
+Results are on [[2006:Score Following Results ]] page.
 == Proposers ==
@@ Line 12: / Line 12: @@
 Score Following is the ''real-time'' alignment of incoming music signal to the music score. The music signal can be symbolic (Midi Score Following) or Audio.
-This page describes a proposal for evaluation of score following systems. Discussion of the evaluation procedures on the [https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com01 MIREX 06 "ScoreFollowing06" contest planning list] will be documented on the [[Score Following]] page. A full digest of the discussions is available to subscribers from the [https://mail.lis.uiuc.edu/mailman/private/mrx-com01/ MIREX 06 "ScoreFollowing06" contest planning list archives].
+This page describes a proposal for evaluation of score following systems. Discussion of the evaluation procedures on the [https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com01 MIREX 06 "ScoreFollowing06" contest planning list] will be documented on the [[2006:Score Following]] page. A full digest of the discussions is available to subscribers from the [https://mail.lis.uiuc.edu/mailman/private/mrx-com01/ MIREX 06 "ScoreFollowing06" contest planning list archives].
 Submissions will be required to estimate alignment precision according to the indexed times, type of alignment (monophonic, polyphonic), type of training and realtime performance, also separated into two domains (upon enough submissions) for symbolic and audio systems.

← Older revision		Revision as of 22:52, 6 October 2006
Line 1:		Line 1:
		+	==Results==
		+	Results are on [[Score Following Results ]] page.
		+
	== Proposers ==		== Proposers ==
	* Arshia Cont (University of California in San Diego (UCSD) and Ircam - Realtime Applications Team, France) - [mailto:cont@ircam.fr cont@ircam.fr]		* Arshia Cont (University of California in San Diego (UCSD) and Ircam - Realtime Applications Team, France) - [mailto:cont@ircam.fr cont@ircam.fr]

← Older revision		Revision as of 09:01, 25 September 2006
Line 122:		Line 122:

	All files were aligned by an external offline score alignment algorithm and corrected by hand.		All files were aligned by an external offline score alignment algorithm and corrected by hand.
		+
		+	=== Statistics ===
		+
		+	{\| border="1"
		+	\|+ MIREX06 Score Following Reference Database
		+	\|-
		+	!Piece name !! Composer !! Instrument !! Number of Files !! Number of score events !! Total duration
		+	\|-
		+	! ... Explosante-Fixe...
		+	\|Pierre Boulez
		+	\|Flute
		+	\|47
		+	\| 2022
		+	\| 17:10
		+	\|-
		+	! Violin Sonatas (excerpt)
		+	\|J.S. Bach
		+	\| Violin
		+	\| 3
		+	\| 3996
		+	\| 13:50
		+	\|-
		+	! Clarinet Quintet KV370
		+	\| Mozart
		+	\| Clarinet
		+	\| 4
		+	\| 2710
		+	\| 14:44
		+	\|-
		+	! Dorabella
		+	\| Mozart
		+	\| Voice
		+	\| 4
		+	\| 229
		+	\| 01:44
		+	\|- style="border-top:1px solid red; border-right:1px solid red; border-bottom:2px solid red; border-left:1px solid red; background:silver"
		+	! Total
		+	\|
		+	\|
		+	\| 58
		+	\| 8957
		+	\| 47:38
		+	\|}
		+
		+

	=== Content Format ===		=== Content Format ===

@@ Line 116: / Line 116: @@
 ** Mozart ''Dorabella'', voice
 ** Mozart ''Clarinet Concerto K370'', clarinet
 * Ircam (Arshia Cont and Diemo Schwarz)
 ** Boulez ''... Explosante-Fixe ...'', flute (47 files, duration approx. 1 hour)
-** Bach ''Violin Sonatas'', performed by Menuhin and Kremer (two sets, three sonatas)
+** Bach ''Violin Sonatas'', performed by Menuhin (Three movements)
-*** Aligned by an external offline score alignment algorithm and corrected by hand.
 === Content Format ===

@@ Line 59: / Line 59: @@
 === Evaluation metrics ===
-* Percentage of ''missed'' events.
+Evaluation is done by comparing the ''reference file'' with a ''result file'' for each audio in the database. Here are the criteria used for this year's evaluation:
-* Average ''Offset'' between the ''detected'' note onsets and reference alignment and statistics of it.
-* Average latency of the system.
+==== Missed Notes ====
 === Evaluator pseudo-code ===

@@ Line 106: / Line 106: @@
 # MIDI note number in score [nn]
-Note that more columns might be added to this definition in the future, so please program your reference file parser in a way that additional columns are ignored.
+Note that more columns might be added to this definition in the future, e.g. to mark trills, so please program your reference file parser in a way that additional columns don't confuse the program and are gracefully are ignored.
 ==== Example ====

@@ Line 54: / Line 54: @@
 # note start time in score (ms)
 # MIDI note number in score (int)
 === Evaluation metrics ===

@@ Line 45: / Line 45: @@
 === Output data ===
 ==== File formats ====
-ASCII output for each score following system as described below. column data should be separated by ''tab''s and rows by ''return''.
+ASCII output for each score following system as described below. column data should be separated by white space and rows by unix-newline '\n'.
 ==== Content ====

@@ Line 20: / Line 20: @@
 === Suggested calling formats for submitted algorithms ===
-During evaluation, each system will be called in '''command line''' with the following format:
+During evaluation, each system will be called in '''command line''' with probably something like the following format:
-:: <system-execution-file> <input-folder> <output-filename>
+:: <system-execution-file> <performance-audio-filename> <MIDI-score-filename> <result-filename> <log-filename>
-The input folder contains the score and audio performance of the score. Your submitted binaries should be able to BROWSE this folder and use the appropriate score and audio file and undertake the score following task, and write the results to the output file as given.
+N.B.: the previous calling format given here in July of <system-execution-file> <input-folder> <output-filename> is discouraged by the IMIRSEL team.
 It is important to be able to create the output ascii file in a "different" path than the default.
-In order to consider the issue of training, an alternative call format would be:
+In order to consider the issue of training and audio following, an alternative call format would be:
-:: <system-execution-file> <input-folder> <output-filename> <training-folder>
+:: <system-execution-file> <performance-audio-filename> <MIDI-score-filename> <audio-score-filename> <reference-alignment-filename> <result-filename> <log-filename>
-where the training folder contains appropriate files for training. Obviously, if this third argument is not given, it is assumed that there is no learning/training phase.
+The reference alignment file links the MIDI to the audio score.
 === Input data ===