2006:Score Following - Revision history

IMIRSELBot: Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

2010-05-13T18:12:27Z

Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

IMIRSELBot: Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

2010-05-13T18:05:35Z

Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

Arshiacont: /* Naming/File conventions */

2006-07-19T21:06:57Z

‎Naming/File conventions

Arshiacont: /* Important threads on the discussion list */

2006-07-19T21:05:16Z

‎Important threads on the discussion list

Arshiacont at 18:47, 19 July 2006

2006-07-19T18:47:33Z

Arshiacont: /* Diemo Schwarz */

2006-07-19T18:46:34Z

‎Diemo Schwarz

Arshiacont: /* Acceptable Systems */

2006-07-19T18:45:53Z

‎Acceptable Systems

Arshiacont: /* Input/Output types */

2006-07-19T18:44:24Z

‎Input/Output types

Diemo: put refs also

2006-07-17T15:03:14Z

put refs also

Diemo: links to articles

2006-07-17T14:46:55Z

links to articles

@@ Line 2: / Line 2: @@
 This page is devoted to discussions of the evaluation of Score Following algorithms at MIREX 2006. Discussions have already begun on the https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com01 MIREX 06 "ScoreFollowing06" contest planning list] and will be briefly digested here. A full digest of the discussions is available to subscribers from the https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com01 MIREX 06 "ScoreFollowing06" contest planning archive].
-As consensus is achieved on the planning list, a full proposal ([[2006:Score Following Proposal]]) will be produced for the format of the evaluation, including pseudocode for the evaluation metric and suggested formats for submitted algorithms. A skeleton of proposal is already available on the [[Score Following Proposal]] page.
+As consensus is achieved on the planning list, a full proposal ([[2006:Score Following Proposal]]) will be produced for the format of the evaluation, including pseudocode for the evaluation metric and suggested formats for submitted algorithms. A skeleton of proposal is already available on the [[2006:Score Following Proposal]] page.
 == Moderators ==

@@ Line 2: / Line 2: @@
 This page is devoted to discussions of the evaluation of Score Following algorithms at MIREX 2006. Discussions have already begun on the https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com01 MIREX 06 "ScoreFollowing06" contest planning list] and will be briefly digested here. A full digest of the discussions is available to subscribers from the https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com01 MIREX 06 "ScoreFollowing06" contest planning archive].
-As consensus is achieved on the planning list, a full proposal ([[Score Following Proposal]]) will be produced for the format of the evaluation, including pseudocode for the evaluation metric and suggested formats for submitted algorithms. A skeleton of proposal is already available on the [[Score Following Proposal]] page.
+As consensus is achieved on the planning list, a full proposal ([[2006:Score Following Proposal]]) will be produced for the format of the evaluation, including pseudocode for the evaluation metric and suggested formats for submitted algorithms. A skeleton of proposal is already available on the [[Score Following Proposal]] page.
 == Moderators ==

← Older revision		Revision as of 21:06, 19 July 2006
Line 192:		Line 192:

	ISSUE: How are we going to define classes such as TRILLS (which should usually be considered as a separate object -- not defined in MIDI)?		ISSUE: How are we going to define classes such as TRILLS (which should usually be considered as a separate object -- not defined in MIDI)?
−
−	~~TODO: Finalize the standardization of the output (time) format.~~

	===== System call format =====		===== System call format =====

@@ Line 14: / Line 14: @@
 == Important threads on the discussion list ==
 Contributors:
-* Christopher Raphael
+* Christopher Raphael [CR]
-* Arshia Cont
+* Arshia Cont [AC]
-* Diemo Schwarz
+* Diemo Schwarz [DS]
-* George Litterst
+* George Litterst  [GL]
-* Kris West
+* Kris West  [KW]
 * Arie Van Schuetterhoef
 * Michael Good
@@ Line 214: / Line 214: @@
 Participating systems are on different platforms and might need a wrapper to prepare audio/score for the evaluation task of their systems.
 At this stage, it was *suggested* by Arshia Cont to simulate ''streaming'' to assure the causality of the submitted system. In this sense, the wrapper sends audio frames into the system (and not the whole audio). Ofcourse, we have to study the feasability of this issue due to time constraints of this year's task.
 == Related Papers ==

← Older revision		Revision as of 18:47, 19 July 2006
Line 216:		Line 216:
	At this stage, it was suggested by Arshia Cont to simulate ''streaming'' to assure the causality of the submitted system. In this sense, the wrapper sends audio frames into the system (and not the whole audio). Ofcourse, we have to study the feasability of this issue due to time constraints of this year's task.		At this stage, it was suggested by Arshia Cont to simulate ''streaming'' to assure the causality of the submitted system. In this sense, the wrapper sends audio frames into the system (and not the whole audio). Ofcourse, we have to study the feasability of this issue due to time constraints of this year's task.

−	~~== Input/Output types ==~~
−
−	~~=== Streaming simulation ===~~
−
−	~~== Evaluation Metrics ==~~

	== Related Papers ==		== Related Papers ==

@@ Line 133: / Line 133: @@
 ===== Diemo Schwarz =====
-To obtain ground truth reference alignments, we can work with the offline polyphonic alignment system described in [3].  We have a growing collection of contemporary music pieces, I will post an inventory soon.
+To obtain ground truth reference alignments, we can work with the offline polyphonic alignment system described in [3].
 === Input/Output types ===

@@ Line 24: / Line 24: @@
 === Acceptable Systems ===
-===== Arshia Cont =====
-I think we need to distinguish between realtime and offline alignment. Offline audio to score alignment can itself be a separate subject for evaluation. For this evaluation session, we are focusing on REALTIME audio to score alignment or score following. The main issue here is causality and ofcourse computation time. Offline audio to score alignment systems are mostly non-causal and use future data for alignment. It would not be fare to compare online and offline systems in one session.
+We need to distinguish between realtime and offline alignment. Offline audio to score alignment can itself be a separate subject for evaluation. For this evaluation session, we are focusing on REALTIME audio to score alignment or score following. The main issue here is causality and ofcourse computation time. Offline audio to score alignment systems are mostly non-causal and use future data for alignment. It would not be fare to compare online and offline systems in one session [AC].
 ===== Christopher Raphael =====

@@ Line 136: / Line 136: @@
 === Input/Output types ===
-===== Kris West =====
+We should come up with a common way of outputing data. An easy way would be to send ASCII markers to the score event + timing (in milliseconds) [AC]. The evaluation framework would have to execute your binary/script, collect the ASCII outputs and log the time the signal was received [KW]. It was suggested that we might forget about the actual real-time component of the problem and just examine "online" (= no lookahead) systems.  In this case I think system needs only to output two times (samples) for each note:
-===== Christopher Raphael =====
+# the time the system believes the note begins
-Someone suggested that we might forget about the actual real-time component
+# the time at which the above estimate was arrived at [CR]
-) the time the system believes the note begins
+Based on this, it was proposed by [AC] and [DS] to format the reference as well as the output of systems as follows:
-) the time at which the above estimate was arrived at
 === Issue of Training ===

@@ Line 83: / Line 83: @@
 ===== Diemo Schwarz =====
-Regarding what to measure, I suggest to take a look at the discussion in our [http://mediatheque.ircam.fr/articles/textes/Orio03a NIME 2003 article], section 4 on evaluation, expanded in [http://www.ircam.fr/anasyn/schwarz/thesis/ my thesis], section 5.6 "Evaluation of Alignment", page 44ff.
+Regarding what to measure, I suggest to take a look at the discussion in our [http://mediatheque.ircam.fr/articles/textes/Orio03a NIME 2003 article] [1], section 4 on evaluation, expanded in [http://www.ircam.fr/anasyn/schwarz/thesis/ my thesis] [2], section 5.6 "Evaluation of Alignment", page 44ff.
 I think Christopher's definition of latency is appropriate for the application of automatic accompaniment (with tempo estimation).  The other main application is real-time score recognition, where latency is the most important quality indicator.  So we define it as the time difference between the reference alignment mark and the detected event onset (note or rest).