MIREX Wiki - User contributions [en]

2010:MIREX 2010 Poster List

2010-08-01T21:31:54Z

Proton: /* Add your author names here, once for each poster along with "title of some sort" and (Task(s) covered) */

==MIREX 2010 Poster Session Planning List==
The MIREX 2010 Poster Session will be held Wednesday, 11 August: 16:00 - 17:45. We will be holding the MIREX plenary meeting 13:00-14:00 as a working lunch on the same day.

Our hosts in Utrecht need to know the number of posters so they can set up the room. Please add you name and the task(s) dealt with in your poster.

We had many groups/individuals submit across tasks. You can choose to create one ISMIR poster bringing all your data together or can split up your data across, say, two or three posters. If you have questions, please contact me at jdownie@illinois.edu or the MIREX mailing list about task poster options.

As a reminder, the MIREX posters need to follow the [http://ismir2010.ismir.net/information-for-authors/information-for-presenters/ ISMIR 2010 poster guidelines] (i.e., A0, portrait orientation).

==Add your author names here, once for each poster along with "title of some sort" and (Task(s) covered)==
# IMIRSEL: ''MIREX 2010 Overview, Part I'' (Train Test Tasks)
# IMIRSEL: ''MIREX 2010 Overview, Part II'' (All Other Tasks)
# Andreas Arzt and Gerhard Widmer: "Real-time Music Tracking using Tempo-aware On-line Dynamic Time Warping" (Real-time Audio to Score Alignment (a.k.a Score Following))
# Pasi Saari and Olivier Lartillot: "SubEnsemble - Classification framework based on the Ensemble Approach and Feature Selection" (Train Test Tasks)
# Gabriel Sargent, Frédéric Bimbot and Emmanuel Vincent: "Structural segmentation of songs using multi-criteria generalized likelihood ratio and regularity constraints" (Structural Segmentation Task)
# Emmanouil Benetos and Simon Dixon: "Multiple fundamental frequency estimation using spectral structure and temporal evolution rules" (Multiple Fundamental Frequency Estimation & Tracking Task)
# J. Urbano, J. Lloréns, J. Morato and S. Sánchez-Cuadrado: ''Local Alignment with Geometric Representations'' (Symbolic Melodic Similarity)
# F.J.Rodriguez-Serrano, P.Vera-Candeas, P.Cabanas-Molero, J.J.Carabias-Orti, N.Ruiz-Reyes: ''AM Sinusoidal Modeling for Onset Detection FOR ONSET DETECTION'' (Audio Onset Detection)
# R.Mata-Campos, F.J.Rodriguez-Serrano, P.Vera-Candeas, J.J.Carabias-Orti, F.J.Canadas-Quesada: ''Beat Tracking improved by AM Sinusoidal Modeled Onsets'' (Audio Beat Tracking)
# F.J.Rodriguez-Serrano, P.Vera-Candeas, J.J.Carabias-Orti,P.Cabanas-Molero, N.Ruiz-Reyes: ''Real time audio to score alignment based on NLS multipitch estimation'' (Real-time Audio to Score Alignment (a.k.a Score Following))
# F.J. Cañadas-Quesada, F. Rodríguez-Serrano, P. Vera-Candeas, N. Ruiz-Reyes and J. Carabias-Orti: ''Multiple Fundamental Frequency Estimation & Tracking in Polyphonic Music for MIREX 2010'' (Multiple Fundamental Frequency Estimation & Tracking)
# Zhiyao Duan and Bryan Pardo: "A Real-time Score Follower for MIREX 2010" (Real-time Audio to Score Alignment (a.k.a Score Following))
# Zhiyao Duan, Jinyu Han and Bryan Pardo: "A Multi-pitch Estimation and Tracking System" (Multiple Fundamental Frequency Estimation & Tracking Task)

==Below are some examples from MIREX 2009==

# Matt Hoffman: ''Using CBA to Automatically Tag Songs'' (Audio tag classification/retrieval)
# Suman Ravuri, Dan Ellis: ''The Hydra System of Cover Song Classification'' (Cover Song Identification)
# Joan Serra, Massimiliano Zanin, Ralph G Andrzejak: ''Cover song retrieval by recurrence quantification and unsupervised set detection'' (Cover Song Identification)
# MTG Team: "Music Type Groupers (MTG): Generic Music Classification Algorithms" (Audio Genre Classification, Mood Classification, Artist Identification, Classical Composer Identification)
# R. Jang: "Poster #2" (placeholder to get the auto-counter to increment)
# R. Jang: "Poster #3" (placeholder to get the auto-counter to increment)

2010:Real-time Audio to Score Alignment (a.k.a Score Following)

2010-07-01T02:40:10Z

Proton: /* Data */

''Real-time Audio to Score Alignment'', also known as ''Score Following''

== Description ==
Score Following is the real-time alignment of an incoming music signal to the music score. The music signal can be symbolic (MIDI) or audio, but we will concentrate here on audio following, unless there are some candidates who'd want their symbolic followers to be evaluated and can propose reference data.

This page describes a proposal for evaluation of score following systems. Discussion of the evaluation procedures on the [https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com01 Score Following contest planning list] will be documented on the [[Score Following]] page. A full digest of the discussions is available to subscribers from the [https://mail.lis.uiuc.edu/mailman/private/mrx-com01/ Score Following contest planning list archives].

Submissions will be required to estimate alignment precision according to the indexed times. In order for your system to participate, please specify the type of alignment (monophonic, polyphonic), type of training and realtime performance, also separated into two domains (upon enough submissions) for symbolic and audio systems. Note that we also do accept systems that don't run in real-time in practice, as soon as their algorithm is on-line, i.e. without making use of global knowledge of the input.

== Data ==
46 recordings and their corresponding MIDI representations of the score will be used in the evaluation. These 46 excerpts were extracted from 4 distinct musical pieces.
Recordings are in 44.1Khz 16bit wav format. The reference scores are in MIDI format.

Zhiyao Duan and Prof. Bryan Pardo contributed another polyphonic dataset. This dataset consists of 10 pieces of four-part J.S. Bach chorales. The audio file was performed by a quartet of instruments: violin, clarinet, saxophone and bassoon. The ground-truth alignment between audio and midi were generated by human annotation.

== Evaluation procedures ==

Evaluation procedure consists of running score followers on a database of aligned audio to score where the database contains score, and performance audio (for system call) and a reference alignment (for evaluations) --
See http://ismir2007.ismir.net/proceedings/ISMIR2007_p315_cont.pdf for details.

See the details of 2006 proposal on the [[2006:Score_Following_Proposal|MIREX 2006 Wiki]]

=== I/O Format ===
Each system should conform to the following format:

''doScofo.sh "/path/to/audiofile.wav" "/path/to/midi_score_file.mid" "/path/to/result/filename.txt"

The stdout and stderr will be logged.

"/path/to/result/filenam.txt" should be have one line per detected note with the following 4 columns

1. estimated note onset time in performance audio file (ms)
2. detection time relative to performance audio file (ms)
3. note start time in score (ms)
4. MIDI note number in score (int)

Example :
''1800 1800 0 75''
''2021 2022 187.5 73''
''... ... ... ...''

Remarks: The third column with the detected note's start time in score serves as the unique identifier of a note (or chord for polyphonic scores) that links it to the ground truth onset of that note within the reference alignment files. The fourth column of MIDI note number is there only for your convenience, to know your way around in the result files, if you know the melody in MIDI.

=== Packaging submissions ===
All submissions should be statically linked to all libraries (the presence of
dynamically linked libraries cannot be guarenteed).

All submissions should include a README file including the following the
information:

* Command line calling format for all executables and an example formatted set of commands
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Any required environments (and versions), e.g. python, java, bash, matlab.

== Time and hardware limits ==
Due to the potentially high number of particpants in this and other audio tasks,
hard limits on the runtime of submissions are specified.

A hard limit of 12 hours will be imposed on rthe total runtime of algorithms. Submissions that exceed this runtime may not receive a result.

== Submission closing date ==

Friday 4th June 2010

2010:Real-time Audio to Score Alignment (a.k.a Score Following)

2010-05-17T15:51:23Z

Proton: /* Potential Participants */

== Title ==
''Real-time Audio to Score Alignment'', also known as ''Score Following''

== Description ==

The text of this section is copied from the 2009 page. Please add your comments and discussions for 2010.

Score Following is the real-time alignment of an incoming music signal to the music score. The music signal can be symbolic (MIDI) or audio, but we will concentrate here on audio following, unless there are some candidates who'd want their symbolic followers to be evaluated and can propose reference data.

This page describes a proposal for evaluation of score following systems. Discussion of the evaluation procedures on the [https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com01 Score Following contest planning list] will be documented on the [[Score Following]] page. A full digest of the discussions is available to subscribers from the [https://mail.lis.uiuc.edu/mailman/private/mrx-com01/ Score Following contest planning list archives].

Submissions will be required to estimate alignment precision according to the indexed times. In order for your system to participate, please specify the type of alignment (monophonic, polyphonic), type of training and realtime performance, also separated into two domains (upon enough submissions) for symbolic and audio systems. Note that we also do accept systems that don't run in real-time in practice, as soon as their algorithm is on-line, i.e. without making use of global knowledge of the input.

== Discussions for 2010 ==

Your comments here.

== Evolution ==
This year's changes are proposed here and on the list, and are currently under discussion. Proposed changes are mainly about the score and reference file formats and the evaluation metrics:

* the proposed new score and reference file format is described here: [[2010:Score File Format]]
* evaluation metrics will more closely reflect the different approaches and applications of score following

See the details of last year's proposal on the [https://www.music-ir.org/mirex2006/index.php/Score_Following_Proposal MIREX 2006 Wiki]

== Title ==
''Real-time Audio to Score Alignment'', also known as ''Score Following''

== Description ==
Score Following is the real-time alignment of an incoming music signal to the music score. The music signal can be symbolic (MIDI) or audio, but we will concentrate here on audio following, unless there are some candidates who'd want their symbolic followers to be evaluated and can propose reference data.

This page describes a proposal for evaluation of score following systems. Discussion of the evaluation procedures on the [https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com01 Score Following contest planning list] will be documented on the [[Score Following]] page. A full digest of the discussions is available to subscribers from the [https://mail.lis.uiuc.edu/mailman/private/mrx-com01/ Score Following contest planning list archives].

Submissions will be required to estimate alignment precision according to the indexed times. In order for your system to participate, please specify the type of alignment (monophonic, polyphonic), type of training and realtime performance, also separated into two domains (upon enough submissions) for symbolic and audio systems. Note that we also do accept systems that don't run in real-time in practice, as soon as their algorithm is on-line, i.e. without making use of global knowledge of the input.

== Evolution ==
This year's changes are proposed here and on the list, and are currently under discussion. Proposed changes are mainly about the score and reference file formats and the evaluation metrics:

* the proposed new score and reference file format is described here: [[Score File Format]]
* evaluation metrics will more closely reflect the different approaches and applications of score following

See the details of last year's proposal on the [https://www.music-ir.org/mirex2006/index.php/Score_Following_Proposal MIREX 2006 Wiki]

== Evaluation procedures ==

Evaluation procedure consists of running score followers on a database of aligned audio to score where the database contains score, and performance audio (for system call) and a reference alignment (for evaluations) -- See below for details.

=== I/O Format ===
Each system should conform to the following format:

''doScofo.sh "/path/to/audiofile.wav" "/path/to/midi_score_file.wav" "/path/to/result/filename.txt"

The stdout and stderr will be logged.

"/path/to/result/filenam.txt" should be have one line per detected note with the following 4 columns

1. estimated note onset time in performance audio file (ms)
2. detection time relative to performance audio file (ms)
3. note start time in score (ms)
4. MIDI note number in score (int)

Example :
''1800 1800 0 75''
''2021 2022 187.5 73''
''... ... ... ...''

Remarks: The third column with the detected note's start time in score serves as the unique identifier of a note (or chord for polyphonic scores) that links it to the ground truth onset of that note within the reference alignment files. The fourth column of MIDI note number is there only for your convenience, to know your way around in the result files, if you know the melody in MIDI.

=== Potential Participants ===

Wei-Ta Chu, National Chung Cheng University, Taiwan. Email: wtchu AT cs DOT ccu DOT edu DOT tw

Zhiyao Duan, Bryan Pardo, Northwestern University, USA. Email: zhiyaoduan00 AT gmail <dot> com

2010:Multiple Fundamental Frequency Estimation & Tracking

2010-05-17T15:50:56Z

Proton: /* Potential Participants */

==Description==

The text of this section is copied from the 2009 page. Please add your comments and discussions for 2010.

That a complex music signal can be represented by the F0 contours of its constituent sources is a very useful concept for most music information retrieval systems. There have been many attempts at multiple (aka polyphonic) F0 estimation and melody extraction, a related area. The goal of multiple F0 estimation and tracking is to identify the active F0s in each time frame and to track notes and timbres continuously in a complex music signal. In this task, we would like to evaluate state-of-the-art multiple-F0 estimation and tracking algorithms. Since F0 tracking of all sources in a complex audio mixture can be very hard, we are restricting the problem to 3 cases:

1. Estimate active fundamental frequencies on a frame-by-frame basis.

2. Track note contours on a continuous time basis. (as in audio-to-midi). This task will also include a piano transcription sub task.

3. Track timbre on a continous time basis.

The deadline For this task is September 8th. Please feel free to request extension if needed.

== Discussions for 2010 ==

== Discussions from 2009 ==

https://www.music-ir.org/mirex/2009/index.php/Multiple_Fundamental_Frequency_Estimation_%26_Tracking#Discussions_for_2009

==Data==

A woodwind quintet transcription of the fifth variation from L. van Beethoven's Variations for String Quartet Op.18 No. 5. Each part (flute, oboe, clarinet, horn, or bassoon) was recorded separately while the performer listened to the other parts (recorded previously) through headphones. Later the parts were mixed to a monaural 44.1kHz/16bits file.

Synthesized pieces using RWC MIDI and RWC samples. Includes pieces from Classical and Jazz collections. Polyphony changes from 1 to 4 sources.

Polyphonic piano recordings generated using a disklavier playback piano.

So, there are 6, 30-sec clips for each polyphony (2-3-4-5) for a total of 30 examples, plus there are 10 30-sec polyphonic piano clips. Please email me about your estimated running time (in terms of n times realtime), if we believe everybodyΓÇÖs algorithm is fast enough, we can increase the number of test samples. (There were 90 x real-time algo`s for melody extraction tasks in the past.)

All files are in 44.1kHz / 16 bit wave format. The development set can be found at
[https://www.music-ir.org/evaluation/MIREX/data/2007/multiF0/index.htm Development Set for MIREX 2007 MultiF0 Estimation Tracking Task].

Send an email to [mailto:mertbay@uiuc.edu mertbay@uiuc.edu] for the username and password.

==Evaluation==

This year, We would like to discuss different evaluation methods. From last year`s result, it can be seen that on note tracking, algorithms performed poorly when evaluated using note offsets. Below is the evaluation methods we used last year:

For Task 1 (frame level evaluation), systems will report the number of active pitches every 10ms. Precision (the portion of correct retrieved pitches for all pitches retrieved for each frame) and Recall (the ratio of correct pitches to all ground truth pitches for each frame) will be reported. A Returned Pitch is assumed to be correct if it is within a half semitone (+ - 3%) of a ground-truth pitch for that frame. Only one ground-truth pitch can be associated with each Returned Pitch.
Also as suggested, an error score as described in [http://www.hindawi.com/GetArticle.aspx?doi=10.1155/2007/48317 Poliner and Ellis p.g. 5 ] will be calculated.
The frame level ground truth will be calculated by [http://www.ircam.fr/pcm/cheveign/sw/yin.zip YIN] and hand corrected.

For Task 2 (note tracking), again Precision (the ratio of correctly transcribed ground truth notes to the number of ground truth notes for that input clip) and Recall (ratio of correctly transcribed ground truth notes to the number of transcribed notes) will be reported. A ground truth note is assumed to be correctly transcribed if the system returns a note that is within a half semitone (+ - 3%) of that note AND the returned note`s onset is within a 100ms range( + - 50ms) of the onset of the ground truth note, and its offset is within 20% range of the ground truth note`s offset. Again, one ground truth note can only be associated with one transcribed note.

The ground truth for this task will be annotated by hand. An amplitude threshold relative to the file/instrument will be determined. Note onset is going to be set to the time where its amplitude rises higher than the threshold and the offset is going to be set to the the time where the note`s amplitude decays lower than the threshold. The ground truth is going to be set as the average F0 between the onset and the offset of the note.
In the case of legato, the onset/offset is going to be set to the time where the F0 deviates more than 3% of the average F0 through out the the note up to that point. There is not going to be any vibrato larger than a half semitone in the test data.

Different statistics can also be reported if agreed by the participants.

== Submission Format ==

Submissions have to conform to the specified format below:

''doMultiF0 "path/to/file.wav" "path/to/output/file.F0" ''

path/to/file.wav: Path to the input audio file.

path/to/output/file.F0: The output file.

Programs can use their working directory if they need to keep temporary cache files or internal debuggin info. Stdout and stderr will be logged.

For each task, the format of the output file is going to be different:
For the first task, F0-estimation on frame basis, the output will be a file where each row has a time stamp and a number of active F0s in that frame, separated by a tab for every 10ms increments.

Example :
''time F01 F02 F03 ''
''time F01 F02 F03 F04''
''time ... ... ... ...''

which might look like:

''0.78 146.83 220.00 349.23''
''0.79 349.23 146.83 369.99 220.00 ''
''0.80 ... ... ... ...''

For the second task, for each row, the file should contain the onset, offset and the F0 of each note event separated by a tab, ordered in terms of onset times:

onset offset F01
onset offset F02
... ... ...
which might look like:

0.68 1.20 349.23
0.72 1.02 220.00
... ... ...
The DEADLINE is TBA.

==Potential Participants==
Emmanouil Benetos, Simon Dixon, Centre for Digital Music, Queen Mary University of London, UK. emmanouil.benetos at elec.qmul.ac.uk

Zhiyao Duan, Jinyu Han, Bryan Pardo, Northwestern University, USA. Email: zhiyaoduan00 AT gmail <dot> com

If you might consider participating, please add your name and email address here and also please sign up for the Multi-F0 mail list:
[https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com03 Multi-F0 Estimation Tracking email list]

2010:Multiple Fundamental Frequency Estimation & Tracking

2010-05-17T15:50:40Z

Proton: /* Potential Participants */

==Description==

The text of this section is copied from the 2009 page. Please add your comments and discussions for 2010.

That a complex music signal can be represented by the F0 contours of its constituent sources is a very useful concept for most music information retrieval systems. There have been many attempts at multiple (aka polyphonic) F0 estimation and melody extraction, a related area. The goal of multiple F0 estimation and tracking is to identify the active F0s in each time frame and to track notes and timbres continuously in a complex music signal. In this task, we would like to evaluate state-of-the-art multiple-F0 estimation and tracking algorithms. Since F0 tracking of all sources in a complex audio mixture can be very hard, we are restricting the problem to 3 cases:

1. Estimate active fundamental frequencies on a frame-by-frame basis.

2. Track note contours on a continuous time basis. (as in audio-to-midi). This task will also include a piano transcription sub task.

3. Track timbre on a continous time basis.

The deadline For this task is September 8th. Please feel free to request extension if needed.

== Discussions for 2010 ==

== Discussions from 2009 ==

https://www.music-ir.org/mirex/2009/index.php/Multiple_Fundamental_Frequency_Estimation_%26_Tracking#Discussions_for_2009

==Data==

A woodwind quintet transcription of the fifth variation from L. van Beethoven's Variations for String Quartet Op.18 No. 5. Each part (flute, oboe, clarinet, horn, or bassoon) was recorded separately while the performer listened to the other parts (recorded previously) through headphones. Later the parts were mixed to a monaural 44.1kHz/16bits file.

Synthesized pieces using RWC MIDI and RWC samples. Includes pieces from Classical and Jazz collections. Polyphony changes from 1 to 4 sources.

Polyphonic piano recordings generated using a disklavier playback piano.

So, there are 6, 30-sec clips for each polyphony (2-3-4-5) for a total of 30 examples, plus there are 10 30-sec polyphonic piano clips. Please email me about your estimated running time (in terms of n times realtime), if we believe everybodyΓÇÖs algorithm is fast enough, we can increase the number of test samples. (There were 90 x real-time algo`s for melody extraction tasks in the past.)

All files are in 44.1kHz / 16 bit wave format. The development set can be found at
[https://www.music-ir.org/evaluation/MIREX/data/2007/multiF0/index.htm Development Set for MIREX 2007 MultiF0 Estimation Tracking Task].

Send an email to [mailto:mertbay@uiuc.edu mertbay@uiuc.edu] for the username and password.

==Evaluation==

This year, We would like to discuss different evaluation methods. From last year`s result, it can be seen that on note tracking, algorithms performed poorly when evaluated using note offsets. Below is the evaluation methods we used last year:

For Task 1 (frame level evaluation), systems will report the number of active pitches every 10ms. Precision (the portion of correct retrieved pitches for all pitches retrieved for each frame) and Recall (the ratio of correct pitches to all ground truth pitches for each frame) will be reported. A Returned Pitch is assumed to be correct if it is within a half semitone (+ - 3%) of a ground-truth pitch for that frame. Only one ground-truth pitch can be associated with each Returned Pitch.
Also as suggested, an error score as described in [http://www.hindawi.com/GetArticle.aspx?doi=10.1155/2007/48317 Poliner and Ellis p.g. 5 ] will be calculated.
The frame level ground truth will be calculated by [http://www.ircam.fr/pcm/cheveign/sw/yin.zip YIN] and hand corrected.

For Task 2 (note tracking), again Precision (the ratio of correctly transcribed ground truth notes to the number of ground truth notes for that input clip) and Recall (ratio of correctly transcribed ground truth notes to the number of transcribed notes) will be reported. A ground truth note is assumed to be correctly transcribed if the system returns a note that is within a half semitone (+ - 3%) of that note AND the returned note`s onset is within a 100ms range( + - 50ms) of the onset of the ground truth note, and its offset is within 20% range of the ground truth note`s offset. Again, one ground truth note can only be associated with one transcribed note.

The ground truth for this task will be annotated by hand. An amplitude threshold relative to the file/instrument will be determined. Note onset is going to be set to the time where its amplitude rises higher than the threshold and the offset is going to be set to the the time where the note`s amplitude decays lower than the threshold. The ground truth is going to be set as the average F0 between the onset and the offset of the note.
In the case of legato, the onset/offset is going to be set to the time where the F0 deviates more than 3% of the average F0 through out the the note up to that point. There is not going to be any vibrato larger than a half semitone in the test data.

Different statistics can also be reported if agreed by the participants.

== Submission Format ==

Submissions have to conform to the specified format below:

''doMultiF0 "path/to/file.wav" "path/to/output/file.F0" ''

path/to/file.wav: Path to the input audio file.

path/to/output/file.F0: The output file.

Programs can use their working directory if they need to keep temporary cache files or internal debuggin info. Stdout and stderr will be logged.

For each task, the format of the output file is going to be different:
For the first task, F0-estimation on frame basis, the output will be a file where each row has a time stamp and a number of active F0s in that frame, separated by a tab for every 10ms increments.

Example :
''time F01 F02 F03 ''
''time F01 F02 F03 F04''
''time ... ... ... ...''

which might look like:

''0.78 146.83 220.00 349.23''
''0.79 349.23 146.83 369.99 220.00 ''
''0.80 ... ... ... ...''

For the second task, for each row, the file should contain the onset, offset and the F0 of each note event separated by a tab, ordered in terms of onset times:

onset offset F01
onset offset F02
... ... ...
which might look like:

0.68 1.20 349.23
0.72 1.02 220.00
... ... ...
The DEADLINE is TBA.

==Potential Participants==
Emmanouil Benetos, Simon Dixon, Centre for Digital Music, Queen Mary University of London, UK. emmanouil.benetos at elec.qmul.ac.uk
Zhiyao Duan, Jinyu Han, Bryan Pardo, Northwestern University, USA. Email: zhiyaoduan00 AT gmail <dot> com

If you might consider participating, please add your name and email address here and also please sign up for the Multi-F0 mail list:
[https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com03 Multi-F0 Estimation Tracking email list]

2010:Real-time Audio to Score Alignment (a.k.a Score Following)

2010-05-17T15:47:18Z

Proton: /* Potential Participants */

== Title ==
''Real-time Audio to Score Alignment'', also known as ''Score Following''

== Description ==

The text of this section is copied from the 2009 page. Please add your comments and discussions for 2010.

Score Following is the real-time alignment of an incoming music signal to the music score. The music signal can be symbolic (MIDI) or audio, but we will concentrate here on audio following, unless there are some candidates who'd want their symbolic followers to be evaluated and can propose reference data.

This page describes a proposal for evaluation of score following systems. Discussion of the evaluation procedures on the [https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com01 Score Following contest planning list] will be documented on the [[Score Following]] page. A full digest of the discussions is available to subscribers from the [https://mail.lis.uiuc.edu/mailman/private/mrx-com01/ Score Following contest planning list archives].

Submissions will be required to estimate alignment precision according to the indexed times. In order for your system to participate, please specify the type of alignment (monophonic, polyphonic), type of training and realtime performance, also separated into two domains (upon enough submissions) for symbolic and audio systems. Note that we also do accept systems that don't run in real-time in practice, as soon as their algorithm is on-line, i.e. without making use of global knowledge of the input.

== Discussions for 2010 ==

Your comments here.

== Evolution ==
This year's changes are proposed here and on the list, and are currently under discussion. Proposed changes are mainly about the score and reference file formats and the evaluation metrics:

* the proposed new score and reference file format is described here: [[2010:Score File Format]]
* evaluation metrics will more closely reflect the different approaches and applications of score following

See the details of last year's proposal on the [https://www.music-ir.org/mirex2006/index.php/Score_Following_Proposal MIREX 2006 Wiki]

== Title ==
''Real-time Audio to Score Alignment'', also known as ''Score Following''

== Description ==
Score Following is the real-time alignment of an incoming music signal to the music score. The music signal can be symbolic (MIDI) or audio, but we will concentrate here on audio following, unless there are some candidates who'd want their symbolic followers to be evaluated and can propose reference data.

This page describes a proposal for evaluation of score following systems. Discussion of the evaluation procedures on the [https://mail.lis.uiuc.edu/mailman/listinfo/mrx-com01 Score Following contest planning list] will be documented on the [[Score Following]] page. A full digest of the discussions is available to subscribers from the [https://mail.lis.uiuc.edu/mailman/private/mrx-com01/ Score Following contest planning list archives].

Submissions will be required to estimate alignment precision according to the indexed times. In order for your system to participate, please specify the type of alignment (monophonic, polyphonic), type of training and realtime performance, also separated into two domains (upon enough submissions) for symbolic and audio systems. Note that we also do accept systems that don't run in real-time in practice, as soon as their algorithm is on-line, i.e. without making use of global knowledge of the input.

== Evolution ==
This year's changes are proposed here and on the list, and are currently under discussion. Proposed changes are mainly about the score and reference file formats and the evaluation metrics:

* the proposed new score and reference file format is described here: [[Score File Format]]
* evaluation metrics will more closely reflect the different approaches and applications of score following

See the details of last year's proposal on the [https://www.music-ir.org/mirex2006/index.php/Score_Following_Proposal MIREX 2006 Wiki]

== Evaluation procedures ==

Evaluation procedure consists of running score followers on a database of aligned audio to score where the database contains score, and performance audio (for system call) and a reference alignment (for evaluations) -- See below for details.

=== I/O Format ===
Each system should conform to the following format:

''doScofo.sh "/path/to/audiofile.wav" "/path/to/midi_score_file.wav" "/path/to/result/filename.txt"

The stdout and stderr will be logged.

"/path/to/result/filenam.txt" should be have one line per detected note with the following 4 columns

1. estimated note onset time in performance audio file (ms)
2. detection time relative to performance audio file (ms)
3. note start time in score (ms)
4. MIDI note number in score (int)

Example :
''1800 1800 0 75''
''2021 2022 187.5 73''
''... ... ... ...''

Remarks: The third column with the detected note's start time in score serves as the unique identifier of a note (or chord for polyphonic scores) that links it to the ground truth onset of that note within the reference alignment files. The fourth column of MIDI note number is there only for your convenience, to know your way around in the result files, if you know the melody in MIDI.

=== Potential Participants ===

Wei-Ta Chu, National Chung Cheng University, Taiwan. Email: wtchu AT cs DOT ccu DOT edu DOT tw
Zhiyao Duan, Bryan Pardo, Northwestern University, USA. Email: zhiyaoduan00 AT gmail <dot> com