MIREX Wiki - User contributions [en]

2010:Symbolic Similarity 2010 Graders

2010-07-12T16:47:05Z

Karishma: Created page with '=SMS 2010 Graders= Welcome to the SMS grader sign-up page. Please give us your name and email contact information. If you obscure your email, please make it relatively obvious t…'

=SMS 2010 Graders=

Welcome to the SMS grader sign-up page. Please give us your name and email contact information. If you obscure your email, please make it relatively obvious to us how to parse the address.

Template: Name. Location. <Email>

Sample: J. Stephen Downie. Illinois, USA. <jdownie@illinois.edu>

==Sign Up Area==

2010:Audio Similarity 2010 Graders

2010-07-12T16:44:30Z

Karishma: /* AMS 2009 Graders */

=AMS 2010 Graders=

Welcome to the AMS grader sign-up page. Please give us your name and email contact information. If you obscure your email, please make it relatively obvious to us how to parse the address.

Template: Name. Location. <Email>

Sample: J. Stephen Downie. Illinois, USA. <jdownie@illinois.edu>

==Sign Up Area==

2010:Audio Similarity 2010 Graders

2010-07-12T16:44:06Z

Karishma: Created page with '=AMS 2009 Graders= Welcome to the AMS grader sign-up page. Please give us your name and email contact information. If you obscure your email, please make it relatively obvious t…'

=AMS 2009 Graders=

Welcome to the AMS grader sign-up page. Please give us your name and email contact information. If you obscure your email, please make it relatively obvious to us how to parse the address.

Template: Name. Location. <Email>

Sample: J. Stephen Downie. Illinois, USA. <jdownie@illinois.edu>

==Sign Up Area==

2006:Audio Beat Tracking

2010-05-20T19:34:32Z

Karishma: /* References */

== Results ==

Results are on [[2006:Audio Beat Tracking Results]] page.

'''NOTE:''' Due to an evaluation error, the results of the Audio Beat Tracking task have been updated as of 17 July 2007, and differ from those presented at ISMIR 2006.

== Proposers ==

* Paul M. Brossier (Queen Mary, University of London) <piem at altern.org>
* Matthew Davies (Queen Mary, University of London) <matthew.davies at elec.qmul.ac.uk>
* Martin F. McKinney (Philips) <mckinney at alum.mit.edu>

== Description ==

The aim of the automatic beat tracking task is to track each beat locations in a collection of sound files. Unlike the [[2006:Audio Tempo Extraction]] task, which aim is to detect tempi for each file, the beat tracking task aims at detecting all beat locations in recordings. The algorithms will be evaluated in terms of their accuracy in predicting beat locations annotated by a group of listeners.

=== Input data ===

''Audio Format'':

The sound files are the same 160 30-second excerpts (WAV format) used for the Audio Tempo contest. Beat locations have been annotated in each excerpt by 40 different listeners (39 listeners for a few excerpts. The length of each excerpt is 30 seconds.

''Audio Content'':

The audio recordings were selected to provide a stable tempo value, a wide distribution of tempi values, and a large variety of instrumentation and musical styles. About 20% of the files contain non-binary meters, and a small number of examples contain changing meters. One disadvantage of using this set for beat tracking is that the tempi are rather stable and this set will not test beat-tracking algorithms in their ability to track tempo changes.

=== Output data ===

Submitted programs should output one beat location per line, with a ┬½new line┬╗ character (\n) at the end of each line. The results should either be saved to a text file.

Example of possible output:

0.0123156 
1.9388662 
3.8777323 
5.8165980 
7.7554634 

Each submission should be accompanied with a README file describing how the program should be used. For instance:

To run the program ''foobar'' on the file input.wav and store the results in the file output.txt, the following command should be used:

foobar -i input.wav > output.txt

== Participants ==

* Miguel Alonso and Ga├½l Richard (ENST, Paris), <miguel.alonso at enst.fr>, <gael.richard at enst.fr> ''(to be confirmed)''
* Paul Brossier (Queen Mary, University of London), <piem at altern.org>
* Matthew Davies (Queen Mary, University of London), <matthew.davies at elec.qmul.ac.uk>
* Douglas Eck (University of Montreal), <eckdoug at iro.umontreal.ca>
* Geoffroy Peeters (IRCAM, Paris), <peeters at ircam.fr>

Other potential participants:

* Fabien Gouyon (University Pompeu Fabra) and Simon Dixon (OFAI), <fabien.gouyon at iua.upf.es>, <simon at oefai.at>
* Anssi Klapuri (Tampere International Center for Signal Processing, Finland), <klap at cs.tut.fi>
* Martin F. McKinney (Philips) <mckinney at alum.mit.edu>
* Dirk Moelants (IPEM, Ghent University) <dirk at moelants.net>
* Bill Sethares (University of Wisconsin-Madison), <sethares at ece.wisc.edu>
* George Tzanetakis (University of Victoria), <gtzan at cs.uvic.ca>
* Christian Uhle (Fraunhofer Institut), <uhle at idmt.fhg.de>

== Evaluation Procedures ==

''This is a major re-write by Martin McKinney and is open to suggestions.''

Evaluation of beat-tracking includes an implicit evaluation of tempo accuracy, however, the focus here will be on proper time position of beats. We propose the following evaluation method, which is quite simple in nature and accounts for ambiguity in the perception of the most salient metrical level: For each excerpt, an impulse train will be created from each of the 40 annotated ground truth beat vectors as well as from the algorithm output. The impulse trains will be 25 seconds long (ignoring tapped beats at times less than 5 seconds), constructed with a 100-Hz sampling rate, and have unit impulses at beat times. Each impulse train of annotations will be denoted by <math>a_s[n]</math>, where the subscript <math>s</math> is the annotator number (1-40), and the impulse train from the algorithm will be donoted by <math>y[n]</math>. The performance, <math>p</math>, of the beat-tracking algorithm for a single excperpt will be measured by calculating the cross-correlation function of <math>a_s[n]</math> and <math>y[n]</math> within a small delay window, <math>W</math>, around zero and then averaged across the number of annotators (<math>S</math>):

<math>P = \frac{1}{S}\sum_{s=1}^{S}\frac{1}{NP}\sum_{m=-W}^{+W}{\sum_{n=1}^{N}{y[n] \cdot a_s[n-m]}}</math>,

where <math>N</math> is the sample-length of pulse trains <math>y[n]</math> and <math>a_s[n]</math>, and NP is a normalization factor defined by the maximum number of impulses in either impulse train:

<math>NP = \mbox{max}(\sum{y[n]},\sum{a_s[n]})</math>.

The "error" window, W, is proportional to (1/5 of) the beat in the annotated taps and is defined (in Matlab notation ;-) as:

<math>W</math> = round(0.2 * median(diff(find((a_s[n])))).

The algorithm with the best average P-score (across excerpts) will win.

''The choice of 1/5 of the beat was somewhat arbitrarily chosen and is open for discussion. I've used this method to examine correlations between taps of different subjects and it works quite well. Comments please. -Martin''

== Evaluation Database ==

A collection of 160 musical exerpts will be used for the evaluation procedure, the same collection used for the [[2006:Audio Tempo Extraction]] contest. Each recording has been annotated by 40 different listeners (39 in a few cases). The annotation procedures were described in [2] and [3].

20 excerpts will be provided for training to the participant, and the remaining 140 excerpts, novel to all participants, will be used for the contest.

== References ==

# Masataka Goto and Yoichi Muraoka. Issues in evaluating beat tracking systems. In Working Notes of IJCAI-97 Workshop on Issues in AI and Music - Evaluation and Assessment, pages 9┬¡16, 1997 [http://staff.aist.go.jp/m.goto/PAPER/AIM97.300dpi.ps postscript]
# McKinney, M.F. and Moelants, D. (2004), Deviations from the resonance theory of tempo induction, Conference on Interdisciplinary Musicology, Graz. [http://gewi.kfunigraz.ac.at/~cim04/CIM04_paper_pdf/McKinney_Moelants_CIM04_proceedings_t.pdf pdf]
# Moelants, D. and McKinney, M.F. (2004), Tempo perception and musical content: What makes a piece slow, fast, or temporally ambiguous? International Conference on Music Perception & Cognition, Evanston, IL. [http://www.northwestern.edu/icmpc/proceedings/ICMPC8/PDF/AUTHOR/MP040237.pdf pdf]

== Comments ==

Paul's comments: as noted off-list by Matthew Davies, Goto proposed evaluation metrics to evaluate beat tracking algorithms accuracy [1] are somewhat difficult to apply to all beat tracking algorithms without modifications, since they assume the algorithms stabilise on a robust tempo value only after 45 seconds. Even after removing this 45s constraint, the four different metrics obtained by this method are somewhat difficult to interpret.

==Practice Data==
You can find it here:

https://www.music-ir.org/evaluation/MIREX/data/2006/beat/

User: beattrack Password: b34trx

https://www.music-ir.org/evaluation/MIREX/data/2006/tempo/

User: tempo Password: t3mp0

Data has been uploaded in both .tgz and .zip format.

2009:MIREX 2009 Submission Instructions

2010-05-20T18:37:52Z

Karishma: /* Extended Abstract Details */

==Introduction==
This is the official set of submission guidelines for MIREX 2009. The URL for MIREX 2009 Submission System can be found at the bottom of this page. We are deliberately placing the Submission URL at the bottom of this page to force folks to thoroughly read through this MIREX 2009 instruction set.

==Step-By-Step Instructions==
# Read this wiki page from top to bottom.
# Go read Andreas Ehmann's [[2009:Best Coding Practices for MIREX]].
# Go read the task-specific instructions for each task to which you will be submitting. Links to each task are available off the [[2009:Main Page|MIREX 2009]] main wiki page.
# For each submission, be sure you understand the Input/Output format issues as laid down by the task organizers.
# For each submission, be absolutely sure you have constructed a README.txt file that:
## Begins with submitter name(s) and contact information (email, etc.).
## Meets whatever task-specific requirements were laid down by the task organizers.
## Provides the team at IMIRSEL with a set of''' clear and complete''' instructions on how to run your program.
# Create a first-draft extended abstract that outlines the basic methods and techniques being deployed in your submission. See Extended Abstract Details below.
# Review, one last time, all the various submission requirements.
# Go to the MIREX 2009 Submission System page (URL at bottom of this page).
# Follow the instructions provided on the MIREX 2009 Submission System page.
# Submit your spiffy code bundle (don't forget the README.txt file) and your first-pass extended abstract!

==Extended Abstract Details==
The extended abstracts provide the outside world with a general understanding of what each submission is trying to accomplish. The extended abstracts need NOT be cutting edge/never-before-published materials. The extended abstracts will be revised by the authors after the data has been collected (to allow for commentary on results data); however, we at IMIRSEL still need the first-pass drafts at submission time to help us understand what is happening in the submission. Like last year we will post the final versions of the extended abstracts as part of the MIREX 2009 results page (see MIREX 2005 Results; https://www.music-ir.org/evaluation/mirex-results/).

The MIREX 2009 extended abstracts:
# Are two to four pages long (think ISMIR poster/demo papers).
# Must be created using the ISMIR 2009 template set: http://ismir2009.ismir.net/submit .
# Must be submitted in PDF format.
# Should include, if exists, references to other publications about your work (yes, self-reference is encouraged!)
# Should have the same general look and feel as these examples from last year:

* https://www.music-ir.org/evaluation/mirex-results/articles/audio_genre/west.pdf
* https://www.music-ir.org/evaluation/mirex-results/articles/audio_genre/pampalk.pdf

=MIREX 2009 Submission System URL=

[https://www.music-ir.org/evaluation/MIREX/submission/ https://www.music-ir.org/evaluation/MIREX/submission/]

2009:MIREX 2009 Submission Instructions

2010-05-20T18:36:38Z

Karishma: /* Extended Abstract Details */

==Introduction==
This is the official set of submission guidelines for MIREX 2009. The URL for MIREX 2009 Submission System can be found at the bottom of this page. We are deliberately placing the Submission URL at the bottom of this page to force folks to thoroughly read through this MIREX 2009 instruction set.

==Step-By-Step Instructions==
# Read this wiki page from top to bottom.
# Go read Andreas Ehmann's [[2009:Best Coding Practices for MIREX]].
# Go read the task-specific instructions for each task to which you will be submitting. Links to each task are available off the [[2009:Main Page|MIREX 2009]] main wiki page.
# For each submission, be sure you understand the Input/Output format issues as laid down by the task organizers.
# For each submission, be absolutely sure you have constructed a README.txt file that:
## Begins with submitter name(s) and contact information (email, etc.).
## Meets whatever task-specific requirements were laid down by the task organizers.
## Provides the team at IMIRSEL with a set of''' clear and complete''' instructions on how to run your program.
# Create a first-draft extended abstract that outlines the basic methods and techniques being deployed in your submission. See Extended Abstract Details below.
# Review, one last time, all the various submission requirements.
# Go to the MIREX 2009 Submission System page (URL at bottom of this page).
# Follow the instructions provided on the MIREX 2009 Submission System page.
# Submit your spiffy code bundle (don't forget the README.txt file) and your first-pass extended abstract!

==Extended Abstract Details==
The extended abstracts provide the outside world with a general understanding of what each submission is trying to accomplish. The extended abstracts need NOT be cutting edge/never-before-published materials. The extended abstracts will be revised by the authors after the data has been collected (to allow for commentary on results data); however, we at IMIRSEL still need the first-pass drafts at submission time to help us understand what is happening in the submission. Like last year we will post the final versions of the extended abstracts as part of the MIREX 2009 results page (see MIREX 2005 Results; https://www.music-ir.org/evaluation/mirex-results/).

The MIREX 2009 extended abstracts:
# Are two to four pages long (think ISMIR poster/demo papers).
# Must be created using the ISMIR 2009 template set: http://www.ismir2009.ismir.net/submit .
# Must be submitted in PDF format.
# Should include, if exists, references to other publications about your work (yes, self-reference is encouraged!)
# Should have the same general look and feel as these examples from last year:

* https://www.music-ir.org/evaluation/mirex-results/articles/audio_genre/west.pdf
* https://www.music-ir.org/evaluation/mirex-results/articles/audio_genre/pampalk.pdf

=MIREX 2009 Submission System URL=

[https://www.music-ir.org/evaluation/MIREX/submission/ https://www.music-ir.org/evaluation/MIREX/submission/]