2010:Audio Beat Tracking - Revision history

Kriswest: /* Collections */

2010-06-07T11:13:20Z

‎Collections

Kriswest at 09:27, 5 June 2010

2010-06-05T09:27:19Z

AndreasEhmann: /* Evaluation Procedures */

2010-05-24T19:26:49Z

‎Evaluation Procedures

AndreasEhmann: /* Evaluation Procedures */

2010-05-24T19:25:35Z

‎Evaluation Procedures

AndreasEhmann at 19:24, 24 May 2010

2010-05-24T19:24:14Z

AndreasEhmann at 19:23, 24 May 2010

2010-05-24T19:23:18Z

AndreasEhmann at 19:22, 24 May 2010

2010-05-24T19:22:05Z

Show changes

Leetc: /* Participants */

2010-05-14T00:53:38Z

‎Participants

MertBay: /* Participants */

2010-03-22T18:06:47Z

‎Participants

MertBay: Created page with '== Description == The aim of the automatic beat tracking task is to track each beat locations in a collection of sound files. Unlike the Audio Tempo Extraction task, which aΓÇª'

2010-03-22T18:06:33Z

Created page with '== Description == The aim of the automatic beat tracking task is to track each beat locations in a collection of sound files. Unlike the Audio Tempo Extraction task, which aΓÇª'

New page

== Description ==

The aim of the automatic beat tracking task is to track each beat locations in a collection of sound files. Unlike the [[Audio Tempo Extraction]] task, which aim is to detect tempi for each file, the beat tracking task aims at detecting all beat locations in recordings. The algorithms will be evaluated in terms of their accuracy in predicting beat locations annotated by a group of listeners.

=== Input data ===

''Audio Format'':

We have an additional data set this year of Chopin Mazurka pieces. There are on the order of 367 pieces with marked up beat locations. Because the tempo varies so much in these pieces, we expect it to be extremely difficult. Nevertheless, it should prove to be interesting for evaluations. We might select a subset of these 367 for the eval database.

The original old dataset are the same 160 30-second excerpts (WAV format) used for the Audio Tempo and Beat contests in 2006. Beat locations have been annotated in each excerpt by 40 different listeners (39 listeners for a few excerpts. The length of each excerpt is 30 seconds.

''Audio Content'':

The older dataset audio recordings were selected to provide a stable tempo value, a wide distribution of tempi values, and a large variety of instrumentation and musical styles. About 20% of the files contain non-binary meters, and a small number of examples contain changing meters. One disadvantage of using this set for beat tracking is that the tempi are rather stable and this set will not test beat-tracking algorithms in their ability to track tempo changes.

The second mazurka dataset contains tempo changes so it will evaluate the ability of algorithms to track these.

=== Output data ===

Submitted programs should output one beat location per line, with a ┬½new line┬╗ character (\n) at the end of each line. The results should either be saved to a text file.

Example of possible output:

0.0123156 
1.9388662 
3.8777323 
5.8165980 
7.7554634 

Each submission should be accompanied with a README file describing how the program should be used. For instance:

To run the program ''foobar'' on the file input.wav and store the results in the file output.txt, the following commands are examples of what should be used:

foobar -i input.wav > output.txt
foobar -i input.wav -o output.txt
foobar input.wav output.txt

== Participants ==

* Matthew Davies (Queen Mary, University of London), <matthew.davies at elec.qmul.ac.uk>
* Douglas Eck (University of Montreal), <eckdoug at iro.umontreal.ca>
* Simon Dixon (Queen Mary, University of London) <simon.dixon at elec.qmul.ac.uk>
* Geoffroy Peeters (Ircam - CNRS) <geoffroy.peeters at ircam.fr>
* Tsung-Chi Lee (National Tsing Hua University of Taiwan), <leetc at mirlab.org>

== Evaluation Procedures ==

''This is a major re-write by Martin McKinney and is open to suggestions.''

Evaluation of beat-tracking includes an implicit evaluation of tempo accuracy, however, the focus here will be on proper time position of beats. We propose the following evaluation method, which is quite simple in nature and accounts for ambiguity in the perception of the most salient metrical level: For each excerpt, an impulse train will be created from each of the 40 annotated ground truth beat vectors as well as from the algorithm output. The impulse trains will be 25 seconds long (ignoring tapped beats at times less than 5 seconds), constructed with a 100-Hz sampling rate, and have unit impulses at beat times. Each impulse train of annotations will be denoted by <math>a_s[n]</math>, where the subscript <math>s</math> is the annotator number (1-40), and the impulse train from the algorithm will be donoted by <math>y[n]</math>. The performance, <math>p</math>, of the beat-tracking algorithm for a single excperpt will be measured by calculating the cross-correlation function of <math>a_s[n]</math> and <math>y[n]</math> within a small delay window, <math>W</math>, around zero and then averaged across the number of annotators (<math>S</math>):

<math>P = \frac{1}{S}\sum_{s=1}^{S}\frac{1}{NP}\sum_{m=-W}^{+W}{\sum_{n=1}^{N}{y[n] \cdot a_s[n-m]}}</math>,

where <math>N</math> is the sample-length of pulse trains <math>y[n]</math> and <math>a_s[n]</math>, and NP is a normalization factor defined by the maximum number of impulses in either impulse train:

<math>NP = \mbox{max}(\sum{y[n]},\sum{a_s[n]})</math>.

The "error" window, W, is proportional to (1/5 of) the beat in the annotated taps and is defined (in Matlab notation ;-) as:

<math>W</math> = round(0.2 * median(diff(find((a_s[n])))).

The algorithm with the best average P-score (across excerpts) will win.

''The choice of 1/5 of the beat was somewhat arbitrarily chosen and is open for discussion. I've used this method to examine correlations between taps of different subjects and it works quite well. Comments please. -Martin''

== Evaluation Database ==

A collection of 160 musical exerpts will be used for the evaluation procedure, the same collection used for the 2006 Audio Tempo Extraction and Beat tasks. Each recording has been annotated by 40 different listeners (39 in a few cases). The annotation procedures were described in [2] and [3].

20 excerpts will be provided for training to the participant, and the remaining 140 excerpts, novel to all participants, will be used for the contest.

The second dataset consists of 367 mazurka performances, although a subset of these may be used.

== References ==

# Masataka Goto and Yoichi Muraoka. Issues in evaluating beat tracking systems. In Working Notes of IJCAI-97 Workshop on Issues in AI and Music - Evaluation and Assessment, pages 9┬¡16, 1997 [http://staff.aist.go.jp/m.goto/PAPER/AIM97.300dpi.ps postscript]
# McKinney, M.F. and Moelants, D. (2004), Deviations from the resonance theory of tempo induction, Conference on Interdisciplinary Musicology, Graz. [http://gewi.kfunigraz.ac.at/~cim04/CIM04_paper_pdf/McKinney_Moelants_CIM04_proceedings_t.pdf pdf]
# Moelants, D. and McKinney, M.F. (2004), Tempo perception and musical content: What makes a piece slow, fast, or temporally ambiguous? International Conference on Music Perception & Cognition, Evanston, IL. [http://www.northwestern.edu/icmpc/proceedings/ICMPC8/PDF/AUTHOR/MP040237.PDF pdf]

== Comments ==
Leave comments here

==Practice Data==
You can find it here:

https://www.music-ir.org/evaluation/MIREX/data/2006/beat/

User: beattrack Password: b34trx

https://www.music-ir.org/evaluation/MIREX/data/2006/tempo/

User: tempo Password: t3mp0

Data has been uploaded in both .tgz and .zip format.

@@ Line 7: / Line 7: @@
 The original 2006 dataset contains 160 30-second excerpts (WAV format) used for the Audio Tempo and Beat contests in 2006.  Beat locations have been annotated in each excerpt by 40 different listeners (39 listeners for a few excerpts. The length of each excerpt is 30 seconds. These audio recordings were selected to provide a stable tempo value, a wide distribution of tempi values, and a large variety of instrumentation and musical styles. About 20% of the files contain non-binary meters, and a small number of examples contain changing meters.  One disadvantage of using this set for beat tracking is that the tempi are rather stable and this set will not test beat-tracking algorithms in their ability to track tempo changes.
-The second collection is comprised of 367 Chopin Mazurkas. The Mazurka dataset contains tempo changes so it will evaluate the ability of algorithms to track these.
+The second collection is comprised of 367 Chopin Mazurkas, represented as full audio tracks (WAV format). The Mazurka dataset contains tempo changes so it will evaluate the ability of algorithms to track these.
 === Audio Formats ===

← Older revision		Revision as of 09:27, 5 June 2010
Line 156:		Line 156:

	Data has been uploaded in both .tgz and .zip format.		Data has been uploaded in both .tgz and .zip format.
		+
		+	== Time and hardware limits ==
		+	Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.
		+
		+	A hard limit of 12 hours will be imposed on analysis times. Submissions exceeding this limit may not receive a result.
		+
		+	== Submission opening date ==
		+
		+	Friday 4th June 2010
		+
		+	== Submission closing date ==
		+	TBA

@@ Line 92: / Line 92: @@
 are described in the following technical report:
-M. E. P. Davies, N. Degara and M. D. Plumbley. "Evaluation methods for musical audio beat tracking algorithms". [https://music-ir.org/mirex/results/2009/beat/techreport_beateval.pdf ''Technical Report C4DM-TR-09-06''].
+ M. E. P. Davies, N. Degara and M. D. Plumbley. "Evaluation methods for musical audio beat tracking algorithms". [https://music-ir.org/mirex/results/2009/beat/techreport_beateval.pdf ''Technical Report C4DM-TR-09-06''].
 For further details on the specifics of the methods please refer to the
@@ Line 100: / Line 100: @@
 with a 70ms window.
-S. Dixon, "Onset detection revisited," in ''Proceedings of 9th
+ S. Dixon, "Onset detection revisited," in ''Proceedings of 9th
-International Conference on Digital Audio Effects (DAFx)'', Montreal,
+ International Conference on Digital Audio Effects (DAFx)'', Montreal,
-Canada, pp. 133-137, 2006.
+ Canada, pp. 133-137, 2006.
-S. Dixon, "Evaluation of audio beat tracking system beatroot," ''Journal
+ S. Dixon, "Evaluation of audio beat tracking system beatroot," ''Journal
-of New Music Research'', vol. 36, no. 1, pp. 39-51, 2007.
+ of New Music Research'', vol. 36, no. 1, pp. 39-51, 2007.
 *'''Cemgil''' - beat accuracy is calculated using a Gaussian error function
 with 40ms standard deviation.
-A. T. Cemgil, B. Kappen, P. Desain, and H. Honing, "On tempo tracking:
+ A. T. Cemgil, B. Kappen, P. Desain, and H. Honing, "On tempo tracking:
-Tempogram representation and Kalman filtering," ''Journal Of New Music
+ Tempogram representation and Kalman filtering," ''Journal Of New Music
-Research'', vol. 28, no. 4, pp. 259-273, 2001
+ Research'', vol. 28, no. 4, pp. 259-273, 2001
 *'''Goto''' - binary decision of correct or incorrect tracking based on
 statistical properties of a beat error sequence.
-M. Goto and Y. Muraoka, "Issues in evaluating beat tracking systems," in
+ M. Goto and Y. Muraoka, "Issues in evaluating beat tracking systems," in
-''Working Notes of the IJCAI-97 Workshop on Issues in AI and Music -
+ ''Working Notes of the IJCAI-97 Workshop on Issues in AI and Music -
-Evaluation and Assessment'', 1997, pp. 9-16.
+ Evaluation and Assessment'', 1997, pp. 9-16.
 *'''PScore''' - McKinney's impulse train cross-correlation method as used in
 .
-M. F. McKinney, D. Moelants, M. E. P. Davies, and A. Klapuri,
+ M. F. McKinney, D. Moelants, M. E. P. Davies, and A. Klapuri,
-"Evaluation of audio beat tracking and music tempo extraction
+ "Evaluation of audio beat tracking and music tempo extraction
-algorithms," ''Journal of New Music Research'', vol. 36, no. 1, pp. 1-16,
+ algorithms," ''Journal of New Music Research'', vol. 36, no. 1, pp. 1-16,
 .
 *'''CMLc''', '''CMLt''', '''AMLc''', '''AMLt''' - continuity-based evaluation methods based on
 the longest continuously correctly tracked section.
-S. Hainsworth, "Techniques for the automated analysis of musical audio,"
+ S. Hainsworth, "Techniques for the automated analysis of musical audio,"
-Ph.D. dissertation, Department of Engineering, Cambridge University,
+ Ph.D. dissertation, Department of Engineering, Cambridge University,
 .
-A. P. Klapuri, A. Eronen, and J. Astola, "Analysis of the meter of
+ A. P. Klapuri, A. Eronen, and J. Astola, "Analysis of the meter of
-acoustic musical signals," IEEE Transactions on Audio, Speech and
+ acoustic musical signals," IEEE Transactions on Audio, Speech and
-Language Processing, vol. 14, no. 1, pp. 342-355, 2006.
+ Language Processing, vol. 14, no. 1, pp. 342-355, 2006.
 *'''D''', '''Dg''' - information based criteria based on analysis of a beat error

@@ Line 102: / Line 102: @@
 S. Dixon, "Onset detection revisited," in ''Proceedings of 9th
 International Conference on Digital Audio Effects (DAFx)'', Montreal,
-Canada, pp. 133ΓÇô137, 2006.
+Canada, pp. 133-137, 2006.
 S. Dixon, "Evaluation of audio beat tracking system beatroot," ''Journal
-of New Music Research'', vol. 36, no. 1, pp. 39ΓÇô51, 2007.
+of New Music Research'', vol. 36, no. 1, pp. 39-51, 2007.
 *'''Cemgil''' - beat accuracy is calculated using a Gaussian error function
@@ Line 112: / Line 112: @@
 A. T. Cemgil, B. Kappen, P. Desain, and H. Honing, "On tempo tracking:
 Tempogram representation and Kalman filtering," ''Journal Of New Music
-Research'', vol. 28, no. 4, pp. 259ΓÇô273, 2001
+Research'', vol. 28, no. 4, pp. 259-273, 2001
 *'''Goto''' - binary decision of correct or incorrect tracking based on
@@ Line 119: / Line 119: @@
 M. Goto and Y. Muraoka, "Issues in evaluating beat tracking systems," in
 ''Working Notes of the IJCAI-97 Workshop on Issues in AI and Music -
-Evaluation and Assessment'', 1997, pp. 9ΓÇô16.
+Evaluation and Assessment'', 1997, pp. 9-16.
 *'''PScore''' - McKinney's impulse train cross-correlation method as used in
@@ Line 126: / Line 126: @@
 M. F. McKinney, D. Moelants, M. E. P. Davies, and A. Klapuri,
 "Evaluation of audio beat tracking and music tempo extraction
-algorithms," ''Journal of New Music Research'', vol. 36, no. 1, pp. 1ΓÇô16,
+algorithms," ''Journal of New Music Research'', vol. 36, no. 1, pp. 1-16,
 .
@@ Line 138: / Line 138: @@
 A. P. Klapuri, A. Eronen, and J. Astola, "Analysis of the meter of
 acoustic musical signals," IEEE Transactions on Audio, Speech and
-Language Processing, vol. 14, no. 1, pp. 342ΓÇô355, 2006.
+Language Processing, vol. 14, no. 1, pp. 342-355, 2006.
 *'''D''', '''Dg''' - information based criteria based on analysis of a beat error
 histogram (note the results are measured in 'bits' and not percentages),
 see the technical report for a description.
 == Relevant Development Collections ==

← Older revision		Revision as of 19:24, 24 May 2010
Line 145:		Line 145:


−	==~~Practice Data~~==	+	== Relevant Development Collections ==
	You can find it here:		You can find it here:

@@ Line 98: / Line 98: @@
 *'''F-measure''' - the standard calculation as used in onset evaluation but
-with a ┬▒ 70ms window.
+with a 70ms window.
 S. Dixon, "Onset detection revisited," in ''Proceedings of 9th

@@ Line 39: / Line 39: @@
 == Participants ==
-your name here
+Tsung-Chi Lee (Department of Computer Science, National Tsing-Hua University, Hsinchu, Taiwan)
 == Evaluation Procedures ==