2010:Audio Onset Detection - Revision history

Kriswest at 09:20, 5 June 2010

2010-06-05T09:20:01Z

Kriswest at 09:16, 5 June 2010

2010-06-05T09:16:24Z

AndreasEhmann: moved 2010:Audio Onset detection to 2010:Audio Onset Detection: nicer capitalization

2010-05-26T18:24:57Z

moved 2010:Audio Onset detection to 2010:Audio Onset Detection: nicer capitalization

Kriswest: /* Evaluation measures */

2010-05-21T15:33:16Z

‎Evaluation measures

Kriswest: /* Evaluation measures: */

2010-05-21T15:32:50Z

‎Evaluation measures:

AndreasEhmann: /* Evaluation Procedures */

2010-05-20T20:46:51Z

‎Evaluation Procedures

AndreasEhmann: /* Evaluation Procedures */

2010-05-20T20:45:46Z

‎Evaluation Procedures

AndreasEhmann: /* Evaluation procedures */

2010-05-20T20:26:57Z

‎Evaluation procedures

AndreasEhmann: /* Submission Format */

2010-05-20T19:46:23Z

‎Submission Format

AndreasEhmann: /* Output Data */

2010-05-20T17:49:01Z

‎Output Data

@@ Line 2: / Line 2: @@
 Audio Onset Detection concerns itself with finding the time-locations of all sonic events in a piece of audio. This task was originally proposed in 2005 by Paul Brossier and Pierre Leveau . It has subsequently been run in 2005, 2006, 2007, 2009.
 == Data ==
@@ Line 15: / Line 16: @@
 Moreover the monophonic pitched instruments class is divided into 6 sub-classes: brass (2 excerpts), winds (4), sustained strings (6), plucked strings (9), bars and bells (4), singing voice (5).
 === Audio Formats ===
@@ Line 23: / Line 25: @@
 * single channel (mono)
 * file length between 2 and 36 seconds (total time: 14 minutes)
@@ Line 73: / Line 112: @@
 === Parameter Sweeps ===
 In past iterations of MIREX, submitters have been allowed to specify a parameter sweep so as to generate a precision-recall operator characteristic to better evaluate and understand the algorithm. If you wish to do so, please specify TEN different settings for your sweepable parameter. There are no guarantees that all ten will be tested and evaluated, however, as the time-constraints for MIREX are getting ever smaller as the number of submissions are getting ever larger. Therefore, please also specify the ONE single parameterization you feel is best in the README. If the whole parameter sweep cannot be evaluated, this single parameterization will be used.
-== Packaging submissions ==
+=== Packaging submissions ===
 * All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).
@@ Line 130: / Line 133: @@
-=== README File ===
+==== README File ====
 A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.

@@ Line 40: / Line 40: @@
 The onset detection algorithms will return onset times in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.
-=== Output File Format (Audio Onset Detection) ===
+==== Output File Format (Audio Onset Detection) ====
 The Audio Onset Detection output file format is an ASCII text format. Each onset time is specified, in seconds, on its own line. Specifically,
@@ Line 70: / Line 70: @@
   foobar('%input','%output')
   foobar(.1,'%input','%output')
 === Parameter Sweeps ===
@@ Line 134: / Line 109: @@
 * percentage of doubled detections
 * speed measurements of the algorithms

@@ Line 132: / Line 132: @@
 * time precision (tolerance from +/- 50 ms to less). For certain file, we can't be much more accurate than 50 ms because of the weak annotation precision. This must be taken into account.
 * separate scoring for different instrument types (percussive, strings, winds, etc)
 * percentage of doubled detections
 * speed measurements of the algorithms

@@ Line 127: / Line 127: @@
 To establish a ranking, we will use the F-measure, widely used in string comparisons. This criterion is arbitrary, but gives an indication of performance. It must be remembered that onset detection is a preprocessing step, so the real cost of an error of each type (false positive or false negative) depends on the application following this task.
-=== Evaluation measures: ===
+=== Evaluation measures ===
 * percentage of correct detections / false positives (can also be expressed as precision/recall)
@@ Line 133: / Line 133: @@
 * separate scoring for different instrument types (percussive, strings, winds, etc)
-==== More detailed data: ====
+==== More detailed data ====
 * percentage of doubled detections
 * speed measurements of the algorithms

@@ Line 109: / Line 109: @@
 *'''F-measure''' F = 2*P*R/(P+R)
-'''with these notations:'''
+with these notations:
 *'''Ocd''' number of correctly detected onsets (CD)
@@ Line 117: / Line 117: @@
 *'''Od''' number of double onsets
-==== Other indicative measurements: ====
+Other indicative measurements:
 *'''FP rate''' FP = 100. * (Ofp) / (Ocd+Ofp)

← Older revision	Revision as of 18:24, 26 May 2010
(No difference)

@@ Line 103: / Line 103: @@
 The detected onset times will be compared with the ground-truth ones. For a given ground-truth onset time, if there is a detection in a tolerance time-window around it, it is considered as a correct detection (CD). If not, there is a false negative (FN). The detections outside all the tolerance windows are counted as false positives (FP). Doubled onsets (two detections for one ground-truth onset) and merged onsets (one detection for two ground-truth onsets) will be taken into account in the evaluation. Doubled onsets are a subset of the FP onsets, and merged onsets a subset of FN onsets.
-'''We define:'''
+We define:
-'''Precision'''
+*'''Precision''' P = Ocd / (Ocd +Ofp)
-    P = Ocd / (Ocd +Ofp)
+*'''Recall''' R = Ocd / (Ocd + Ofn)
-'''Recall'''
+*'''F-measure''' F = 2*P*R/(P+R)
 '''with these notations:'''
-'''Ocd'''
+*'''Ocd''' number of correctly detected onsets (CD)
-    number of correctly detected onsets (CD)
+*'''Ofn''' number of missed onsets (FN)
-'''Ofn'''
+*'''Om''' number of merged onsets
-    number of missed onsets (FN)
+*'''Ofp''' number of false positive onsets (FP)
-'''Om'''
+*'''Od''' number of double onsets
 ==== Other indicative measurements: ====
-'''FP rate'''
+*'''FP rate''' FP = 100. * (Ofp) / (Ocd+Ofp)
-    FP = 100. * (Ofp) / (Ocd+Ofp)
+*'''Doubled Onset rate in FP''' D = 100 * Od / Ofp
-'''Doubled Onset rate in FP'''
+*'''Merged Onset rate in FN''' M = 100 * Om / Ofn
 Because files are cross-annotated, the mean Precision and Recall rates are defined by averaging Precision and Recall rates computed for each annotation.

@@ Line 99: / Line 99: @@
 In past iterations of MIREX, submitters have been allowed to specify a parameter sweep so as to generate a precision-recall operator characteristic to better evaluate and understand the algorithm. If you wish to do so, please specify TEN different settings for your sweepable parameter. There are no guarantees that all ten will be tested and evaluated, however, as the time-constraints for MIREX are getting ever smaller as the number of submissions are getting ever larger. Therefore, please also specify the ONE single parameterization you feel is best in the README. If the whole parameter sweep cannot be evaluated, this single parameterization will be used.
-== Evaluation procedures ==
+== Evaluation Procedures ==
 The detected onset times will be compared with the ground-truth ones. For a given ground-truth onset time, if there is a detection in a tolerance time-window around it, it is considered as a correct detection (CD). If not, there is a false negative (FN). The detections outside all the tolerance windows are counted as false positives (FP). Doubled onsets (two detections for one ground-truth onset) and merged onsets (one detection for two ground-truth onsets) will be taken into account in the evaluation. Doubled onsets are a subset of the FP onsets, and merged onsets a subset of FN onsets.

@@ Line 56: / Line 56: @@
 === Algorithm Calling Format ===
-The submitted algorithm must take as arguments a SINGLE .wav file to perform the onset detection on as well as the full output path and filename of the output file. Specifying the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:
+The submitted algorithm must take as arguments a SINGLE .wav file to perform the onset detection on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:
   foobar %input %output
@@ Line 74: / Line 74: @@
 === README File ===
-A README file accompanying each submission should contain explicit instructions on how to to run the program. In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.
+A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.
 For instance, to test the program foobar with different values for parameters param1, the README file would look like:

@@ Line 38: / Line 38: @@
 === Output Data ===
-The onset detection algorithms will return onset times in an ASCII text file. The specification is immediately below.
+The onset detection algorithms will return onset times in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.
 === Output File Format (Audio Onset Detection) ===