Difference between revisions of "2006:Audio Onset Detection Results"

From MIREX Wiki
(→‎Team ID)
(→‎MIREX 2006 Audio Onset Detection Summary Plot)
 
(10 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
[[Category: Results]]
 
[[Category: Results]]
 
==Introduction==
 
==Introduction==
−
These are the results for the 2006 running of the Audio Onset Detection task set. For background information about this task set please refer to the [[Audio Onset Detection]] page.
+
These are the results for the 2006 running of the Audio Onset Detection task set. For background information about this task set please refer to the [[2006:Audio Onset Detection]] page.
 +
 
 +
The aim of the Audio Onset Detection task is to find the time locations at which all musical events in a recording begin. The dataset consists of 85 recordings across 9 different "classes" (e.g. solo drums, polyphonic pitched, etc.). For each sound file, ground truth annotations produced by 3-5 listeners were used for the evaluation. Each algorithm was tested across 10-20 different parameterizations (e.g. thresholds) in order to produce Precision vs. Recall Operating Characteristic (P-ROC) curves. The primary evauluation metric used was the F1-Measure (the equal weighted harmonic mean of precision and recall).  
  
 
===General Legend===
 
===General Legend===
 
====Team ID====
 
====Team ID====
  
−
'''dixon''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/OD_dixon.pdf Simon Dixon]<br />
+
'''dixon''' = [https://www.music-ir.org/mirex/abstracts/2006/OD_dixon.pdf Simon Dixon]<br />
−
'''roebel''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/OD_roebel.pdf A. R├╢bel]<br />
+
'''roebel''' = [https://www.music-ir.org/mirex/abstracts/2006/OD_roebel.pdf A. Röbel]<br />
−
'''brossier''' = Paul Brossier<br />
+
'''brossier''' = [https://www.music-ir.org/mirex/abstracts/2006/AME_BT_OD_TE_brossier.pdf Paul Brossier]<br />
−
'''du''' = Yunfeng Du, Ming Li, Jian Liu<br />
+
'''du''' = [https://www.music-ir.org/mirex/abstracts/2006/OD_du.pdf Yunfeng Du, Ming Li, Jian Liu]<br />
  
 
*Dixon's NWPD submission was modified by Andreas Ehmann, and requires the author's verification
 
*Dixon's NWPD submission was modified by Andreas Ehmann, and requires the author's verification
Line 16: Line 18:
 
===MIREX 2006 Audio Onset Detection Summary Results - Peak F-measure performance across all parameterizations===
 
===MIREX 2006 Audio Onset Detection Summary Results - Peak F-measure performance across all parameterizations===
  
−
<csv>onset06_sum.csv</csv>  
+
<csv>2006/onset06_sum.csv</csv>  
  
 
===MIREX 2006 Audio Onset Detection Summary Plot===
 
===MIREX 2006 Audio Onset Detection Summary Plot===
Line 24: Line 26:
 
===MIREX 2006 Audio Onset Detection Runtime Data===
 
===MIREX 2006 Audio Onset Detection Runtime Data===
  
−
<csv>onset06_runtime.csv</csv>
+
<csv>2006/onset06_runtime.csv</csv>
 +
 
 +
==Results by Class==
 +
*[[2006:Audio_Onset_Detection_Results:_Complex]]
 +
*[[2006:Audio_Onset_Detection_Results:_Poly_Pitched]]
 +
*[[2006:Audio_Onset_Detection_Results:_Solo_Bars_and_Bells]]
 +
*[[2006:Audio_Onset_Detection_Results:_Solo_Brass]]
 +
*[[2006:Audio_Onset_Detection_Results:_Solo_Drum]]
 +
*[[2006:Audio_Onset_Detection_Results:_Solo_Plucked_Strings]]
 +
*[[2006:Audio_Onset_Detection_Results:_Solo_Singing_Voice]]
 +
*[[2006:Audio_Onset_Detection_Results:_Solo_Sustained_Strings]]
 +
*[[2006:Audio_Onset_Detection_Results:_Solo_Winds]]
  
 
==Individual Results ==
 
==Individual Results ==
−
* [[Audio_Onset_Detection_Results:_Brossier_-_complex]]
+
* [[2006:Audio_Onset_Detection_Results:_Brossier_-_complex]]
−
*[[Audio_Onset_Detection_Results:_Brossier_-_dual]]
+
*[[2006:Audio_Onset_Detection_Results:_Brossier_-_dual]]
−
*[[Audio_Onset_Detection_Results:_Brossier_-_hfc]]
+
*[[2006:Audio_Onset_Detection_Results:_Brossier_-_hfc]]
−
*[[Audio_Onset_Detection_Results:_Brossier_-_specdiff]]
+
*[[2006:Audio_Onset_Detection_Results:_Brossier_-_specdiff]]
−
*[[Audio_Onset_Detection_Results:_Dixon_-_cd]]
+
*[[2006:Audio_Onset_Detection_Results:_Dixon_-_cd]]
−
*[[Audio_Onset_Detection_Results:_Dixon_-_nwpd]]
+
*[[2006:Audio_Onset_Detection_Results:_Dixon_-_nwpd]]
−
*[[Audio_Onset_Detection_Results:_Dixon_-_rcd]]
+
*[[2006:Audio_Onset_Detection_Results:_Dixon_-_rcd]]
−
*[[Audio_Onset_Detection_Results:_Dixon_-_sf]]
+
*[[2006:Audio_Onset_Detection_Results:_Dixon_-_sf]]
−
*[[Audio_Onset_Detection_Results:_Dixon_-_wpd]]
+
*[[2006:Audio_Onset_Detection_Results:_Dixon_-_wpd]]
−
*[[Audio_Onset_Detection_Results:_Du]]
+
*[[2006:Audio_Onset_Detection_Results:_Du]]
−
*[[Audio_Onset_Detection_Results:_Roebel_1]]
+
*[[2006:Audio_Onset_Detection_Results:_Roebel_1]]
−
*[[Audio_Onset_Detection_Results:_Roebel_2]]
+
*[[2006:Audio_Onset_Detection_Results:_Roebel_2]]
−
*[[Audio_Onset_Detection_Results:_Roebel_3]]
+
*[[2006:Audio_Onset_Detection_Results:_Roebel_3]]

Latest revision as of 23:58, 14 December 2011

Introduction

These are the results for the 2006 running of the Audio Onset Detection task set. For background information about this task set please refer to the 2006:Audio Onset Detection page.

The aim of the Audio Onset Detection task is to find the time locations at which all musical events in a recording begin. The dataset consists of 85 recordings across 9 different "classes" (e.g. solo drums, polyphonic pitched, etc.). For each sound file, ground truth annotations produced by 3-5 listeners were used for the evaluation. Each algorithm was tested across 10-20 different parameterizations (e.g. thresholds) in order to produce Precision vs. Recall Operating Characteristic (P-ROC) curves. The primary evauluation metric used was the F1-Measure (the equal weighted harmonic mean of precision and recall).

General Legend

Team ID

dixon = Simon Dixon
roebel = A. Röbel
brossier = Paul Brossier
du = Yunfeng Du, Ming Li, Jian Liu

  • Dixon's NWPD submission was modified by Andreas Ehmann, and requires the author's verification

Overall Summary Results

MIREX 2006 Audio Onset Detection Summary Results - Peak F-measure performance across all parameterizations

Contestant Parameters Total Correct Total FP Total FN Total Merged Total Doubled Avg. Correct Avg. FP Avg. FN Avg. Merged Avg. Doubled Avg. Precision Avg. Recall Avg. F-Measure
brossier_complex 0.45 6407 1709 3092 133 387 22.169 6.067 9.5 0.429 1.3 0.78 0.725 0.721
brossier_dual 0.4 6930 1979 2569 109 869 23.271 6.459 8.398 0.347 2.777 0.769 0.735 0.724
brossier_hfc 0.25 7368 2573 2131 115 884 24.645 8.402 7.024 0.358 2.706 0.752 0.774 0.734
brossier_specdiff 0.4 6475 1757 3024 126 481 21.963 5.731 9.705 0.394 1.515 0.764 0.701 0.707
dixon_cd (0.85/ 0.30) 6945 3948 2554 172 120 23.94 13.319 7.729 0.536 0.408 0.709 0.776 0.71
dixon_nwpd (0.89/ 0.60) 8460 10431 1039 176 820 28.522 35.842 3.146 0.551 2.693 0.524 0.908 0.62
dixon_rcd (0.88/ 0.70) 6867 3014 2632 161 167 23.598 10.202 8.071 0.492 0.591 0.735 0.765 0.716
dixon_sf (0.90/ 0.55) 7217 3684 2282 176 159 24.655 12.369 7.013 0.538 0.536 0.736 0.79 0.726
dixon_wpd (0.89/ 0.65) 7435 4820 2064 187 335 25.204 15.749 6.464 0.576 1.071 0.663 0.786 0.685
du 1.3 7353 3142 2146 193 7 24.697 10.491 6.971 0.597 0.023 0.797 0.799 0.762
roebel_1 0.09 6795 1005 2704 192 28 23.104 3.367 8.565 0.591 0.086 0.861 0.746 0.777
roebel_2 0.09 7048 1282 2451 201 50 23.885 4.279 7.783 0.619 0.146 0.831 0.769 0.78
roebel_3 0.06 7173 1323 2326 200 51 24.321 4.385 7.347 0.612 0.153 0.836 0.779 0.788

download these results as csv

MIREX 2006 Audio Onset Detection Summary Plot

File:Onset06 summary.png

MIREX 2006 Audio Onset Detection Runtime Data

Contestant Machine Avg. run time per parameter set
brossier LINUX 34
dixon FAST 966
du FAST 64
roebel LINUX 327

download these results as csv

Results by Class

Individual Results

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2006:Audio_Onset_Detection_Results&oldid=8596"