Difference between revisions of "2009:Audio Melody Extraction Results"

From MIREX Wiki
(→‎Overall Summary Results)
(→‎Team ID)
 
(17 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
== Introduction ==
 
== Introduction ==
−
 
+
These are the results for the 2009 running of the Audio Melody Extraction task set. For background information about this task set please refer to the [[2009:Audio Melody Extraction]] page. There are multiple datasets for this task. ADC04 consists of 20 pieces with vocal and nonvocal melodies. MIREX05 consists of 25 pieces with vocal and nonvocal melodies. MIREX08 consists of 8 pieces all with vocal melodies. MIREX09 consists of 374 pieces all with vocal melodies. MIREX09 is mixed with three different Melodic Voice - to accompaniment ratios. +5dB, 0dB, and -5 dB RMS.
  
 
=== General Legend ===
 
=== General Legend ===
Line 7: Line 7:
 
==== Team ID ====
 
==== Team ID ====
  
−
'''CL1'''   [[Chuan Cao, Ming Li]]
+
'''CL1''' = [https://www.music-ir.org/mirex/abstracts/2009/CLmelody.pdf Chuan Cao, Ming Li]<br />
−
 
+
'''CL2''' = [https://www.music-ir.org/mirex/abstracts/2009/CLmelody.pdf Chuan Cao, Ming Li]<br />
−
'''CL2'''   [[Chuan Cao, Ming Li]]
+
'''DR1''' = [https://www.music-ir.org/mirex/abstracts/2009/DR.pdf Jean-Louis Durrieu, Gaël Richard, Bertrand David (GSMM)]<br />
−
 
+
'''DR2''' = [https://www.music-ir.org/mirex/abstracts/2009/DR.pdf Jean-Louis Durrieu, Gaël Richard, Bertrand David (SIMM)]<br />
−
'''DR1'''   [[Jean-Louis Durrieu, Ga├½l Richard (GSMM)]]
+
'''HJC1''' = [https://www.music-ir.org/mirex/abstracts/2009/HJC.pdf Chao-Ling Hsu, Jyh-Shing Roger Jang, Liang-Yu Chen (DP)]<br />
−
 
+
'''HJC2''' =  [https://www.music-ir.org/mirex/abstracts/2009/HJC.pdf Chao-Ling Hsu, Jyh-Shing Roger Jang, Liang-Yu Chen (HMM)]<br />
−
'''DR2'''   [[Jean-Louis Durrieu, Ga├½l Richard (SIMM)]]
+
'''JJY''' = [https://www.music-ir.org/mirex/abstracts/2009/JJY.pdf Sihyun Joo, Seokhwan Jo, Chang D. Yoo]<br />
−
 
+
'''KD''' = [https://www.music-ir.org/mirex/abstracts/2009/KD.pdf Karin Dressler]<br />
−
'''HJC1'''   [[Chao-Ling Hsu, Jyh-Shing Roger Jang, Liang-Yu Chen (DP)]]
+
'''MW''' = [http://morwen.dk/university/master-thesis/Melody-Extraction-MIREX2009_Morten-Wendelboe.pdf Morten Wendelboe]<br />
−
 
+
'''PC''' = [https://www.music-ir.org/mirex/abstracts/2009/PC.txt Pablo Cancela]<br />
−
'''HJC2'''   [[Chao-Ling Hsu, Jyh-Shing Roger Jang, Liang-Yu Chen (HMM)]]
+
'''RR''' = [https://www.music-ir.org/mirex/abstracts/2009/RR.pdf Vishweshwara Rao,Preeti Rao]<br />
−
 
+
'''TOOS''' = [https://www.music-ir.org/mirex/abstracts/2009/TOOS.pdf Hideyuki Tachibana, Takuma Ono, Nobutaka Ono, Shigeki Sagayama]<br />
−
'''JJY'''   [[Sihyun Joo, Seokhwan Jo, Chang D. Yoo]]
 
−
 
 
−
'''KD'''   [[Karin Dressler]]
 
−
 
 
−
'''MW'''   [[Morten Wendelboe]]
 
−
 
 
−
'''PC'''   [[Pablo Cancela]]
 
−
 
 
−
'''RR'''   [[Vishweshwara Rao,Preeti Rao]]
 
−
 
 
−
'''TOOS''' [[Hideyuki Tachibana, Takuma Ono, Nobutaka Ono and Shigeki Sagayama]]
 
  
 
====Table Headings====
 
====Table Headings====
Line 39: Line 28:
  
 
==Overall Summary Results==
 
==Overall Summary Results==
 +
===MIREX 2009 Audio Melody Extraction Overall Summary results - Unweighted Avg. of all Datasets===
 +
 +
<csv>2009/melody/melody.unweighted_avg.csv</csv>
  
 
===MIREX 2009 Audio Melody Extraction Overall Summary results - Weighted (by Number of Files) Avg. of all Datasets===
 
===MIREX 2009 Audio Melody Extraction Overall Summary results - Weighted (by Number of Files) Avg. of all Datasets===
−
<csv>melody/melody.weighted_avg.csv</csv>
+
Note, because the MIREX09 dataset is so large in comparison, the weighted average is heavily weighted to performance on this dataset.
−
 
+
−
===MIREX 2009 Audio Melody Extraction Overall Summary results - Average Over Datasets (Unweighted)===
+
<csv>2009/melody/melody.weighted_avg.csv</csv>
−
<csv>melody/melody.unweighted_avg.csv</csv>
 
  
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2009 Dataset - -5dB mix===
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2009 Dataset - -5dB mix===
  
−
<csv>melody/melody.mir09_m5db.csv</csv>
+
<csv>2009/melody/melody.mir09_m5db.csv</csv>
  
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2009 Dataset - 0dB mix===
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2009 Dataset - 0dB mix===
  
−
<csv>melody/melody.mir09_0db.csv</csv>
+
<csv>2009/melody/melody.mir09_0db.csv</csv>
  
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2009 Dataset - +5dB mix===
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2009 Dataset - +5dB mix===
  
−
<csv>melody/melody.mir09_p5db.csv</csv>
+
<csv>2009/melody/melody.mir09_p5db.csv</csv>
  
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2008 Dataset - All===
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2008 Dataset - All===
  
−
<csv>melody/melody.ind08.csv</csv>
+
<csv>2009/melody/melody.ind08.csv</csv>
  
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2005 Dataset - vocal===
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2005 Dataset - vocal===
  
−
<csv>melody/melody.mir05_vocal.csv</csv>
+
<csv>2009/melody/melody.mir05_vocal.csv</csv>
  
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2005 Dataset - nonvocal===
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2005 Dataset - nonvocal===
  
−
<csv>melody/melody.mir05_nonvocal.csv</csv>
+
<csv>2009/melody/melody.mir05_nonvocal.csv</csv>
  
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2005 Dataset - All===
 
===MIREX 2009 Audio Melody Extraction Summary results - MIREX 2005 Dataset - All===
  
−
<csv>melody/melody.mir05_all.csv</csv>
+
<csv>2009/melody/melody.mir05_all.csv</csv>
  
 
===MIREX 2009 Audio Melody Extraction Summary results - ADC 2004 Dataset - vocal===
 
===MIREX 2009 Audio Melody Extraction Summary results - ADC 2004 Dataset - vocal===
  
−
<csv>melody/melody.adc04_vocal.csv</csv>
+
<csv>2009/melody/melody.adc04_vocal.csv</csv>
  
 
===MIREX 2009 Audio Melody Extraction Summary results - ADC 2004 Dataset - nonvocal===
 
===MIREX 2009 Audio Melody Extraction Summary results - ADC 2004 Dataset - nonvocal===
  
−
<csv>melody/melody.adc04_nonvocal.csv</csv>
+
<csv>2009/melody/melody.adc04_nonvocal.csv</csv>
  
 
===MIREX 2009 Audio Melody Extraction Summary results - ADC 2004 Dataset - All===
 
===MIREX 2009 Audio Melody Extraction Summary results - ADC 2004 Dataset - All===
  
−
<csv>melody/melody.adc04_all.csv</csv>
+
<csv>2009/melody/melody.adc04_all.csv</csv>
  
 
===MIREX 2009 Audio Melody Extraction Runtime Data===
 
===MIREX 2009 Audio Melody Extraction Runtime Data===
−
TBA
+
<csv>2009/melody/melody.runtime.csv</csv>
 +
 
 
[[Category: Results]]
 
[[Category: Results]]

Latest revision as of 15:27, 23 July 2010

Introduction

These are the results for the 2009 running of the Audio Melody Extraction task set. For background information about this task set please refer to the 2009:Audio Melody Extraction page. There are multiple datasets for this task. ADC04 consists of 20 pieces with vocal and nonvocal melodies. MIREX05 consists of 25 pieces with vocal and nonvocal melodies. MIREX08 consists of 8 pieces all with vocal melodies. MIREX09 consists of 374 pieces all with vocal melodies. MIREX09 is mixed with three different Melodic Voice - to accompaniment ratios. +5dB, 0dB, and -5 dB RMS.

General Legend

Team ID

CL1 = Chuan Cao, Ming Li
CL2 = Chuan Cao, Ming Li
DR1 = Jean-Louis Durrieu, Gaël Richard, Bertrand David (GSMM)
DR2 = Jean-Louis Durrieu, Gaël Richard, Bertrand David (SIMM)
HJC1 = Chao-Ling Hsu, Jyh-Shing Roger Jang, Liang-Yu Chen (DP)
HJC2 = Chao-Ling Hsu, Jyh-Shing Roger Jang, Liang-Yu Chen (HMM)
JJY = Sihyun Joo, Seokhwan Jo, Chang D. Yoo
KD = Karin Dressler
MW = Morten Wendelboe
PC = Pablo Cancela
RR = Vishweshwara Rao,Preeti Rao
TOOS = Hideyuki Tachibana, Takuma Ono, Nobutaka Ono, Shigeki Sagayama

Table Headings

Vx Recall = Voicing Detection
Vx False Alm = Voicing False Alarm
Raw pitch = Raw Pitch Accuracy
Raw Chroma = Raw Chroma Accuracy
Overall Acc = Overall Acuuracy

Overall Summary Results

MIREX 2009 Audio Melody Extraction Overall Summary results - Unweighted Avg. of all Datasets

Participant Vx Recall(%) Vx False Alm(%) Raw Pitch(%) Raw Chroma(%) Overall Acc(%)
cl1 93.0122 80.7121 63.4537 66.2928 52.192
cl2 80.2955 57.4239 63.4537 66.2928 55.1912
dr1 92.4039 51.7437 74.4454 76.8213 66.8572
dr2 87.687 41.2182 72.0925 75.7223 66.172
hjc1 43.6213 9.7101 66.1219 72.581 50.4858
hjc2 43.6213 9.7101 51.1347 67.1245 49.0116
jjy 61.023 29.3892 73.3253 79.6779 56.6353
kd 90.9304 40.9859 80.5833 82.5201 73.3523
mw 99.9724 98.6594 73.4386 77.4986 55.0722
pc 79.3165 40.292 64.1009 65.8397 62.8812
rr 91.28 51.1058 72.2126 76.3331 65.2212
toos 99.8687 98.3127 75.0514 80.3403 55.0774

download these results as csv

MIREX 2009 Audio Melody Extraction Overall Summary results - Weighted (by Number of Files) Avg. of all Datasets

Note, because the MIREX09 dataset is so large in comparison, the weighted average is heavily weighted to performance on this dataset.

Participant Vx Recall(%) Vx False Alm(%) Raw Pitch(%) Raw Chroma(%) Overall Acc(%)
cl1 92.3264 83.1113 58.9205 62.7661 44.3575
cl2 77.2452 59.2516 58.9205 62.7661 49.5138
dr1 91.7457 53.7476 68.6411 71.3884 60.0492
dr2 87.2511 45.2327 65.3401 69.8261 59.5668
hjc1 37.8763 2.8033 68.4293 72.0766 54.9707
hjc2 37.8763 2.8033 50.4366 66.9551 54.1855
jjy 38.0818 17.8244 73.0217 77.7374 48.6962
kd 90.5445 47.2188 77.5947 79.5145 66.7341
mw 99.9896 99.3837 66.4069 70.34 43.6651
pc 73.4387 42.4767 51.778 54.1777 52.5412
rr 89.2142 49.0418 67.4012 70.4479 60.618
toos 99.8103 99.1475 80.0558 83.7591 52.711

download these results as csv

MIREX 2009 Audio Melody Extraction Summary results - MIREX 2009 Dataset - -5dB mix

Participant Vx Recall(%) Vx False Alm(%) Raw Pitch(%) Raw Chroma(%) Overall Acc(%)
cl1 90.6211 84.8568 45.3909 50.9368 34.504
cl2 72.8248 61.2662 45.3909 50.9368 39.9455
dr1 88.5333 67.0915 53.7796 58.0902 45.5482
dr2 85.3196 60.2824 50.5318 57.5288 44.7759
hjc1 8.5966 1.1051 48.658 54.5224 38.4559
hjc2 8.5966 1.1051 21.4502 44.5494 37.5497
jjy 39.0921 26.4556 58.5304 64.7866 42.2335
kd 86.3897 60.1029 62.4877 66.2816 51.6864
mw 99.992 99.4688 53.0621 58.7232 34.2691
pc 67.6162 48.1973 37.3777 40.5081 41.6186
rr 92.7872 75.7569 54.6785 58.7592 43.3962
toos 99.9829 99.4185 74.8896 78.5338 48.6449

download these results as csv

MIREX 2009 Audio Melody Extraction Summary results - MIREX 2009 Dataset - 0dB mix

Participant Vx Recall(%) Vx False Alm(%) Raw Pitch(%) Raw Chroma(%) Overall Acc(%)
cl1 92.4858 83.5749 59.138 62.9508 43.9659
cl2 77.2085 59.7352 59.138 62.9508 49.2294
dr1 91.8716 55.3555 69.8804 72.5138 60.1294
dr2 87.3985 47.3422 66.549 70.7923 59.5076
hjc1 34.1722 1.7909 72.6577 75.2906 53.1752
hjc2 34.1722 1.7909 51.6871 70.002 51.7469
jjy 38.906 19.4063 75.9354 80.2461 49.686
kd 91.1846 47.7842 80.4565 81.8811 68.2237
mw 99.992 99.4688 67.2905 71.0018 43.6365
pc 73.1175 43.4773 50.8895 53.3672 51.5001
rr 88.8091 50.7595 68.6242 71.3714 60.7733
toos 99.9829 99.4185 82.2943 85.7474 53.5623

download these results as csv

MIREX 2009 Audio Melody Extraction Summary results - MIREX 2009 Dataset - +5dB mix

Participant Vx Recall(%) Vx False Alm(%) Raw Pitch(%) Raw Chroma(%) Overall Acc(%)
cl1 93.6769 82.0974 70.2637 72.6758 51.7453
cl2 80.8723 57.6872 70.2637 72.6758 57.1658
dr1 94.785 39.3475 80.8947 82.2161 72.7971
dr2 89.1418 29.2436 77.2989 79.6474 72.7917
hjc1 69.3829 2.9927 84.8561 86.3581 74.7639
hjc2 69.3829 2.9927 78.3801 86.5939 74.9723
jjy 29.378 3.9263 84.3853 87.6795 51.7425
kd 94.1297 36.2944 89.1898 89.6585 78.4061
mw 99.992 99.4688 77.0268 79.3507 50.029
pc 78.2207 37.4586 63.6794 65.4185 61.5177
rr 85.4713 19.6651 77.8827 79.6838 76.7448
toos 99.4507 98.8844 84.8473 88.289 55.6746

download these results as csv

MIREX 2009 Audio Melody Extraction Summary results - MIREX 2008 Dataset - All

Participant Vx Recall(%) Vx False Alm(%) Raw Pitch(%) Raw Chroma(%) Overall Acc(%)
cl1 93.3948 90.5445 50.8092 51.3331 45.3441
cl2 84.4221 66.1472 50.8092 51.3331 46.8145
dr1 96.9986 51.0862 88.0057 88.1693 81.1762
dr2 94.1888 39.2986 86.5807 86.808 80.0331
hjc1 55.9319 7.7332 67.5624 74.8665 48.0678
hjc2 55.9319 7.7332 60.8391 74.8106 46.5056
jjy 85.1639 38.9125 68.2963 81.8753 61.1604
kd 95.0744 53.3054 87.82 88.8227 80.6539
mw 100 99.8667 85.9869 88.8665 73.4976
pc 94.0546 66.0993 81.8281 81.9825 73.6415
rr 94.8175 48.1976 86.161 86.6712 78.9667
toos 100 99.2636 79.7606 83.6605 68.5002

download these results as csv

MIREX 2009 Audio Melody Extraction Summary results - MIREX 2005 Dataset - vocal

Participant Vx Recall(%) Vx False Alm(%) Raw Pitch(%) Raw Chroma(%) Overall Acc(%)
cl1 91.2267 67.7071 70.807 73.924 59.7095
cl2 80.9512 44.8062 70.807 73.924 64.4609
dr1 93.7543 53.5255 76.1145 77.7138 66.9613
dr2 88.0635 38.0207 70.9258 75.9161 66.5175
hjc1 65.8441 19.8206 62.6594 73.4868 54.8538
hjc2 65.8441 19.8206 54.1294 69.4221 52.2905
jjy 88.8546 41.9813 76.2696 79.3236 66.3123
kd 82.6017 15.2554 77.4622 80.8218 76.962
mw 99.9345 99.7947 75.7398 80.3791 53.7323
pc 75.6436 21.0879 71.7068 72.5089 70.465
rr 92.908 56.3639 75.9506 79.1084 65.7745
toos 99.883 99.5427 73.4258 77.8075 52.1017

download these results as csv

MIREX 2009 Audio Melody Extraction Summary results - MIREX 2005 Dataset - nonvocal

Participant Vx Recall(%) Vx False Alm(%) Raw Pitch(%) Raw Chroma(%) Overall Acc(%)
cl1 94.1197 84.9564 68.7094 72.7874 63.2993
cl2 78.3576 69.1015 68.7094 72.7874 59.2786
dr1 88.5869 61.4615 66.6591 74.3354 63.7449
dr2 82.038 56.025 69.5099 76.0359 62.9567
hjc1 16.0485 40.823 52.8876 66.1191 16.5249
hjc2 16.0485 40.823 25.8824 48.5695 14.5299
jjy 83.2623 69.2165 57.5975 71.4552 48.4402
kd 95.2984 63.5702 74.6221 80.9244 71.0659
mw 99.8919 99.9457 73.7136 81.1205 66.1035
pc 82.8826 48.6806 61.3362 66.5582 59.4657
rr 95.3665 78.4275 56.781 71.6471 51.7068
toos 99.8969 100 56.9256 67.3221 50.6871

download these results as csv

MIREX 2009 Audio Melody Extraction Summary results - MIREX 2005 Dataset - All

Participant Vx Recall(%) Vx False Alm(%) Raw Pitch(%) Raw Chroma(%) Overall Acc(%)
cl1 92.2682 73.9168 70.0518 73.5148 61.0018
cl2 80.0175 53.5525 70.0518 73.5148 62.5953
dr1 91.894 56.3825 72.7105 76.4976 65.8034
dr2 85.8943 44.5023 70.4161 75.9592 65.2356
hjc1 47.9177 27.3815 59.1416 70.8344 41.0554
hjc2 47.9177 27.3815 43.9605 61.9152 38.6967
jjy 86.8414 51.786 69.5477 76.491 59.8783
kd 87.1725 32.6487 76.4398 80.8588 74.8394
mw 99.9191 99.8491 75.0104 80.646 58.1859
pc 78.2496 31.0213 67.9734 70.3666 66.5052
rr 93.7931 64.3068 69.0496 76.4223 60.7101
toos 99.888 99.7073 67.4857 74.0327 51.5925

download these results as csv

MIREX 2009 Audio Melody Extraction Summary results - ADC 2004 Dataset - vocal

Participant Vx Recall(%) Vx False Alm(%) Raw Pitch(%) Raw Chroma(%) Overall Acc(%)
cl1 95.8377 79.3617 85.6252 86.2052 75.1569
cl2 88.0757 51.9066 85.6252 86.2052 75.3229
dr1 92.3581 49.2541 86.9605 87.398 79.9939
dr2 86.6218 28.4689 83.2614 85.1916 78.0742
hjc1 50.0165 25.3791 63.1101 74.101 48.765
hjc2 50.0165 25.3791 46.5192 64.4711 44.9189
jjy 85.8299 39.0501 81.9596 85.7982 74.6814
kd 91.6258 14.9754 85.9698 86.4237 85.8694
mw 99.9368 98.4374 83.1351 86.593 70.9614
pc 88.781 19.3581 86.9624 87.5452 85.9451
rr 92.8267 59.1767 81.446 88.0381 73.7674
toos 99.9052 97.7855 59.7683 72.1289 50.745

download these results as csv

MIREX 2009 Audio Melody Extraction Summary results - ADC 2004 Dataset - nonvocal

Participant Vx Recall(%) Vx False Alm(%) Raw Pitch(%) Raw Chroma(%) Overall Acc(%)
cl1 95.3094 54.1628 84.2344 86.5563 78.7418
cl2 83.9553 37.5275 84.2344 86.5563 75.5083
dr1 87.315 29.117 73.0634 77.5043 69.2323
dr2 80.5139 23.8977 78.0547 81.2086 69.6093
hjc1 39.2916 5.0734 64.9745 72.8834 45.3448
hjc2 39.2916 5.0734 56.4498 65.483 44.1171
jjy 88.1466 31.0457 85.2026 88.7742 75.7553
kd 91.6407 16.9873 88.8104 89.4088 86.9568
mw 99.9436 86.9298 80.9344 86.1186 70.596
pc 78.4299 9.7087 76.6996 77.1712 77.3419
rr 90.7642 31.1078 70.0296 80.6697 66.189
toos 99.9114 86.2809 62.9242 71.2532 55.1078

download these results as csv

MIREX 2009 Audio Melody Extraction Summary results - ADC 2004 Dataset - All

Participant Vx Recall(%) Vx False Alm(%) Raw Pitch(%) Raw Chroma(%) Overall Acc(%)
cl1 95.6264 69.2822 85.0689 86.3457 76.5909
cl2 86.4276 46.155 85.0689 86.3457 75.397
dr1 90.3409 41.1992 81.4017 83.4406 75.6892
dr2 84.1787 26.6404 81.1787 83.5984 74.6882
hjc1 45.7265 17.2569 63.8558 73.614 47.3969
hjc2 45.7265 17.2569 50.4915 64.8759 44.5982
jjy 86.7566 35.8484 83.2568 86.9886 75.111
kd 91.6318 15.7801 87.106 87.6177 86.3043
mw 99.9395 93.8344 82.2548 86.4033 70.8152
pc 84.6406 15.4983 82.8573 83.3956 82.5038
rr 92.0017 47.9491 76.8794 85.0907 70.7361
toos 99.9076 93.1837 61.0307 71.7786 52.4901

download these results as csv

MIREX 2009 Audio Melody Extraction Runtime Data

Participant Machine Runtime (dd:hh:mm)
CL1 ALE 00:00:28
CL2 ALE 00:00:33
DR1 ALE 16:00:00
DR2 ALE 00:08:44
HJC1 FAST3 00:05:44
HJC2 FAST3 00:09:38
JJY ALE 02:14:06
KD ALE 00:00:24
MW ALE 00:02:12
PC BIGWIN 03:05:57
RR BIGWIN 00:00:26
TOOS ALE 01:00:28

download these results as csv

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2009:Audio_Melody_Extraction_Results&oldid=7363"