Difference between revisions of "2008:Audio Music Mood Classification Results"

From MIREX Wiki
m (Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2008:\1))
m (Robot: Automated text replacement (-<csv([^>]*)> +<csv\1>2008/))
Line 21: Line 21:
 
===MIREX 2008 Audio Mood Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds===
 
===MIREX 2008 Audio Mood Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds===
  
−
<csv>mood/audiomood.avg.results.csv</csv>
+
<csv>2008/mood/audiomood.avg.results.csv</csv>
  
 
=====Accuracy Across Folds=====
 
=====Accuracy Across Folds=====
  
−
<csv>mood/audiomood.results.fold.csv</csv>
+
<csv>2008/mood/audiomood.results.fold.csv</csv>
  
 
=====Accuracy Across Categories=====
 
=====Accuracy Across Categories=====
  
−
<csv>mood/audiomood.results.class.csv</csv>
+
<csv>2008/mood/audiomood.results.class.csv</csv>
  
 
===MIREX 2008 Audio Artist Classification Evaluation Logs and Confusion Matrices===
 
===MIREX 2008 Audio Artist Classification Evaluation Logs and Confusion Matrices===
Line 35: Line 35:
 
====MIREX 2008 Audio Mood Classification Run Times====
 
====MIREX 2008 Audio Mood Classification Run Times====
  
−
<csv>mood.runtime.csv</csv>
+
<csv>2008/mood.runtime.csv</csv>
  
 
====CSV Files Without Rounding====
 
====CSV Files Without Rounding====
Line 63: Line 63:
 
=====Friedman's Anova Table=====
 
=====Friedman's Anova Table=====
  
−
<csv>mood/perClassAccuracy.friedman.csv</csv>
+
<csv>2008/mood/perClassAccuracy.friedman.csv</csv>
  
 
=====Tukey-Kramer HSD Multi-Comparison=====
 
=====Tukey-Kramer HSD Multi-Comparison=====
Line 69: Line 69:
 
Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
 
Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
  
−
<csv>mood/perClassAccuracy.friedman.detail.csv</csv>
+
<csv>2008/mood/perClassAccuracy.friedman.detail.csv</csv>
  
 
[[Image:Mood.perClassAccuracy.friedman.tukeyKramerHSD.png]]
 
[[Image:Mood.perClassAccuracy.friedman.tukeyKramerHSD.png]]
Line 78: Line 78:
 
=====Friedman's Anova Table=====
 
=====Friedman's Anova Table=====
  
−
<csv>mood/perFoldAccuracy.friedman.csv</csv>
+
<csv>2008/mood/perFoldAccuracy.friedman.csv</csv>
  
 
=====Tukey-Kramer HSD Multi-Comparison=====
 
=====Tukey-Kramer HSD Multi-Comparison=====
Line 84: Line 84:
 
Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
 
Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
  
−
<csv>mood/perFoldAccuracy.friedman.detail.csv</csv>
+
<csv>2008/mood/perFoldAccuracy.friedman.detail.csv</csv>
  
 
[[Image:Mood.perFoldAccuracy.friedman.tukeyKramerHSD.png]]
 
[[Image:Mood.perFoldAccuracy.friedman.tukeyKramerHSD.png]]

Revision as of 18:49, 13 May 2010

Introduction

These are the results for the 2008 running of the Audio Music Mood Classification task. For background information about this task set please refer to the 2008:Audio Music Mood Classification page.

General Legend

Team ID

GP1 = G. Peeters
GT1 = G. Tzanetakis
GT2 = G. Tzanetakis
GT3 = G. Tzanetakis
HW = H. Wang
KL = K. Lee
LRPPI1 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 1
LRPPI2 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 2
LRPPI3 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 3
LRPPI4 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 4
ME1 = M. I. Mandel, D. P. W. Ellis 1
ME2 = M. I. Mandel, D. P. W. Ellis 2
ME3 = M. I. Mandel, D. P. W. Ellis 3

Overall Summary Results

MIREX 2008 Audio Mood Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds

Participant Average Classifcation Accuracy
GP1 63.67%
GT1 55.00%
GT2 52.50%
GT3 58.20%
HW 30.33%
KL 49.83%
LRPPI1 56.00%
LRPPI2 55.50%
LRPPI3 54.50%
LRPPI4 55.50%
ME1 50.33%
ME2 50.00%
ME3 49.67%

download these results as csv

Accuracy Across Folds
Classification fold GP1 GT1 GT2 GT3 HW KL LRPPI1 LRPPI2 LRPPI3 LRPPI4 ME1 ME2 ME3
0 0.715 0.630 0.565 0.679 0.365 0.515 0.660 0.610 0.635 0.625 0.545 0.540 0.535
1 0.610 0.550 0.535 0.549 0.315 0.520 0.480 0.505 0.485 0.510 0.510 0.505 0.510
2 0.585 0.470 0.475 0.518 0.230 0.460 0.540 0.550 0.515 0.530 0.455 0.455 0.445

download these results as csv

Accuracy Across Categories
Class GP1 GT1 GT2 GT3 HW KL LRPPI1 LRPPI2 LRPPI3 LRPPI4 ME1 ME2 ME3
1 0.517 0.542 0.633 0.408 0.175 0.250 0.467 0.492 0.533 0.517 0.450 0.450 0.450
2 0.517 0.550 0.350 0.492 0.100 0.367 0.508 0.467 0.475 0.450 0.358 0.350 0.350
3 0.833 0.775 0.683 0.758 0.817 0.842 0.775 0.792 0.750 0.783 0.550 0.550 0.550
4 0.500 0.417 0.467 0.492 0.100 0.250 0.425 0.417 0.408 0.400 0.492 0.483 0.475
5 0.817 0.467 0.492 0.798 0.325 0.783 0.625 0.608 0.558 0.625 0.667 0.667 0.658

download these results as csv

MIREX 2008 Audio Artist Classification Evaluation Logs and Confusion Matrices

MIREX 2008 Audio Mood Classification Run Times

Participant Runtime (hh:mm) / Fold
GP1 Feat Ex: 01:01 Train/Classify: 00:02
GT1 Feat Ex/Train/Classify: 00:03
GT2 Feat Ex/Train/Classify: 00:07
GT3 Feat Ex: 00:01 Train/Classify: 00:00 (1 sec)
HW Feat Ex/Train/Classify: 09:33
KL Feat Ex/Train/Classify: 00:09
LRPPI1 Feat Ex: 02:48 Train/Classify: 00:00 (11 sec)
LRPPI2 Feat Ex: 02:48 Train/Classify: 00:00 (29 sec)
LRPPI3 Feat Ex: 02:48 Train/Classify: 00:00 (30 sec)
LRPPI4 Feat Ex: 02:48 Train/Classify: 00:00 (46 sec)
ME1 Feat Ex: 0:20 Train/Classify: 00:00 (2 sec)
ME2 Feat Ex: 0:20 Train/Classify: 00:00 (2 sec)
ME3 Feat Ex: 0:20 Train/Classify: 00:00 (2 sec)

download these results as csv

CSV Files Without Rounding

audiomood_results_fold.csv
audiomood_results_class.csv

Results By Algorithm

(.tar.gz)
GP1 = G. Peeters
GT1 = G. Tzanetakis
GT2 = G. Tzanetakis
GT3 = G. Tzanetakis
HW = G. H. Wang
KL = K. Lee
LRPPI1 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 1
LRPPI2 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 2
LRPPI3 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 3
LRPPI4 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 4
ME1 = I. M. Mandel, D. P. W. Ellis 1
ME2 = I. M. Mandel, D. P. W. Ellis 2
ME3 = I. M. Mandel, D. P. W. Ellis 3

Friedman's Test for Significant Differences

Classes vs. Systems

The Friedman test was run in MATLAB against the average accuracy for each class.

Friedman's Anova Table
Source SS df MS Chi-sq Prob>Chi-sq
Columns 157.2 10 15.72 14.34 0.1579
Error 390.8 40 9.77
Total 548 54

download these results as csv

Tukey-Kramer HSD Multi-Comparison

The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);

TeamID TeamID Lowerbound Mean Upperbound Significance
GP1 GT1 -3.6392 3.1000 9.8392 FALSE
GP1 GT2 -3.3392 3.4000 10.1392 FALSE
GP1 GT3 -2.4392 4.3000 11.0392 FALSE
GP1 HW -2.8392 3.9000 10.6392 FALSE
GP1 KL -3.5392 3.2000 9.9392 FALSE
GP1 LRPPI1 -2.1392 4.6000 11.3392 FALSE
GP1 LRPPI2 -1.9392 4.8000 11.5392 FALSE
GP1 LRPPI3 -1.8392 4.9000 11.6392 FALSE
GP1 LRPPI4 -2.3392 4.4000 11.1392 FALSE
GP1 ME1 0.6608 7.4000 14.1392 TRUE
GT1 GT2 -6.4392 0.3000 7.0392 FALSE
GT1 GT3 -5.5392 1.2000 7.9392 FALSE
GT1 HW -5.9392 0.8000 7.5392 FALSE
GT1 KL -6.6392 0.1000 6.8392 FALSE
GT1 LRPPI1 -5.2392 1.5000 8.2392 FALSE
GT1 LRPPI2 -5.0392 1.7000 8.4392 FALSE
GT1 LRPPI3 -4.9392 1.8000 8.5392 FALSE
GT1 LRPPI4 -5.4392 1.3000 8.0392 FALSE
GT1 ME1 -2.4392 4.3000 11.0392 FALSE
GT2 GT3 -5.8392 0.9000 7.6392 FALSE
GT2 HW -6.2392 0.5000 7.2392 FALSE
GT2 KL -6.9392 -0.2000 6.5392 FALSE
GT2 LRPPI1 -5.5392 1.2000 7.9392 FALSE
GT2 LRPPI2 -5.3392 1.4000 8.1392 FALSE
GT2 LRPPI3 -5.2392 1.5000 8.2392 FALSE
GT2 LRPPI4 -5.7392 1.0000 7.7392 FALSE
GT2 ME1 -2.7392 4.0000 10.7392 FALSE
GT3 HW -7.1392 -0.4000 6.3392 FALSE
GT3 KL -7.8392 -1.1000 5.6392 FALSE
GT3 LRPPI1 -6.4392 0.3000 7.0392 FALSE
GT3 LRPPI2 -6.2392 0.5000 7.2392 FALSE
GT3 LRPPI3 -6.1392 0.6000 7.3392 FALSE
GT3 LRPPI4 -6.6392 0.1000 6.8392 FALSE
GT3 ME1 -3.6392 3.1000 9.8392 FALSE
HW KL -7.4392 -0.7000 6.0392 FALSE
HW LRPPI1 -6.0392 0.7000 7.4392 FALSE
HW LRPPI2 -5.8392 0.9000 7.6392 FALSE
HW LRPPI3 -5.7392 1.0000 7.7392 FALSE
HW LRPPI4 -6.2392 0.5000 7.2392 FALSE
HW ME1 -3.2392 3.5000 10.2392 FALSE
KL LRPPI1 -5.3392 1.4000 8.1392 FALSE
KL LRPPI2 -5.1392 1.6000 8.3392 FALSE
KL LRPPI3 -5.0392 1.7000 8.4392 FALSE
KL LRPPI4 -5.5392 1.2000 7.9392 FALSE
KL ME1 -2.5392 4.2000 10.9392 FALSE
LRPPI1 LRPPI2 -6.5392 0.2000 6.9392 FALSE
LRPPI1 LRPPI3 -6.4392 0.3000 7.0392 FALSE
LRPPI1 LRPPI4 -6.9392 -0.2000 6.5392 FALSE
LRPPI1 ME1 -3.9392 2.8000 9.5392 FALSE
LRPPI2 LRPPI3 -6.6392 0.1000 6.8392 FALSE
LRPPI2 LRPPI4 -7.1392 -0.4000 6.3392 FALSE
LRPPI2 ME1 -4.1392 2.6000 9.3392 FALSE
LRPPI3 LRPPI4 -7.2392 -0.5000 6.2392 FALSE
LRPPI3 ME1 -4.2392 2.5000 9.2392 FALSE
LRPPI4 ME1 -3.7392 3.0000 9.7392 FALSE

download these results as csv

File:Mood.perClassAccuracy.friedman.tukeyKramerHSD.png

Folds vs. Systems

The Friedman test was run in MATLAB against the accuracy for each fold.

Friedman's Anova Table
Source SS df MS Chi-sq Prob>Chi-sq
Columns 208.167 10 20.8167 18.95 0.0409
Error 121.333 20 6.0667
Total 329.5 32

download these results as csv

Tukey-Kramer HSD Multi-Comparison

The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);

TeamID TeamID Lowerbound Mean Upperbound Significance
GP1 GT1 -6.3762 2.3333 11.0429 FALSE
GP1 GT2 -4.3762 4.3333 13.0429 FALSE
GP1 GT3 -4.2095 4.5000 13.2095 FALSE
GP1 HW -4.0429 4.6667 13.3762 FALSE
GP1 KL -4.7095 4.0000 12.7095 FALSE
GP1 LRPPI1 -3.3762 5.3333 14.0429 FALSE
GP1 LRPPI2 -3.3762 5.3333 14.0429 FALSE
GP1 LRPPI3 -1.2095 7.5000 16.2095 FALSE
GP1 LRPPI4 -1.7095 7.0000 15.7095 FALSE
GP1 ME1 1.2905 10.0000 18.7095 TRUE
GT1 GT2 -6.7095 2.0000 10.7095 FALSE
GT1 GT3 -6.5429 2.1667 10.8762 FALSE
GT1 HW -6.3762 2.3333 11.0429 FALSE
GT1 KL -7.0429 1.6667 10.3762 FALSE
GT1 LRPPI1 -5.7095 3.0000 11.7095 FALSE
GT1 LRPPI2 -5.7095 3.0000 11.7095 FALSE
GT1 LRPPI3 -3.5429 5.1667 13.8762 FALSE
GT1 LRPPI4 -4.0429 4.6667 13.3762 FALSE
GT1 ME1 -1.0429 7.6667 16.3762 FALSE
GT2 GT3 -8.5429 0.1667 8.8762 FALSE
GT2 HW -8.3762 0.3333 9.0429 FALSE
GT2 KL -9.0429 -0.3333 8.3762 FALSE
GT2 LRPPI1 -7.7095 1.0000 9.7095 FALSE
GT2 LRPPI2 -7.7095 1.0000 9.7095 FALSE
GT2 LRPPI3 -5.5429 3.1667 11.8762 FALSE
GT2 LRPPI4 -6.0429 2.6667 11.3762 FALSE
GT2 ME1 -3.0429 5.6667 14.3762 FALSE
GT3 HW -8.5429 0.1667 8.8762 FALSE
GT3 KL -9.2095 -0.5000 8.2095 FALSE
GT3 LRPPI1 -7.8762 0.8333 9.5429 FALSE
GT3 LRPPI2 -7.8762 0.8333 9.5429 FALSE
GT3 LRPPI3 -5.7095 3.0000 11.7095 FALSE
GT3 LRPPI4 -6.2095 2.5000 11.2095 FALSE
GT3 ME1 -3.2095 5.5000 14.2095 FALSE
HW KL -9.3762 -0.6667 8.0429 FALSE
HW LRPPI1 -8.0429 0.6667 9.3762 FALSE
HW LRPPI2 -8.0429 0.6667 9.3762 FALSE
HW LRPPI3 -5.8762 2.8333 11.5429 FALSE
HW LRPPI4 -6.3762 2.3333 11.0429 FALSE
HW ME1 -3.3762 5.3333 14.0429 FALSE
KL LRPPI1 -7.3762 1.3333 10.0429 FALSE
KL LRPPI2 -7.3762 1.3333 10.0429 FALSE
KL LRPPI3 -5.2095 3.5000 12.2095 FALSE
KL LRPPI4 -5.7095 3.0000 11.7095 FALSE
KL ME1 -2.7095 6.0000 14.7095 FALSE
LRPPI1 LRPPI2 -8.7095 0.0000 8.7095 FALSE
LRPPI1 LRPPI3 -6.5429 2.1667 10.8762 FALSE
LRPPI1 LRPPI4 -7.0429 1.6667 10.3762 FALSE
LRPPI1 ME1 -4.0429 4.6667 13.3762 FALSE
LRPPI2 LRPPI3 -6.5429 2.1667 10.8762 FALSE
LRPPI2 LRPPI4 -7.0429 1.6667 10.3762 FALSE
LRPPI2 ME1 -4.0429 4.6667 13.3762 FALSE
LRPPI3 LRPPI4 -9.2095 -0.5000 8.2095 FALSE
LRPPI3 ME1 -6.2095 2.5000 11.2095 FALSE
LRPPI4 ME1 -5.7095 3.0000 11.7095 FALSE

download these results as csv

File:Mood.perFoldAccuracy.friedman.tukeyKramerHSD.png

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2008:Audio_Music_Mood_Classification_Results&oldid=6513"