2008:Audio Music Mood Classification Results
Contents
[hide]Introduction
These are the results for the 2008 running of the Audio Music Mood Classification task. For background information about this task set please refer to the 2008:Audio Music Mood Classification page.
General Legend
Team ID
GP1 = G. Peeters
GT1 = G. Tzanetakis
GT2 = G. Tzanetakis
GT3 = G. Tzanetakis
HW = H. Wang
KL = K. Lee
LRPPI1 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 1
LRPPI2 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 2
LRPPI3 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 3
LRPPI4 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 4
ME1 = M. I. Mandel, D. P. W. Ellis 1
ME2 = M. I. Mandel, D. P. W. Ellis 2
ME3 = M. I. Mandel, D. P. W. Ellis 3
Overall Summary Results
MIREX 2008 Audio Mood Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds
| Participant | Average Classifcation Accuracy |
|---|---|
| GP1 | 63.67% |
| GT1 | 55.00% |
| GT2 | 52.50% |
| GT3 | 58.20% |
| HW | 30.33% |
| KL | 49.83% |
| LRPPI1 | 56.00% |
| LRPPI2 | 55.50% |
| LRPPI3 | 54.50% |
| LRPPI4 | 55.50% |
| ME1 | 50.33% |
| ME2 | 50.00% |
| ME3 | 49.67% |
Accuracy Across Folds
| Classification fold | GP1 | GT1 | GT2 | GT3 | HW | KL | LRPPI1 | LRPPI2 | LRPPI3 | LRPPI4 | ME1 | ME2 | ME3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0.715 | 0.630 | 0.565 | 0.679 | 0.365 | 0.515 | 0.660 | 0.610 | 0.635 | 0.625 | 0.545 | 0.540 | 0.535 |
| 1 | 0.610 | 0.550 | 0.535 | 0.549 | 0.315 | 0.520 | 0.480 | 0.505 | 0.485 | 0.510 | 0.510 | 0.505 | 0.510 |
| 2 | 0.585 | 0.470 | 0.475 | 0.518 | 0.230 | 0.460 | 0.540 | 0.550 | 0.515 | 0.530 | 0.455 | 0.455 | 0.445 |
Accuracy Across Categories
| Class | GP1 | GT1 | GT2 | GT3 | HW | KL | LRPPI1 | LRPPI2 | LRPPI3 | LRPPI4 | ME1 | ME2 | ME3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 0.517 | 0.542 | 0.633 | 0.408 | 0.175 | 0.250 | 0.467 | 0.492 | 0.533 | 0.517 | 0.450 | 0.450 | 0.450 |
| 2 | 0.517 | 0.550 | 0.350 | 0.492 | 0.100 | 0.367 | 0.508 | 0.467 | 0.475 | 0.450 | 0.358 | 0.350 | 0.350 |
| 3 | 0.833 | 0.775 | 0.683 | 0.758 | 0.817 | 0.842 | 0.775 | 0.792 | 0.750 | 0.783 | 0.550 | 0.550 | 0.550 |
| 4 | 0.500 | 0.417 | 0.467 | 0.492 | 0.100 | 0.250 | 0.425 | 0.417 | 0.408 | 0.400 | 0.492 | 0.483 | 0.475 |
| 5 | 0.817 | 0.467 | 0.492 | 0.798 | 0.325 | 0.783 | 0.625 | 0.608 | 0.558 | 0.625 | 0.667 | 0.667 | 0.658 |
MIREX 2008 Audio Artist Classification Evaluation Logs and Confusion Matrices
MIREX 2008 Audio Mood Classification Run Times
| Participant | Runtime (hh:mm) / Fold |
|---|---|
| GP1 | Feat Ex: 01:01 Train/Classify: 00:02 |
| GT1 | Feat Ex/Train/Classify: 00:03 |
| GT2 | Feat Ex/Train/Classify: 00:07 |
| GT3 | Feat Ex: 00:01 Train/Classify: 00:00 (1 sec) |
| HW | Feat Ex/Train/Classify: 09:33 |
| KL | Feat Ex/Train/Classify: 00:09 |
| LRPPI1 | Feat Ex: 02:48 Train/Classify: 00:00 (11 sec) |
| LRPPI2 | Feat Ex: 02:48 Train/Classify: 00:00 (29 sec) |
| LRPPI3 | Feat Ex: 02:48 Train/Classify: 00:00 (30 sec) |
| LRPPI4 | Feat Ex: 02:48 Train/Classify: 00:00 (46 sec) |
| ME1 | Feat Ex: 0:20 Train/Classify: 00:00 (2 sec) |
| ME2 | Feat Ex: 0:20 Train/Classify: 00:00 (2 sec) |
| ME3 | Feat Ex: 0:20 Train/Classify: 00:00 (2 sec) |
CSV Files Without Rounding
audiomood_results_fold.csv
audiomood_results_class.csv
Results By Algorithm
(.tar.gz)
GP1 = G. Peeters
GT1 = G. Tzanetakis
GT2 = G. Tzanetakis
GT3 = G. Tzanetakis
HW = G. H. Wang
KL = K. Lee
LRPPI1 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 1
LRPPI2 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 2
LRPPI3 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 3
LRPPI4 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 4
ME1 = I. M. Mandel, D. P. W. Ellis 1
ME2 = I. M. Mandel, D. P. W. Ellis 2
ME3 = I. M. Mandel, D. P. W. Ellis 3
Friedman's Test for Significant Differences
Classes vs. Systems
The Friedman test was run in MATLAB against the average accuracy for each class.
Friedman's Anova Table
| Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
|---|---|---|---|---|---|
| Columns | 157.2 | 10 | 15.72 | 14.34 | 0.1579 |
| Error | 390.8 | 40 | 9.77 | ||
| Total | 548 | 54 |
Tukey-Kramer HSD Multi-Comparison
The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| GP1 | GT1 | -3.6392 | 3.1000 | 9.8392 | FALSE |
| GP1 | GT2 | -3.3392 | 3.4000 | 10.1392 | FALSE |
| GP1 | GT3 | -2.4392 | 4.3000 | 11.0392 | FALSE |
| GP1 | HW | -2.8392 | 3.9000 | 10.6392 | FALSE |
| GP1 | KL | -3.5392 | 3.2000 | 9.9392 | FALSE |
| GP1 | LRPPI1 | -2.1392 | 4.6000 | 11.3392 | FALSE |
| GP1 | LRPPI2 | -1.9392 | 4.8000 | 11.5392 | FALSE |
| GP1 | LRPPI3 | -1.8392 | 4.9000 | 11.6392 | FALSE |
| GP1 | LRPPI4 | -2.3392 | 4.4000 | 11.1392 | FALSE |
| GP1 | ME1 | 0.6608 | 7.4000 | 14.1392 | TRUE |
| GT1 | GT2 | -6.4392 | 0.3000 | 7.0392 | FALSE |
| GT1 | GT3 | -5.5392 | 1.2000 | 7.9392 | FALSE |
| GT1 | HW | -5.9392 | 0.8000 | 7.5392 | FALSE |
| GT1 | KL | -6.6392 | 0.1000 | 6.8392 | FALSE |
| GT1 | LRPPI1 | -5.2392 | 1.5000 | 8.2392 | FALSE |
| GT1 | LRPPI2 | -5.0392 | 1.7000 | 8.4392 | FALSE |
| GT1 | LRPPI3 | -4.9392 | 1.8000 | 8.5392 | FALSE |
| GT1 | LRPPI4 | -5.4392 | 1.3000 | 8.0392 | FALSE |
| GT1 | ME1 | -2.4392 | 4.3000 | 11.0392 | FALSE |
| GT2 | GT3 | -5.8392 | 0.9000 | 7.6392 | FALSE |
| GT2 | HW | -6.2392 | 0.5000 | 7.2392 | FALSE |
| GT2 | KL | -6.9392 | -0.2000 | 6.5392 | FALSE |
| GT2 | LRPPI1 | -5.5392 | 1.2000 | 7.9392 | FALSE |
| GT2 | LRPPI2 | -5.3392 | 1.4000 | 8.1392 | FALSE |
| GT2 | LRPPI3 | -5.2392 | 1.5000 | 8.2392 | FALSE |
| GT2 | LRPPI4 | -5.7392 | 1.0000 | 7.7392 | FALSE |
| GT2 | ME1 | -2.7392 | 4.0000 | 10.7392 | FALSE |
| GT3 | HW | -7.1392 | -0.4000 | 6.3392 | FALSE |
| GT3 | KL | -7.8392 | -1.1000 | 5.6392 | FALSE |
| GT3 | LRPPI1 | -6.4392 | 0.3000 | 7.0392 | FALSE |
| GT3 | LRPPI2 | -6.2392 | 0.5000 | 7.2392 | FALSE |
| GT3 | LRPPI3 | -6.1392 | 0.6000 | 7.3392 | FALSE |
| GT3 | LRPPI4 | -6.6392 | 0.1000 | 6.8392 | FALSE |
| GT3 | ME1 | -3.6392 | 3.1000 | 9.8392 | FALSE |
| HW | KL | -7.4392 | -0.7000 | 6.0392 | FALSE |
| HW | LRPPI1 | -6.0392 | 0.7000 | 7.4392 | FALSE |
| HW | LRPPI2 | -5.8392 | 0.9000 | 7.6392 | FALSE |
| HW | LRPPI3 | -5.7392 | 1.0000 | 7.7392 | FALSE |
| HW | LRPPI4 | -6.2392 | 0.5000 | 7.2392 | FALSE |
| HW | ME1 | -3.2392 | 3.5000 | 10.2392 | FALSE |
| KL | LRPPI1 | -5.3392 | 1.4000 | 8.1392 | FALSE |
| KL | LRPPI2 | -5.1392 | 1.6000 | 8.3392 | FALSE |
| KL | LRPPI3 | -5.0392 | 1.7000 | 8.4392 | FALSE |
| KL | LRPPI4 | -5.5392 | 1.2000 | 7.9392 | FALSE |
| KL | ME1 | -2.5392 | 4.2000 | 10.9392 | FALSE |
| LRPPI1 | LRPPI2 | -6.5392 | 0.2000 | 6.9392 | FALSE |
| LRPPI1 | LRPPI3 | -6.4392 | 0.3000 | 7.0392 | FALSE |
| LRPPI1 | LRPPI4 | -6.9392 | -0.2000 | 6.5392 | FALSE |
| LRPPI1 | ME1 | -3.9392 | 2.8000 | 9.5392 | FALSE |
| LRPPI2 | LRPPI3 | -6.6392 | 0.1000 | 6.8392 | FALSE |
| LRPPI2 | LRPPI4 | -7.1392 | -0.4000 | 6.3392 | FALSE |
| LRPPI2 | ME1 | -4.1392 | 2.6000 | 9.3392 | FALSE |
| LRPPI3 | LRPPI4 | -7.2392 | -0.5000 | 6.2392 | FALSE |
| LRPPI3 | ME1 | -4.2392 | 2.5000 | 9.2392 | FALSE |
| LRPPI4 | ME1 | -3.7392 | 3.0000 | 9.7392 | FALSE |
Folds vs. Systems
The Friedman test was run in MATLAB against the accuracy for each fold.
Friedman's Anova Table
| Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
|---|---|---|---|---|---|
| Columns | 208.167 | 10 | 20.8167 | 18.95 | 0.0409 |
| Error | 121.333 | 20 | 6.0667 | ||
| Total | 329.5 | 32 |
Tukey-Kramer HSD Multi-Comparison
The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| GP1 | GT1 | -6.3762 | 2.3333 | 11.0429 | FALSE |
| GP1 | GT2 | -4.3762 | 4.3333 | 13.0429 | FALSE |
| GP1 | GT3 | -4.2095 | 4.5000 | 13.2095 | FALSE |
| GP1 | HW | -4.0429 | 4.6667 | 13.3762 | FALSE |
| GP1 | KL | -4.7095 | 4.0000 | 12.7095 | FALSE |
| GP1 | LRPPI1 | -3.3762 | 5.3333 | 14.0429 | FALSE |
| GP1 | LRPPI2 | -3.3762 | 5.3333 | 14.0429 | FALSE |
| GP1 | LRPPI3 | -1.2095 | 7.5000 | 16.2095 | FALSE |
| GP1 | LRPPI4 | -1.7095 | 7.0000 | 15.7095 | FALSE |
| GP1 | ME1 | 1.2905 | 10.0000 | 18.7095 | TRUE |
| GT1 | GT2 | -6.7095 | 2.0000 | 10.7095 | FALSE |
| GT1 | GT3 | -6.5429 | 2.1667 | 10.8762 | FALSE |
| GT1 | HW | -6.3762 | 2.3333 | 11.0429 | FALSE |
| GT1 | KL | -7.0429 | 1.6667 | 10.3762 | FALSE |
| GT1 | LRPPI1 | -5.7095 | 3.0000 | 11.7095 | FALSE |
| GT1 | LRPPI2 | -5.7095 | 3.0000 | 11.7095 | FALSE |
| GT1 | LRPPI3 | -3.5429 | 5.1667 | 13.8762 | FALSE |
| GT1 | LRPPI4 | -4.0429 | 4.6667 | 13.3762 | FALSE |
| GT1 | ME1 | -1.0429 | 7.6667 | 16.3762 | FALSE |
| GT2 | GT3 | -8.5429 | 0.1667 | 8.8762 | FALSE |
| GT2 | HW | -8.3762 | 0.3333 | 9.0429 | FALSE |
| GT2 | KL | -9.0429 | -0.3333 | 8.3762 | FALSE |
| GT2 | LRPPI1 | -7.7095 | 1.0000 | 9.7095 | FALSE |
| GT2 | LRPPI2 | -7.7095 | 1.0000 | 9.7095 | FALSE |
| GT2 | LRPPI3 | -5.5429 | 3.1667 | 11.8762 | FALSE |
| GT2 | LRPPI4 | -6.0429 | 2.6667 | 11.3762 | FALSE |
| GT2 | ME1 | -3.0429 | 5.6667 | 14.3762 | FALSE |
| GT3 | HW | -8.5429 | 0.1667 | 8.8762 | FALSE |
| GT3 | KL | -9.2095 | -0.5000 | 8.2095 | FALSE |
| GT3 | LRPPI1 | -7.8762 | 0.8333 | 9.5429 | FALSE |
| GT3 | LRPPI2 | -7.8762 | 0.8333 | 9.5429 | FALSE |
| GT3 | LRPPI3 | -5.7095 | 3.0000 | 11.7095 | FALSE |
| GT3 | LRPPI4 | -6.2095 | 2.5000 | 11.2095 | FALSE |
| GT3 | ME1 | -3.2095 | 5.5000 | 14.2095 | FALSE |
| HW | KL | -9.3762 | -0.6667 | 8.0429 | FALSE |
| HW | LRPPI1 | -8.0429 | 0.6667 | 9.3762 | FALSE |
| HW | LRPPI2 | -8.0429 | 0.6667 | 9.3762 | FALSE |
| HW | LRPPI3 | -5.8762 | 2.8333 | 11.5429 | FALSE |
| HW | LRPPI4 | -6.3762 | 2.3333 | 11.0429 | FALSE |
| HW | ME1 | -3.3762 | 5.3333 | 14.0429 | FALSE |
| KL | LRPPI1 | -7.3762 | 1.3333 | 10.0429 | FALSE |
| KL | LRPPI2 | -7.3762 | 1.3333 | 10.0429 | FALSE |
| KL | LRPPI3 | -5.2095 | 3.5000 | 12.2095 | FALSE |
| KL | LRPPI4 | -5.7095 | 3.0000 | 11.7095 | FALSE |
| KL | ME1 | -2.7095 | 6.0000 | 14.7095 | FALSE |
| LRPPI1 | LRPPI2 | -8.7095 | 0.0000 | 8.7095 | FALSE |
| LRPPI1 | LRPPI3 | -6.5429 | 2.1667 | 10.8762 | FALSE |
| LRPPI1 | LRPPI4 | -7.0429 | 1.6667 | 10.3762 | FALSE |
| LRPPI1 | ME1 | -4.0429 | 4.6667 | 13.3762 | FALSE |
| LRPPI2 | LRPPI3 | -6.5429 | 2.1667 | 10.8762 | FALSE |
| LRPPI2 | LRPPI4 | -7.0429 | 1.6667 | 10.3762 | FALSE |
| LRPPI2 | ME1 | -4.0429 | 4.6667 | 13.3762 | FALSE |
| LRPPI3 | LRPPI4 | -9.2095 | -0.5000 | 8.2095 | FALSE |
| LRPPI3 | ME1 | -6.2095 | 2.5000 | 11.2095 | FALSE |
| LRPPI4 | ME1 | -5.7095 | 3.0000 | 11.7095 | FALSE |
