Difference between revisions of "2008:Audio Music Mood Classification Results"
From MIREX Wiki
IMIRSELBot (talk | contribs) m (Robot: Automated text replacement (-mirex/abs/ +mirex/abstracts/)) |
IMIRSELBot (talk | contribs) m (Robot: Automated text replacement (-/mirex/2008/results/ +/mirex/results/2008/)) |
||
| Line 38: | Line 38: | ||
====CSV Files Without Rounding==== | ====CSV Files Without Rounding==== | ||
| − | [https://www.music-ir.org/mirex/2008 | + | [https://www.music-ir.org/mirex/results/2008/mood/audiomood_results_fold.csv audiomood_results_fold.csv]<br /> |
| − | [https://www.music-ir.org/mirex/2008 | + | [https://www.music-ir.org/mirex/results/2008/mood/audiomood_results_class.csv audiomood_results_class.csv]<br /> |
====Results By Algorithm==== | ====Results By Algorithm==== | ||
(.tar.gz) <br /> | (.tar.gz) <br /> | ||
| − | '''GP1''' = [https://www.music-ir.org/mirex/2008 | + | '''GP1''' = [https://www.music-ir.org/mirex/results/2008/mood/GP1.tar.gz G. Peeters]<br /> |
| − | '''GT1''' = [https://www.music-ir.org/mirex/2008 | + | '''GT1''' = [https://www.music-ir.org/mirex/results/2008/mood/GT1.tar.gz G. Tzanetakis]<br /> |
| − | '''GT2''' = [https://www.music-ir.org/mirex/2008 | + | '''GT2''' = [https://www.music-ir.org/mirex/results/2008/mood/GT2.tar.gz G. Tzanetakis]<br /> |
| − | '''GT3''' = [https://www.music-ir.org/mirex/2008 | + | '''GT3''' = [https://www.music-ir.org/mirex/results/2008/mood/GT3.tar.gz G. Tzanetakis]<br /> |
| − | '''HW''' = [https://www.music-ir.org/mirex/2008 | + | '''HW''' = [https://www.music-ir.org/mirex/results/2008/mood/HW.tar.gz G. H. Wang]<br /> |
| − | '''KL''' = [https://www.music-ir.org/mirex/2008 | + | '''KL''' = [https://www.music-ir.org/mirex/results/2008/mood/KL.tar.gz K. Lee]<br /> |
| − | '''LRPPI1''' = [https://www.music-ir.org/mirex/2008 | + | '''LRPPI1''' = [https://www.music-ir.org/mirex/results/2008/mood/LRPPI1.tar.gz T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 1]<br /> |
| − | '''LRPPI2''' = [https://www.music-ir.org/mirex/2008 | + | '''LRPPI2''' = [https://www.music-ir.org/mirex/results/2008/mood/LRPPI2.tar.gz T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 2]<br /> |
| − | '''LRPPI3''' = [https://www.music-ir.org/mirex/2008 | + | '''LRPPI3''' = [https://www.music-ir.org/mirex/results/2008/mood/LRPPI3.tar.gz T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 3]<br /> |
| − | '''LRPPI4''' = [https://www.music-ir.org/mirex/2008 | + | '''LRPPI4''' = [https://www.music-ir.org/mirex/results/2008/mood/LRPPI4.tar.gz T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 4]<br /> |
| − | '''ME1''' = [https://www.music-ir.org/mirex/2008 | + | '''ME1''' = [https://www.music-ir.org/mirex/results/2008/mood/ME1.tar.gz I. M. Mandel, D. P. W. Ellis 1]<br /> |
| − | '''ME2''' = [https://www.music-ir.org/mirex/2008 | + | '''ME2''' = [https://www.music-ir.org/mirex/results/2008/mood/ME2.tar.gz I. M. Mandel, D. P. W. Ellis 2]<br /> |
| − | '''ME3''' = [https://www.music-ir.org/mirex/2008 | + | '''ME3''' = [https://www.music-ir.org/mirex/results/2008/mood/ME3.tar.gz I. M. Mandel, D. P. W. Ellis 3]<br /> |
===Friedman's Test for Significant Differences=== | ===Friedman's Test for Significant Differences=== | ||
Revision as of 12:57, 7 June 2010
Contents
Introduction
These are the results for the 2008 running of the Audio Music Mood Classification task. For background information about this task set please refer to the 2008:Audio Music Mood Classification page.
General Legend
Team ID
GP1 = G. Peeters
GT1 = G. Tzanetakis
GT2 = G. Tzanetakis
GT3 = G. Tzanetakis
HW = H. Wang
KL = K. Lee
LRPPI1 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 1
LRPPI2 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 2
LRPPI3 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 3
LRPPI4 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 4
ME1 = M. I. Mandel, D. P. W. Ellis 1
ME2 = M. I. Mandel, D. P. W. Ellis 2
ME3 = M. I. Mandel, D. P. W. Ellis 3
Overall Summary Results
MIREX 2008 Audio Mood Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds
| Participant | Average Classifcation Accuracy |
|---|---|
| GP1 | 63.67% |
| GT1 | 55.00% |
| GT2 | 52.50% |
| GT3 | 58.20% |
| HW | 30.33% |
| KL | 49.83% |
| LRPPI1 | 56.00% |
| LRPPI2 | 55.50% |
| LRPPI3 | 54.50% |
| LRPPI4 | 55.50% |
| ME1 | 50.33% |
| ME2 | 50.00% |
| ME3 | 49.67% |
Accuracy Across Folds
| Classification fold | GP1 | GT1 | GT2 | GT3 | HW | KL | LRPPI1 | LRPPI2 | LRPPI3 | LRPPI4 | ME1 | ME2 | ME3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0.715 | 0.630 | 0.565 | 0.679 | 0.365 | 0.515 | 0.660 | 0.610 | 0.635 | 0.625 | 0.545 | 0.540 | 0.535 |
| 1 | 0.610 | 0.550 | 0.535 | 0.549 | 0.315 | 0.520 | 0.480 | 0.505 | 0.485 | 0.510 | 0.510 | 0.505 | 0.510 |
| 2 | 0.585 | 0.470 | 0.475 | 0.518 | 0.230 | 0.460 | 0.540 | 0.550 | 0.515 | 0.530 | 0.455 | 0.455 | 0.445 |
Accuracy Across Categories
| Class | GP1 | GT1 | GT2 | GT3 | HW | KL | LRPPI1 | LRPPI2 | LRPPI3 | LRPPI4 | ME1 | ME2 | ME3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 0.517 | 0.542 | 0.633 | 0.408 | 0.175 | 0.250 | 0.467 | 0.492 | 0.533 | 0.517 | 0.450 | 0.450 | 0.450 |
| 2 | 0.517 | 0.550 | 0.350 | 0.492 | 0.100 | 0.367 | 0.508 | 0.467 | 0.475 | 0.450 | 0.358 | 0.350 | 0.350 |
| 3 | 0.833 | 0.775 | 0.683 | 0.758 | 0.817 | 0.842 | 0.775 | 0.792 | 0.750 | 0.783 | 0.550 | 0.550 | 0.550 |
| 4 | 0.500 | 0.417 | 0.467 | 0.492 | 0.100 | 0.250 | 0.425 | 0.417 | 0.408 | 0.400 | 0.492 | 0.483 | 0.475 |
| 5 | 0.817 | 0.467 | 0.492 | 0.798 | 0.325 | 0.783 | 0.625 | 0.608 | 0.558 | 0.625 | 0.667 | 0.667 | 0.658 |
MIREX 2008 Audio Artist Classification Evaluation Logs and Confusion Matrices
MIREX 2008 Audio Mood Classification Run Times
| Participant | Runtime (hh:mm) / Fold |
|---|---|
| GP1 | Feat Ex: 01:01 Train/Classify: 00:02 |
| GT1 | Feat Ex/Train/Classify: 00:03 |
| GT2 | Feat Ex/Train/Classify: 00:07 |
| GT3 | Feat Ex: 00:01 Train/Classify: 00:00 (1 sec) |
| HW | Feat Ex/Train/Classify: 09:33 |
| KL | Feat Ex/Train/Classify: 00:09 |
| LRPPI1 | Feat Ex: 02:48 Train/Classify: 00:00 (11 sec) |
| LRPPI2 | Feat Ex: 02:48 Train/Classify: 00:00 (29 sec) |
| LRPPI3 | Feat Ex: 02:48 Train/Classify: 00:00 (30 sec) |
| LRPPI4 | Feat Ex: 02:48 Train/Classify: 00:00 (46 sec) |
| ME1 | Feat Ex: 0:20 Train/Classify: 00:00 (2 sec) |
| ME2 | Feat Ex: 0:20 Train/Classify: 00:00 (2 sec) |
| ME3 | Feat Ex: 0:20 Train/Classify: 00:00 (2 sec) |
CSV Files Without Rounding
audiomood_results_fold.csv
audiomood_results_class.csv
Results By Algorithm
(.tar.gz)
GP1 = G. Peeters
GT1 = G. Tzanetakis
GT2 = G. Tzanetakis
GT3 = G. Tzanetakis
HW = G. H. Wang
KL = K. Lee
LRPPI1 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 1
LRPPI2 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 2
LRPPI3 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 3
LRPPI4 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 4
ME1 = I. M. Mandel, D. P. W. Ellis 1
ME2 = I. M. Mandel, D. P. W. Ellis 2
ME3 = I. M. Mandel, D. P. W. Ellis 3
Friedman's Test for Significant Differences
Classes vs. Systems
The Friedman test was run in MATLAB against the average accuracy for each class.
Friedman's Anova Table
| Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
|---|---|---|---|---|---|
| Columns | 157.2 | 10 | 15.72 | 14.34 | 0.1579 |
| Error | 390.8 | 40 | 9.77 | ||
| Total | 548 | 54 |
Tukey-Kramer HSD Multi-Comparison
The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| GP1 | GT1 | -3.6392 | 3.1000 | 9.8392 | FALSE |
| GP1 | GT2 | -3.3392 | 3.4000 | 10.1392 | FALSE |
| GP1 | GT3 | -2.4392 | 4.3000 | 11.0392 | FALSE |
| GP1 | HW | -2.8392 | 3.9000 | 10.6392 | FALSE |
| GP1 | KL | -3.5392 | 3.2000 | 9.9392 | FALSE |
| GP1 | LRPPI1 | -2.1392 | 4.6000 | 11.3392 | FALSE |
| GP1 | LRPPI2 | -1.9392 | 4.8000 | 11.5392 | FALSE |
| GP1 | LRPPI3 | -1.8392 | 4.9000 | 11.6392 | FALSE |
| GP1 | LRPPI4 | -2.3392 | 4.4000 | 11.1392 | FALSE |
| GP1 | ME1 | 0.6608 | 7.4000 | 14.1392 | TRUE |
| GT1 | GT2 | -6.4392 | 0.3000 | 7.0392 | FALSE |
| GT1 | GT3 | -5.5392 | 1.2000 | 7.9392 | FALSE |
| GT1 | HW | -5.9392 | 0.8000 | 7.5392 | FALSE |
| GT1 | KL | -6.6392 | 0.1000 | 6.8392 | FALSE |
| GT1 | LRPPI1 | -5.2392 | 1.5000 | 8.2392 | FALSE |
| GT1 | LRPPI2 | -5.0392 | 1.7000 | 8.4392 | FALSE |
| GT1 | LRPPI3 | -4.9392 | 1.8000 | 8.5392 | FALSE |
| GT1 | LRPPI4 | -5.4392 | 1.3000 | 8.0392 | FALSE |
| GT1 | ME1 | -2.4392 | 4.3000 | 11.0392 | FALSE |
| GT2 | GT3 | -5.8392 | 0.9000 | 7.6392 | FALSE |
| GT2 | HW | -6.2392 | 0.5000 | 7.2392 | FALSE |
| GT2 | KL | -6.9392 | -0.2000 | 6.5392 | FALSE |
| GT2 | LRPPI1 | -5.5392 | 1.2000 | 7.9392 | FALSE |
| GT2 | LRPPI2 | -5.3392 | 1.4000 | 8.1392 | FALSE |
| GT2 | LRPPI3 | -5.2392 | 1.5000 | 8.2392 | FALSE |
| GT2 | LRPPI4 | -5.7392 | 1.0000 | 7.7392 | FALSE |
| GT2 | ME1 | -2.7392 | 4.0000 | 10.7392 | FALSE |
| GT3 | HW | -7.1392 | -0.4000 | 6.3392 | FALSE |
| GT3 | KL | -7.8392 | -1.1000 | 5.6392 | FALSE |
| GT3 | LRPPI1 | -6.4392 | 0.3000 | 7.0392 | FALSE |
| GT3 | LRPPI2 | -6.2392 | 0.5000 | 7.2392 | FALSE |
| GT3 | LRPPI3 | -6.1392 | 0.6000 | 7.3392 | FALSE |
| GT3 | LRPPI4 | -6.6392 | 0.1000 | 6.8392 | FALSE |
| GT3 | ME1 | -3.6392 | 3.1000 | 9.8392 | FALSE |
| HW | KL | -7.4392 | -0.7000 | 6.0392 | FALSE |
| HW | LRPPI1 | -6.0392 | 0.7000 | 7.4392 | FALSE |
| HW | LRPPI2 | -5.8392 | 0.9000 | 7.6392 | FALSE |
| HW | LRPPI3 | -5.7392 | 1.0000 | 7.7392 | FALSE |
| HW | LRPPI4 | -6.2392 | 0.5000 | 7.2392 | FALSE |
| HW | ME1 | -3.2392 | 3.5000 | 10.2392 | FALSE |
| KL | LRPPI1 | -5.3392 | 1.4000 | 8.1392 | FALSE |
| KL | LRPPI2 | -5.1392 | 1.6000 | 8.3392 | FALSE |
| KL | LRPPI3 | -5.0392 | 1.7000 | 8.4392 | FALSE |
| KL | LRPPI4 | -5.5392 | 1.2000 | 7.9392 | FALSE |
| KL | ME1 | -2.5392 | 4.2000 | 10.9392 | FALSE |
| LRPPI1 | LRPPI2 | -6.5392 | 0.2000 | 6.9392 | FALSE |
| LRPPI1 | LRPPI3 | -6.4392 | 0.3000 | 7.0392 | FALSE |
| LRPPI1 | LRPPI4 | -6.9392 | -0.2000 | 6.5392 | FALSE |
| LRPPI1 | ME1 | -3.9392 | 2.8000 | 9.5392 | FALSE |
| LRPPI2 | LRPPI3 | -6.6392 | 0.1000 | 6.8392 | FALSE |
| LRPPI2 | LRPPI4 | -7.1392 | -0.4000 | 6.3392 | FALSE |
| LRPPI2 | ME1 | -4.1392 | 2.6000 | 9.3392 | FALSE |
| LRPPI3 | LRPPI4 | -7.2392 | -0.5000 | 6.2392 | FALSE |
| LRPPI3 | ME1 | -4.2392 | 2.5000 | 9.2392 | FALSE |
| LRPPI4 | ME1 | -3.7392 | 3.0000 | 9.7392 | FALSE |
Folds vs. Systems
The Friedman test was run in MATLAB against the accuracy for each fold.
Friedman's Anova Table
| Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
|---|---|---|---|---|---|
| Columns | 208.167 | 10 | 20.8167 | 18.95 | 0.0409 |
| Error | 121.333 | 20 | 6.0667 | ||
| Total | 329.5 | 32 |
Tukey-Kramer HSD Multi-Comparison
The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| GP1 | GT1 | -6.3762 | 2.3333 | 11.0429 | FALSE |
| GP1 | GT2 | -4.3762 | 4.3333 | 13.0429 | FALSE |
| GP1 | GT3 | -4.2095 | 4.5000 | 13.2095 | FALSE |
| GP1 | HW | -4.0429 | 4.6667 | 13.3762 | FALSE |
| GP1 | KL | -4.7095 | 4.0000 | 12.7095 | FALSE |
| GP1 | LRPPI1 | -3.3762 | 5.3333 | 14.0429 | FALSE |
| GP1 | LRPPI2 | -3.3762 | 5.3333 | 14.0429 | FALSE |
| GP1 | LRPPI3 | -1.2095 | 7.5000 | 16.2095 | FALSE |
| GP1 | LRPPI4 | -1.7095 | 7.0000 | 15.7095 | FALSE |
| GP1 | ME1 | 1.2905 | 10.0000 | 18.7095 | TRUE |
| GT1 | GT2 | -6.7095 | 2.0000 | 10.7095 | FALSE |
| GT1 | GT3 | -6.5429 | 2.1667 | 10.8762 | FALSE |
| GT1 | HW | -6.3762 | 2.3333 | 11.0429 | FALSE |
| GT1 | KL | -7.0429 | 1.6667 | 10.3762 | FALSE |
| GT1 | LRPPI1 | -5.7095 | 3.0000 | 11.7095 | FALSE |
| GT1 | LRPPI2 | -5.7095 | 3.0000 | 11.7095 | FALSE |
| GT1 | LRPPI3 | -3.5429 | 5.1667 | 13.8762 | FALSE |
| GT1 | LRPPI4 | -4.0429 | 4.6667 | 13.3762 | FALSE |
| GT1 | ME1 | -1.0429 | 7.6667 | 16.3762 | FALSE |
| GT2 | GT3 | -8.5429 | 0.1667 | 8.8762 | FALSE |
| GT2 | HW | -8.3762 | 0.3333 | 9.0429 | FALSE |
| GT2 | KL | -9.0429 | -0.3333 | 8.3762 | FALSE |
| GT2 | LRPPI1 | -7.7095 | 1.0000 | 9.7095 | FALSE |
| GT2 | LRPPI2 | -7.7095 | 1.0000 | 9.7095 | FALSE |
| GT2 | LRPPI3 | -5.5429 | 3.1667 | 11.8762 | FALSE |
| GT2 | LRPPI4 | -6.0429 | 2.6667 | 11.3762 | FALSE |
| GT2 | ME1 | -3.0429 | 5.6667 | 14.3762 | FALSE |
| GT3 | HW | -8.5429 | 0.1667 | 8.8762 | FALSE |
| GT3 | KL | -9.2095 | -0.5000 | 8.2095 | FALSE |
| GT3 | LRPPI1 | -7.8762 | 0.8333 | 9.5429 | FALSE |
| GT3 | LRPPI2 | -7.8762 | 0.8333 | 9.5429 | FALSE |
| GT3 | LRPPI3 | -5.7095 | 3.0000 | 11.7095 | FALSE |
| GT3 | LRPPI4 | -6.2095 | 2.5000 | 11.2095 | FALSE |
| GT3 | ME1 | -3.2095 | 5.5000 | 14.2095 | FALSE |
| HW | KL | -9.3762 | -0.6667 | 8.0429 | FALSE |
| HW | LRPPI1 | -8.0429 | 0.6667 | 9.3762 | FALSE |
| HW | LRPPI2 | -8.0429 | 0.6667 | 9.3762 | FALSE |
| HW | LRPPI3 | -5.8762 | 2.8333 | 11.5429 | FALSE |
| HW | LRPPI4 | -6.3762 | 2.3333 | 11.0429 | FALSE |
| HW | ME1 | -3.3762 | 5.3333 | 14.0429 | FALSE |
| KL | LRPPI1 | -7.3762 | 1.3333 | 10.0429 | FALSE |
| KL | LRPPI2 | -7.3762 | 1.3333 | 10.0429 | FALSE |
| KL | LRPPI3 | -5.2095 | 3.5000 | 12.2095 | FALSE |
| KL | LRPPI4 | -5.7095 | 3.0000 | 11.7095 | FALSE |
| KL | ME1 | -2.7095 | 6.0000 | 14.7095 | FALSE |
| LRPPI1 | LRPPI2 | -8.7095 | 0.0000 | 8.7095 | FALSE |
| LRPPI1 | LRPPI3 | -6.5429 | 2.1667 | 10.8762 | FALSE |
| LRPPI1 | LRPPI4 | -7.0429 | 1.6667 | 10.3762 | FALSE |
| LRPPI1 | ME1 | -4.0429 | 4.6667 | 13.3762 | FALSE |
| LRPPI2 | LRPPI3 | -6.5429 | 2.1667 | 10.8762 | FALSE |
| LRPPI2 | LRPPI4 | -7.0429 | 1.6667 | 10.3762 | FALSE |
| LRPPI2 | ME1 | -4.0429 | 4.6667 | 13.3762 | FALSE |
| LRPPI3 | LRPPI4 | -9.2095 | -0.5000 | 8.2095 | FALSE |
| LRPPI3 | ME1 | -6.2095 | 2.5000 | 11.2095 | FALSE |
| LRPPI4 | ME1 | -5.7095 | 3.0000 | 11.7095 | FALSE |
