Difference between revisions of "2008:Audio Genre Classification Results"
From MIREX Wiki
IMIRSELBot (talk | contribs) m (Bot: Fixing image links) |
IMIRSELBot (talk | contribs) m (Robot: Automated text replacement (-mirex/abs/ +mirex/abstracts/)) |
||
| Line 4: | Line 4: | ||
===General Legend=== | ===General Legend=== | ||
====Team ID==== | ====Team ID==== | ||
| − | '''CL1''' = [https://www.music-ir.org/mirex/ | + | '''CL1''' = [https://www.music-ir.org/mirex/abstracts/2008/mirex08_genre_CC.pdf C. Cao, M. Li 1]<br /> |
| − | '''CL2''' = [https://www.music-ir.org/mirex/ | + | '''CL2''' = [https://www.music-ir.org/mirex/abstracts/2008/mirex08_genre_CC.pdf C. Cao, M. Li 2]<br /> |
| − | '''GP1''' = [https://www.music-ir.org/mirex/ | + | '''GP1''' = [https://www.music-ir.org/mirex/abstracts/2008/Peeters_2008_ISMIR_MIREX.pdf G. Peeters]<br /> |
| − | '''GT1 (mono)''' = [https://www.music-ir.org/mirex/ | + | '''GT1 (mono)''' = [https://www.music-ir.org/mirex/abstracts/2008/mirex2007.pdf G. Tzanetakis]<br /> |
| − | '''GT2 (stereo)''' = [https://www.music-ir.org/mirex/ | + | '''GT2 (stereo)''' = [https://www.music-ir.org/mirex/abstracts/2008/mirex2007.pdf G. Tzanetakis]<br /> |
| − | '''GT3 (multicore)''' = [https://www.music-ir.org/mirex/ | + | '''GT3 (multicore)''' = [https://www.music-ir.org/mirex/abstracts/2008/mirex2007.pdf G. Tzanetakis]<br /> |
| − | '''LRPPI1''' = [https://www.music-ir.org/mirex/ | + | '''LRPPI1''' = [https://www.music-ir.org/mirex/abstracts/2008/abstract_mirex08_class.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 1]<br /> |
| − | '''LRPPI2''' = [https://www.music-ir.org/mirex/ | + | '''LRPPI2''' = [https://www.music-ir.org/mirex/abstracts/2008/abstract_mirex08_class.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 2]<br /> |
| − | '''LRPPI3''' = [https://www.music-ir.org/mirex/ | + | '''LRPPI3''' = [https://www.music-ir.org/mirex/abstracts/2008/abstract_mirex08_class.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 3]<br /> |
| − | '''LRPPI4''' = [https://www.music-ir.org/mirex/ | + | '''LRPPI4''' = [https://www.music-ir.org/mirex/abstracts/2008/abstract_mirex08_class.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 4]<br /> |
| − | '''ME1''' = [https://www.music-ir.org/mirex/ | + | '''ME1''' = [https://www.music-ir.org/mirex/abstracts/2008/AA_AG_AT_MM_CC_mandel.pdf I. M. Mandel, D. P. W. Ellis 1]<br /> |
| − | '''ME2''' = [https://www.music-ir.org/mirex/ | + | '''ME2''' = [https://www.music-ir.org/mirex/abstracts/2008/AA_AG_AT_MM_CC_mandel.pdf I. M. Mandel, D. P. W. Ellis 2]<br /> |
| − | '''ME3''' = [https://www.music-ir.org/mirex/ | + | '''ME3''' = [https://www.music-ir.org/mirex/abstracts/2008/AA_AG_AT_MM_CC_mandel.pdf I. M. Mandel, D. P. W. Ellis 3]<br /> |
==Overall Summary Results== | ==Overall Summary Results== | ||
Revision as of 22:47, 13 May 2010
Contents
- 1 Introduction
- 2 Overall Summary Results
- 2.1 Task 1 (MIXED) Results
- 2.1.1 MIREX 2008 Audio Genre Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds
- 2.1.2 MIREX 2008 Audio Genre Classification Evaluation Logs and Confusion Matrices
- 2.1.3 MIREX 2008 Audio Genre Classification Run Times
- 2.1.4 CSV Files Without Rounding
- 2.1.5 Results By Algorithm
- 2.2 Task 2 (LATIN) Results
- 2.2.1 MIREX 2008 Audio Genre Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds
- 2.2.2 MIREX 2008 Audio Genre Classification Evaluation Logs and Confusion Matrices
- 2.2.3 MIREX 2008 Audio Genre Classification Run Times
- 2.2.4 CSV Files Without Rounding
- 2.2.5 Results By Algorithm
- 2.3 Friedman's Test for Significant Differences
- 2.1 Task 1 (MIXED) Results
Introduction
These are the results for the 2008 running of the Audio Genre Classification task. For background information about this task set please refer to the 2008:Audio Genre Classification page.
General Legend
Team ID
CL1 = C. Cao, M. Li 1
CL2 = C. Cao, M. Li 2
GP1 = G. Peeters
GT1 (mono) = G. Tzanetakis
GT2 (stereo) = G. Tzanetakis
GT3 (multicore) = G. Tzanetakis
LRPPI1 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 1
LRPPI2 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 2
LRPPI3 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 3
LRPPI4 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 4
ME1 = I. M. Mandel, D. P. W. Ellis 1
ME2 = I. M. Mandel, D. P. W. Ellis 2
ME3 = I. M. Mandel, D. P. W. Ellis 3
Overall Summary Results
Task 1 (MIXED) Results
MIREX 2008 Audio Genre Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds
| Participant | Average Classifcation Accuracy |
|---|---|
| CL1 | 62.04% |
| CL2 | 63.39% |
| GP1 | 63.90% |
| GT1 | 64.71% |
| GT2 | 66.41% |
| GT3 | 65.62% |
| LRPPI1 | 65.06% |
| LRPPI2 | 62.26% |
| LRPPI3 | 60.84% |
| LRPPI4 | 60.46% |
| ME1 | 65.41% |
| ME2 | 65.30% |
| ME3 | 65.20% |
Accuracy Across Folds
| Classification fold | CL1 | CL2 | GP1 | GT1 | GT2 | GT3 | LRPPI1 | LRPPI2 | LRPPI3 | LRPPI4 | ME1 | ME2 | ME3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0.592 | 0.598 | 0.634 | 0.639 | 0.642 | 0.654 | 0.650 | 0.610 | 0.598 | 0.606 | 0.631 | 0.631 | 0.628 |
| 1 | 0.644 | 0.661 | 0.634 | 0.651 | 0.682 | 0.664 | 0.669 | 0.637 | 0.626 | 0.617 | 0.668 | 0.665 | 0.666 |
| 2 | 0.625 | 0.643 | 0.649 | 0.652 | 0.669 | 0.651 | 0.633 | 0.622 | 0.602 | 0.592 | 0.663 | 0.662 | 0.663 |
Accuracy Across Categories
| Class | CL1 | CL2 | GP1 | GT1 | GT2 | GT3 | LRPPI1 | LRPPI2 | LRPPI3 | LRPPI4 | ME1 | ME2 | ME3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BAROQUE | 0.616 | 0.637 | 0.750 | 0.669 | 0.724 | 0.673 | 0.673 | 0.660 | 0.666 | 0.629 | 0.754 | 0.759 | 0.757 |
| BLUES | 0.711 | 0.741 | 0.674 | 0.690 | 0.677 | 0.701 | 0.700 | 0.703 | 0.713 | 0.689 | 0.713 | 0.706 | 0.706 |
| CLASSICAL | 0.608 | 0.598 | 0.592 | 0.559 | 0.649 | 0.606 | 0.563 | 0.603 | 0.559 | 0.524 | 0.666 | 0.669 | 0.672 |
| COUNTRY | 0.624 | 0.596 | 0.697 | 0.793 | 0.830 | 0.679 | 0.669 | 0.640 | 0.621 | 0.617 | 0.660 | 0.656 | 0.653 |
| EDANCE | 0.560 | 0.591 | 0.536 | 0.590 | 0.624 | 0.648 | 0.672 | 0.626 | 0.646 | 0.686 | 0.657 | 0.649 | 0.639 |
| JAZZ | 0.679 | 0.699 | 0.606 | 0.627 | 0.682 | 0.626 | 0.640 | 0.602 | 0.566 | 0.574 | 0.679 | 0.676 | 0.680 |
| METAL | 0.677 | 0.709 | 0.750 | 0.713 | 0.656 | 0.733 | 0.707 | 0.642 | 0.623 | 0.643 | 0.612 | 0.627 | 0.629 |
| RAPHIPHOP | 0.809 | 0.823 | 0.873 | 0.846 | 0.846 | 0.854 | 0.860 | 0.837 | 0.826 | 0.848 | 0.841 | 0.836 | 0.837 |
| ROCKROLL | 0.420 | 0.418 | 0.406 | 0.384 | 0.414 | 0.447 | 0.448 | 0.391 | 0.377 | 0.391 | 0.450 | 0.450 | 0.448 |
| ROMANTIC | 0.501 | 0.527 | 0.508 | 0.602 | 0.540 | 0.597 | 0.574 | 0.523 | 0.488 | 0.444 | 0.510 | 0.505 | 0.500 |
MIREX 2008 Audio Genre Classification Evaluation Logs and Confusion Matrices
MIREX 2008 Audio Genre Classification Run Times
| Participant | Runtime (hh:mm) / Fold |
|---|---|
| CL1 | Feat Ex: 01:29 Train/Classify: 0:33 |
| CL2 | Feat Ex: 01:31 Train/Classify: 01:01 |
| GP1 | Feat Ex: 11:37 Train/Classify: 00:25 |
| GT1 | Feat Ex/Train/Classify: 00:36 |
| GT2 | Feat Ex/Train/Classify: 00:35 |
| GT3 | Feat Ex: 00:12 Train/Classify: 00:01 |
| LRPPI1 | Feat Ex: 28:50 Train/Classify: 00:02 |
| LRPPI2 | Feat Ex: 28:50 Train/Classify: 00:17 |
| LRPPI3 | Feat Ex: 28:50 Train/Classify: 00:20 |
| LRPPI4 | Feat Ex: 28:50 Train/Classify: 00:35 |
| ME1 | Feat Ex: 3:35 Train/Classify: 00:02 |
| ME2 | Feat Ex: 3:35 Train/Classify: 00:02 |
| ME3 | Feat Ex: 3:35 Train/Classify: 00:02 |
CSV Files Without Rounding
audiogenre_results_fold.csv
audiogenre_results_class.csv
Results By Algorithm
(.tar.gz)
CL1 = C. Cao, M. Li 1
CL2 = C. Cao, M. Li 2
LRPPI1 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 1
LRPPI2 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 2
LRPPI3 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 3
LRPPI4 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 4
ME1 = I. M. Mandel, D. P. W. Ellis 1
ME2 = I. M. Mandel, D. P. W. Ellis 2
ME3 = I. M. Mandel, D. P. W. Ellis 3
GP = G. Peeters
GT1 = G. Tzanetakis
GT2 = G. Tzanetakis
GT3 = G. Tzanetakis
Task 2 (LATIN) Results
MIREX 2008 Audio Genre Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds
| Participant | Average Classifcation Accuracy |
|---|---|
| CL1 | 65.17% |
| CL2 | 64.04% |
| GP1 | 62.72% |
| GT1 | 53.65% |
| GT2 | 53.79% |
| GT3 | 53.67% |
| LRPPI1 | 58.64% |
| LRPPI2 | 62.23% |
| LRPPI3 | 59.55% |
| LRPPI4 | 59.00% |
| ME1 | 54.15% |
| ME2 | 54.70% |
| ME3 | 54.99% |
Accuracy Across Folds
| Classification fold | CL1 | CL2 | GP1 | GT1 | GT2 | GT3 | LRPPI1 | LRPPI2 | LRPPI3 | LRPPI4 | ME1 | ME2 | ME3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0.755 | 0.750 | 0.694 | 0.674 | 0.677 | 0.657 | 0.661 | 0.697 | 0.671 | 0.680 | 0.681 | 0.684 | 0.685 |
| 1 | 0.541 | 0.528 | 0.553 | 0.435 | 0.435 | 0.422 | 0.512 | 0.573 | 0.526 | 0.506 | 0.403 | 0.409 | 0.415 |
| 2 | 0.660 | 0.644 | 0.634 | 0.501 | 0.502 | 0.531 | 0.587 | 0.597 | 0.590 | 0.585 | 0.541 | 0.548 | 0.550 |
Accuracy Across Categories
| Class | CL1 | CL2 | GP1 | GT1 | GT2 | GT3 | LRPPI1 | LRPPI2 | LRPPI3 | LRPPI4 | ME1 | ME2 | ME3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| axe | 0.753 | 0.745 | 0.558 | 0.637 | 0.640 | 0.695 | 0.529 | 0.560 | 0.537 | 0.528 | 0.679 | 0.679 | 0.681 |
| bachata | 0.957 | 0.622 | 0.969 | 0.595 | 0.592 | 0.587 | 0.957 | 0.950 | 0.956 | 0.957 | 0.932 | 0.935 | 0.935 |
| bolero | 0.630 | 0.633 | 0.768 | 0.702 | 0.705 | 0.746 | 0.683 | 0.726 | 0.646 | 0.668 | 0.664 | 0.662 | 0.666 |
| forro | 0.356 | 0.335 | 0.270 | 0.146 | 0.145 | 0.127 | 0.258 | 0.342 | 0.292 | 0.287 | 0.174 | 0.186 | 0.188 |
| gaucha | 0.501 | 0.491 | 0.345 | 0.348 | 0.348 | 0.299 | 0.345 | 0.357 | 0.327 | 0.338 | 0.435 | 0.436 | 0.434 |
| merengue | 0.895 | 0.898 | 0.897 | 0.812 | 0.806 | 0.784 | 0.847 | 0.794 | 0.825 | 0.833 | 0.698 | 0.699 | 0.728 |
| pagode | 0.355 | 0.368 | 0.303 | 0.307 | 0.297 | 0.249 | 0.322 | 0.391 | 0.361 | 0.318 | 0.231 | 0.240 | 0.243 |
| salsa | 0.886 | 0.850 | 0.750 | 0.715 | 0.719 | 0.668 | 0.788 | 0.793 | 0.769 | 0.766 | 0.698 | 0.710 | 0.716 |
| sertaneja | 0.209 | 0.205 | 0.200 | 0.159 | 0.186 | 0.212 | 0.132 | 0.227 | 0.210 | 0.170 | 0.090 | 0.090 | 0.094 |
| tango | 0.590 | 0.587 | 0.585 | 0.588 | 0.588 | 0.582 | 0.592 | 0.581 | 0.586 | 0.590 | 0.588 | 0.588 | 0.588 |
MIREX 2008 Audio Genre Classification Evaluation Logs and Confusion Matrices
MIREX 2008 Audio Genre Classification Run Times
| Participant | Runtime (hh:mm) / Fold |
|---|---|
| CL1 | Feat Ex: 00:47 Train/Classify: 0:13 |
| CL2 | Feat Ex: 00:48 Train/Classify: 00:23 |
| GP1 | Feat Ex: 07:12 Train/Classify: 00:15 |
| GT1 | Feat Ex/Train/Classify: 00:16 |
| GT2 | Feat Ex/Train/Classify: 00:17 |
| GT3 | Feat Ex: 00:06 Train/Classify: 00:00 (6 sec) |
| LRPPI1 | Feat Ex: 15:33 Train/Classify: 00:01 |
| LRPPI2 | Feat Ex: 15:33 Train/Classify: 00:06 |
| LRPPI3 | Feat Ex: 15:33 Train/Classify: 00:06 |
| LRPPI4 | Feat Ex: 15:33 Train/Classify: 00:11 |
| ME1 | Feat Ex: 1:58 Train/Classify: 00:00 (28 sec) |
| ME2 | Feat Ex: 1:58 Train/Classify: 00:00 (28 sec) |
| ME3 | Feat Ex: 1:58 Train/Classify: 00:00 (28 sec) |
CSV Files Without Rounding
audiolatin_results_fold.csv
audiolatin_results_class.csv
Results By Algorithm
(.tar.gz)
CL1 = C. Cao, M. Li 1
CL2 = C. Cao, M. Li 2
GP1 = G. Peeters
GT1 = G. Tzanetakis
GT2 = G. Tzanetakis
GT3 = G. Tzanetakis
LRPPI1 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 1
LRPPI2 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 2
LRPPI3 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 3
LRPPI4 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 4
ME1 = I. M. Mandel, D. P. W. Ellis 1
ME2 = I. M. Mandel, D. P. W. Ellis 2
ME3 = I. M. Mandel, D. P. W. Ellis 3
Friedman's Test for Significant Differences
Task 1 (Mixed) Classes vs. Systems
The Friedman test was run in MATLAB against the average accuracy for each class.
Friedman's Anova Table
| Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
|---|---|---|---|---|---|
| Columns | 243.6 | 10 | 24.36 | 22.15 | 0.0144 |
| Error | 856.4 | 90 | 9.5156 | ||
| Total | 1100 | 109 |
Tukey-Kramer HSD Multi-Comparison
The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| CL1 | CL2 | -5.5740 | -0.8000 | 3.9740 | FALSE |
| CL1 | GP1 | -5.1740 | -0.4000 | 4.3740 | FALSE |
| CL1 | GT1 | -5.0740 | -0.3000 | 4.4740 | FALSE |
| CL1 | GT2 | -3.3740 | 1.4000 | 6.1740 | FALSE |
| CL1 | GT3 | -3.5740 | 1.2000 | 5.9740 | FALSE |
| CL1 | LRPPI1 | -3.4740 | 1.3000 | 6.0740 | FALSE |
| CL1 | LRPPI2 | -2.3740 | 2.4000 | 7.1740 | FALSE |
| CL1 | LRPPI3 | -2.4740 | 2.3000 | 7.0740 | FALSE |
| CL1 | LRPPI4 | -1.1740 | 3.6000 | 8.3740 | FALSE |
| CL1 | ME1 | -1.1740 | 3.6000 | 8.3740 | FALSE |
| CL2 | GP1 | -4.3740 | 0.4000 | 5.1740 | FALSE |
| CL2 | GT1 | -4.2740 | 0.5000 | 5.2740 | FALSE |
| CL2 | GT2 | -2.5740 | 2.2000 | 6.9740 | FALSE |
| CL2 | GT3 | -2.7740 | 2.0000 | 6.7740 | FALSE |
| CL2 | LRPPI1 | -2.6740 | 2.1000 | 6.8740 | FALSE |
| CL2 | LRPPI2 | -1.5740 | 3.2000 | 7.9740 | FALSE |
| CL2 | LRPPI3 | -1.6740 | 3.1000 | 7.8740 | FALSE |
| CL2 | LRPPI4 | -0.3740 | 4.4000 | 9.1740 | FALSE |
| CL2 | ME1 | -0.3740 | 4.4000 | 9.1740 | FALSE |
| GP1 | GT1 | -4.6740 | 0.1000 | 4.8740 | FALSE |
| GP1 | GT2 | -2.9740 | 1.8000 | 6.5740 | FALSE |
| GP1 | GT3 | -3.1740 | 1.6000 | 6.3740 | FALSE |
| GP1 | LRPPI1 | -3.0740 | 1.7000 | 6.4740 | FALSE |
| GP1 | LRPPI2 | -1.9740 | 2.8000 | 7.5740 | FALSE |
| GP1 | LRPPI3 | -2.0740 | 2.7000 | 7.4740 | FALSE |
| GP1 | LRPPI4 | -0.7740 | 4.0000 | 8.7740 | FALSE |
| GP1 | ME1 | -0.7740 | 4.0000 | 8.7740 | FALSE |
| GT1 | GT2 | -3.0740 | 1.7000 | 6.4740 | FALSE |
| GT1 | GT3 | -3.2740 | 1.5000 | 6.2740 | FALSE |
| GT1 | LRPPI1 | -3.1740 | 1.6000 | 6.3740 | FALSE |
| GT1 | LRPPI2 | -2.0740 | 2.7000 | 7.4740 | FALSE |
| GT1 | LRPPI3 | -2.1740 | 2.6000 | 7.3740 | FALSE |
| GT1 | LRPPI4 | -0.8740 | 3.9000 | 8.6740 | FALSE |
| GT1 | ME1 | -0.8740 | 3.9000 | 8.6740 | FALSE |
| GT2 | GT3 | -4.9740 | -0.2000 | 4.5740 | FALSE |
| GT2 | LRPPI1 | -4.8740 | -0.1000 | 4.6740 | FALSE |
| GT2 | LRPPI2 | -3.7740 | 1.0000 | 5.7740 | FALSE |
| GT2 | LRPPI3 | -3.8740 | 0.9000 | 5.6740 | FALSE |
| GT2 | LRPPI4 | -2.5740 | 2.2000 | 6.9740 | FALSE |
| GT2 | ME1 | -2.5740 | 2.2000 | 6.9740 | FALSE |
| GT3 | LRPPI1 | -4.6740 | 0.1000 | 4.8740 | FALSE |
| GT3 | LRPPI2 | -3.5740 | 1.2000 | 5.9740 | FALSE |
| GT3 | LRPPI3 | -3.6740 | 1.1000 | 5.8740 | FALSE |
| GT3 | LRPPI4 | -2.3740 | 2.4000 | 7.1740 | FALSE |
| GT3 | ME1 | -2.3740 | 2.4000 | 7.1740 | FALSE |
| LRPPI1 | LRPPI2 | -3.6740 | 1.1000 | 5.8740 | FALSE |
| LRPPI1 | LRPPI3 | -3.7740 | 1.0000 | 5.7740 | FALSE |
| LRPPI1 | LRPPI4 | -2.4740 | 2.3000 | 7.0740 | FALSE |
| LRPPI1 | ME1 | -2.4740 | 2.3000 | 7.0740 | FALSE |
| LRPPI2 | LRPPI3 | -4.8740 | -0.1000 | 4.6740 | FALSE |
| LRPPI2 | LRPPI4 | -3.5740 | 1.2000 | 5.9740 | FALSE |
| LRPPI2 | ME1 | -3.5740 | 1.2000 | 5.9740 | FALSE |
| LRPPI3 | LRPPI4 | -3.4740 | 1.3000 | 6.0740 | FALSE |
| LRPPI3 | ME1 | -3.4740 | 1.3000 | 6.0740 | FALSE |
| LRPPI4 | ME1 | -4.7740 | 0.0000 | 4.7740 | FALSE |
Task 1 (Mixed) Folds vs. Systems
The Friedman test was run in MATLAB against the accuracy for each fold.
Friedman's Anova Table
| Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
|---|---|---|---|---|---|
| Columns | 255.333 | 10 | 25.5333 | 23.21 | 0.01 |
| Error | 74.667 | 20 | 3.7333 | ||
| Total | 330 | 32 |
Tukey-Kramer HSD Multi-Comparison
The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| CL1 | CL2 | -7.3828 | 1.3333 | 10.0495 | FALSE |
| CL1 | GP1 | -6.7162 | 2.0000 | 10.7162 | FALSE |
| CL1 | GT1 | -6.7162 | 2.0000 | 10.7162 | FALSE |
| CL1 | GT2 | -6.0495 | 2.6667 | 11.3828 | FALSE |
| CL1 | GT3 | -4.0495 | 4.6667 | 13.3828 | FALSE |
| CL1 | LRPPI1 | -3.7162 | 5.0000 | 13.7162 | FALSE |
| CL1 | LRPPI2 | -2.3828 | 6.3333 | 15.0495 | FALSE |
| CL1 | LRPPI3 | -1.7162 | 7.0000 | 15.7162 | FALSE |
| CL1 | LRPPI4 | -0.3828 | 8.3333 | 17.0495 | FALSE |
| CL1 | ME1 | -0.3828 | 8.3333 | 17.0495 | FALSE |
| CL2 | GP1 | -8.0495 | 0.6667 | 9.3828 | FALSE |
| CL2 | GT1 | -8.0495 | 0.6667 | 9.3828 | FALSE |
| CL2 | GT2 | -7.3828 | 1.3333 | 10.0495 | FALSE |
| CL2 | GT3 | -5.3828 | 3.3333 | 12.0495 | FALSE |
| CL2 | LRPPI1 | -5.0495 | 3.6667 | 12.3828 | FALSE |
| CL2 | LRPPI2 | -3.7162 | 5.0000 | 13.7162 | FALSE |
| CL2 | LRPPI3 | -3.0495 | 5.6667 | 14.3828 | FALSE |
| CL2 | LRPPI4 | -1.7162 | 7.0000 | 15.7162 | FALSE |
| CL2 | ME1 | -1.7162 | 7.0000 | 15.7162 | FALSE |
| GP1 | GT1 | -8.7162 | 0.0000 | 8.7162 | FALSE |
| GP1 | GT2 | -8.0495 | 0.6667 | 9.3828 | FALSE |
| GP1 | GT3 | -6.0495 | 2.6667 | 11.3828 | FALSE |
| GP1 | LRPPI1 | -5.7162 | 3.0000 | 11.7162 | FALSE |
| GP1 | LRPPI2 | -4.3828 | 4.3333 | 13.0495 | FALSE |
| GP1 | LRPPI3 | -3.7162 | 5.0000 | 13.7162 | FALSE |
| GP1 | LRPPI4 | -2.3828 | 6.3333 | 15.0495 | FALSE |
| GP1 | ME1 | -2.3828 | 6.3333 | 15.0495 | FALSE |
| GT1 | GT2 | -8.0495 | 0.6667 | 9.3828 | FALSE |
| GT1 | GT3 | -6.0495 | 2.6667 | 11.3828 | FALSE |
| GT1 | LRPPI1 | -5.7162 | 3.0000 | 11.7162 | FALSE |
| GT1 | LRPPI2 | -4.3828 | 4.3333 | 13.0495 | FALSE |
| GT1 | LRPPI3 | -3.7162 | 5.0000 | 13.7162 | FALSE |
| GT1 | LRPPI4 | -2.3828 | 6.3333 | 15.0495 | FALSE |
| GT1 | ME1 | -2.3828 | 6.3333 | 15.0495 | FALSE |
| GT2 | GT3 | -6.7162 | 2.0000 | 10.7162 | FALSE |
| GT2 | LRPPI1 | -6.3828 | 2.3333 | 11.0495 | FALSE |
| GT2 | LRPPI2 | -5.0495 | 3.6667 | 12.3828 | FALSE |
| GT2 | LRPPI3 | -4.3828 | 4.3333 | 13.0495 | FALSE |
| GT2 | LRPPI4 | -3.0495 | 5.6667 | 14.3828 | FALSE |
| GT2 | ME1 | -3.0495 | 5.6667 | 14.3828 | FALSE |
| GT3 | LRPPI1 | -8.3828 | 0.3333 | 9.0495 | FALSE |
| GT3 | LRPPI2 | -7.0495 | 1.6667 | 10.3828 | FALSE |
| GT3 | LRPPI3 | -6.3828 | 2.3333 | 11.0495 | FALSE |
| GT3 | LRPPI4 | -5.0495 | 3.6667 | 12.3828 | FALSE |
| GT3 | ME1 | -5.0495 | 3.6667 | 12.3828 | FALSE |
| LRPPI1 | LRPPI2 | -7.3828 | 1.3333 | 10.0495 | FALSE |
| LRPPI1 | LRPPI3 | -6.7162 | 2.0000 | 10.7162 | FALSE |
| LRPPI1 | LRPPI4 | -5.3828 | 3.3333 | 12.0495 | FALSE |
| LRPPI1 | ME1 | -5.3828 | 3.3333 | 12.0495 | FALSE |
| LRPPI2 | LRPPI3 | -8.0495 | 0.6667 | 9.3828 | FALSE |
| LRPPI2 | LRPPI4 | -6.7162 | 2.0000 | 10.7162 | FALSE |
| LRPPI2 | ME1 | -6.7162 | 2.0000 | 10.7162 | FALSE |
| LRPPI3 | LRPPI4 | -7.3828 | 1.3333 | 10.0495 | FALSE |
| LRPPI3 | ME1 | -7.3828 | 1.3333 | 10.0495 | FALSE |
| LRPPI4 | ME1 | -8.7162 | 0.0000 | 8.7162 | FALSE |
Task 2 (Latin) Classes vs. Systems
The Friedman test was run in MATLAB against the average accuracy for each class.
Friedman's Anova Table
| Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
|---|---|---|---|---|---|
| Columns | 235 | 10 | 23.5 | 21.38 | 0.0186 |
| Error | 864 | 90 | 9.6 | ||
| Total | 1099 | 109 |
Tukey-Kramer HSD Multi-Comparison
The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| CL1 | CL2 | -3.7219 | 1.0500 | 5.8219 | FALSE |
| CL1 | GP1 | -3.2219 | 1.5500 | 6.3219 | FALSE |
| CL1 | GT1 | -2.3219 | 2.4500 | 7.2219 | FALSE |
| CL1 | GT2 | -1.7219 | 3.0500 | 7.8219 | FALSE |
| CL1 | GT3 | -1.7719 | 3.0000 | 7.7719 | FALSE |
| CL1 | LRPPI1 | -2.1219 | 2.6500 | 7.4219 | FALSE |
| CL1 | LRPPI2 | -0.2219 | 4.5500 | 9.3219 | FALSE |
| CL1 | LRPPI3 | -0.7719 | 4.0000 | 8.7719 | FALSE |
| CL1 | LRPPI4 | -0.6719 | 4.1000 | 8.8719 | FALSE |
| CL1 | ME1 | 0.1781 | 4.9500 | 9.7219 | TRUE |
| CL2 | GP1 | -4.2719 | 0.5000 | 5.2719 | FALSE |
| CL2 | GT1 | -3.3719 | 1.4000 | 6.1719 | FALSE |
| CL2 | GT2 | -2.7719 | 2.0000 | 6.7719 | FALSE |
| CL2 | GT3 | -2.8219 | 1.9500 | 6.7219 | FALSE |
| CL2 | LRPPI1 | -3.1719 | 1.6000 | 6.3719 | FALSE |
| CL2 | LRPPI2 | -1.2719 | 3.5000 | 8.2719 | FALSE |
| CL2 | LRPPI3 | -1.8219 | 2.9500 | 7.7219 | FALSE |
| CL2 | LRPPI4 | -1.7219 | 3.0500 | 7.8219 | FALSE |
| CL2 | ME1 | -0.8719 | 3.9000 | 8.6719 | FALSE |
| GP1 | GT1 | -3.8719 | 0.9000 | 5.6719 | FALSE |
| GP1 | GT2 | -3.2719 | 1.5000 | 6.2719 | FALSE |
| GP1 | GT3 | -3.3219 | 1.4500 | 6.2219 | FALSE |
| GP1 | LRPPI1 | -3.6719 | 1.1000 | 5.8719 | FALSE |
| GP1 | LRPPI2 | -1.7719 | 3.0000 | 7.7719 | FALSE |
| GP1 | LRPPI3 | -2.3219 | 2.4500 | 7.2219 | FALSE |
| GP1 | LRPPI4 | -2.2219 | 2.5500 | 7.3219 | FALSE |
| GP1 | ME1 | -1.3719 | 3.4000 | 8.1719 | FALSE |
| GT1 | GT2 | -4.1719 | 0.6000 | 5.3719 | FALSE |
| GT1 | GT3 | -4.2219 | 0.5500 | 5.3219 | FALSE |
| GT1 | LRPPI1 | -4.5719 | 0.2000 | 4.9719 | FALSE |
| GT1 | LRPPI2 | -2.6719 | 2.1000 | 6.8719 | FALSE |
| GT1 | LRPPI3 | -3.2219 | 1.5500 | 6.3219 | FALSE |
| GT1 | LRPPI4 | -3.1219 | 1.6500 | 6.4219 | FALSE |
| GT1 | ME1 | -2.2719 | 2.5000 | 7.2719 | FALSE |
| GT2 | GT3 | -4.8219 | -0.0500 | 4.7219 | FALSE |
| GT2 | LRPPI1 | -5.1719 | -0.4000 | 4.3719 | FALSE |
| GT2 | LRPPI2 | -3.2719 | 1.5000 | 6.2719 | FALSE |
| GT2 | LRPPI3 | -3.8219 | 0.9500 | 5.7219 | FALSE |
| GT2 | LRPPI4 | -3.7219 | 1.0500 | 5.8219 | FALSE |
| GT2 | ME1 | -2.8719 | 1.9000 | 6.6719 | FALSE |
| GT3 | LRPPI1 | -5.1219 | -0.3500 | 4.4219 | FALSE |
| GT3 | LRPPI2 | -3.2219 | 1.5500 | 6.3219 | FALSE |
| GT3 | LRPPI3 | -3.7719 | 1.0000 | 5.7719 | FALSE |
| GT3 | LRPPI4 | -3.6719 | 1.1000 | 5.8719 | FALSE |
| GT3 | ME1 | -2.8219 | 1.9500 | 6.7219 | FALSE |
| LRPPI1 | LRPPI2 | -2.8719 | 1.9000 | 6.6719 | FALSE |
| LRPPI1 | LRPPI3 | -3.4219 | 1.3500 | 6.1219 | FALSE |
| LRPPI1 | LRPPI4 | -3.3219 | 1.4500 | 6.2219 | FALSE |
| LRPPI1 | ME1 | -2.4719 | 2.3000 | 7.0719 | FALSE |
| LRPPI2 | LRPPI3 | -5.3219 | -0.5500 | 4.2219 | FALSE |
| LRPPI2 | LRPPI4 | -5.2219 | -0.4500 | 4.3219 | FALSE |
| LRPPI2 | ME1 | -4.3719 | 0.4000 | 5.1719 | FALSE |
| LRPPI3 | LRPPI4 | -4.6719 | 0.1000 | 4.8719 | FALSE |
| LRPPI3 | ME1 | -3.8219 | 0.9500 | 5.7219 | FALSE |
| LRPPI4 | ME1 | -3.9219 | 0.8500 | 5.6219 | FALSE |
Task 2 (Latin) Folds vs. Systems
The Friedman test was run in MATLAB against the accuracy for each fold.
Friedman's Anova Table
| Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
|---|---|---|---|---|---|
| Columns | 265.833 | 10 | 26.5833 | 24.2 | 0.0071 |
| Error | 63.667 | 20 | 3.1833 | ||
| Total | 329.5 | 32 |
Tukey-Kramer HSD Multi-Comparison
The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| CL1 | CL2 | -7.7095 | 1.0000 | 9.7095 | FALSE |
| CL1 | GP1 | -7.3762 | 1.3333 | 10.0429 | FALSE |
| CL1 | GT1 | -7.7095 | 1.0000 | 9.7095 | FALSE |
| CL1 | GT2 | -4.0429 | 4.6667 | 13.3762 | FALSE |
| CL1 | GT3 | -3.7095 | 5.0000 | 13.7095 | FALSE |
| CL1 | LRPPI1 | -3.0429 | 5.6667 | 14.3762 | FALSE |
| CL1 | LRPPI2 | -2.3762 | 6.3333 | 15.0429 | FALSE |
| CL1 | LRPPI3 | -1.8762 | 6.8333 | 15.5429 | FALSE |
| CL1 | LRPPI4 | -0.3762 | 8.3333 | 17.0429 | FALSE |
| CL1 | ME1 | -1.2095 | 7.5000 | 16.2095 | FALSE |
| CL2 | GP1 | -8.3762 | 0.3333 | 9.0429 | FALSE |
| CL2 | GT1 | -8.7095 | 0.0000 | 8.7095 | FALSE |
| CL2 | GT2 | -5.0429 | 3.6667 | 12.3762 | FALSE |
| CL2 | GT3 | -4.7095 | 4.0000 | 12.7095 | FALSE |
| CL2 | LRPPI1 | -4.0429 | 4.6667 | 13.3762 | FALSE |
| CL2 | LRPPI2 | -3.3762 | 5.3333 | 14.0429 | FALSE |
| CL2 | LRPPI3 | -2.8762 | 5.8333 | 14.5429 | FALSE |
| CL2 | LRPPI4 | -1.3762 | 7.3333 | 16.0429 | FALSE |
| CL2 | ME1 | -2.2095 | 6.5000 | 15.2095 | FALSE |
| GP1 | GT1 | -9.0429 | -0.3333 | 8.3762 | FALSE |
| GP1 | GT2 | -5.3762 | 3.3333 | 12.0429 | FALSE |
| GP1 | GT3 | -5.0429 | 3.6667 | 12.3762 | FALSE |
| GP1 | LRPPI1 | -4.3762 | 4.3333 | 13.0429 | FALSE |
| GP1 | LRPPI2 | -3.7095 | 5.0000 | 13.7095 | FALSE |
| GP1 | LRPPI3 | -3.2095 | 5.5000 | 14.2095 | FALSE |
| GP1 | LRPPI4 | -1.7095 | 7.0000 | 15.7095 | FALSE |
| GP1 | ME1 | -2.5429 | 6.1667 | 14.8762 | FALSE |
| GT1 | GT2 | -5.0429 | 3.6667 | 12.3762 | FALSE |
| GT1 | GT3 | -4.7095 | 4.0000 | 12.7095 | FALSE |
| GT1 | LRPPI1 | -4.0429 | 4.6667 | 13.3762 | FALSE |
| GT1 | LRPPI2 | -3.3762 | 5.3333 | 14.0429 | FALSE |
| GT1 | LRPPI3 | -2.8762 | 5.8333 | 14.5429 | FALSE |
| GT1 | LRPPI4 | -1.3762 | 7.3333 | 16.0429 | FALSE |
| GT1 | ME1 | -2.2095 | 6.5000 | 15.2095 | FALSE |
| GT2 | GT3 | -8.3762 | 0.3333 | 9.0429 | FALSE |
| GT2 | LRPPI1 | -7.7095 | 1.0000 | 9.7095 | FALSE |
| GT2 | LRPPI2 | -7.0429 | 1.6667 | 10.3762 | FALSE |
| GT2 | LRPPI3 | -6.5429 | 2.1667 | 10.8762 | FALSE |
| GT2 | LRPPI4 | -5.0429 | 3.6667 | 12.3762 | FALSE |
| GT2 | ME1 | -5.8762 | 2.8333 | 11.5429 | FALSE |
| GT3 | LRPPI1 | -8.0429 | 0.6667 | 9.3762 | FALSE |
| GT3 | LRPPI2 | -7.3762 | 1.3333 | 10.0429 | FALSE |
| GT3 | LRPPI3 | -6.8762 | 1.8333 | 10.5429 | FALSE |
| GT3 | LRPPI4 | -5.3762 | 3.3333 | 12.0429 | FALSE |
| GT3 | ME1 | -6.2095 | 2.5000 | 11.2095 | FALSE |
| LRPPI1 | LRPPI2 | -8.0429 | 0.6667 | 9.3762 | FALSE |
| LRPPI1 | LRPPI3 | -7.5429 | 1.1667 | 9.8762 | FALSE |
| LRPPI1 | LRPPI4 | -6.0429 | 2.6667 | 11.3762 | FALSE |
| LRPPI1 | ME1 | -6.8762 | 1.8333 | 10.5429 | FALSE |
| LRPPI2 | LRPPI3 | -8.2095 | 0.5000 | 9.2095 | FALSE |
| LRPPI2 | LRPPI4 | -6.7095 | 2.0000 | 10.7095 | FALSE |
| LRPPI2 | ME1 | -7.5429 | 1.1667 | 9.8762 | FALSE |
| LRPPI3 | LRPPI4 | -7.2095 | 1.5000 | 10.2095 | FALSE |
| LRPPI3 | ME1 | -8.0429 | 0.6667 | 9.3762 | FALSE |
| LRPPI4 | ME1 | -9.5429 | -0.8333 | 7.8762 | FALSE |
