Difference between revisions of "2009:Audio Music Similarity and Retrieval Results"

From MIREX Wiki
(→‎Overall Summary Results)
(→‎Reports)
 
(38 intermediate revisions by 7 users not shown)
Line 1: Line 1:
 
== Introduction ==
 
== Introduction ==
 +
These are the results for the 2009 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.
  
 +
Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 100 songs were randomly selected from the 10 genre groups (10 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical '''BROAD''' score with 3 categories: NS,SS,VS as explained below, and one '''FINE''' score (in the range from 0 to 10). A description and analysis is provided below.
 +
 +
The systems read in 30 second audio clips as their raw data. The same 30 second clips were used in the grading stage.
  
  
Line 8: Line 12:
 
==== Team ID ====
 
==== Team ID ====
  
−
'''ANO''' = [[Anonymous]]<br />
+
'''ANO''' = [https://www.music-ir.org/mirex/abstracts/2009/ANO_train_simi.pdf Anonymous]<br />
−
'''BF''' = [[Benjamin Fields]]<br />
+
'''BF1''' = [https://www.music-ir.org/mirex/abstracts/2009/BF.pdf Benjamin Fields (chr12)]<br />
−
'''BSWH''' = [[Dmitry Bogdanov, Joan Serrà, Nicolas Wack, and Perfecto Herrera]]<br />
+
'''BF2''' = [https://www.music-ir.org/mirex/abstracts/2009/BF.pdf Benjamin Fields (mfcc10)]<br />
−
'''CL1''' = [[Chuan Cao, Ming Li]]<br />
+
'''BSWH1''' = [https://www.music-ir.org/mirex/abstracts/2009/MIREX2009-sim-BSWH1-BSWH2.pdf Dmitry Bogdanov, Joan Serrà, Nicolas Wack, Perfecto Herrera (clas)]<br />
−
'''CL2''' = [[Chuan Cao, Ming Li]]<br />
+
'''BSWH2''' = [https://www.music-ir.org/mirex/abstracts/2009/MIREX2009-sim-BSWH1-BSWH2.pdf Dmitry Bogdanov, Joan Serrà, Nicolas Wack, Perfecto Herrera (hybrid)]<br />
−
'''GT''' = [[George Tzanetakis]]<br />
+
'''CL1''' = [https://www.music-ir.org/mirex/abstracts/2009/CL.pdf Chuan Cao, Ming Li]<br />
−
'''LR''' = [[Thomas Lidy,Andreas Rauber]]<br />
+
'''CL2''' = [https://www.music-ir.org/mirex/abstracts/2009/CL.pdf Chuan Cao, Ming Li]<br />
−
'''ME'''  = [[Franc┬╕ois Maillet,Douglas Eck]]<br />
+
'''GT''' = [https://www.music-ir.org/mirex/abstracts/2009/GTfinal.pdf George Tzanetakis]<br />
−
'''PS1''' = [[Tim Pohle1, Dominik Schnitzer1]]<br />
+
'''LR''' = [https://www.music-ir.org/mirex/abstracts/2009/LR.pdf Thomas Lidy, Andreas Rauber]<br />
−
'''PS2''' = [[Tim Pohle1, Dominik Schnitzer1]]<br />
+
'''ME1'''  = [https://www.music-ir.org/mirex/abstracts/2009/ME.pdf François Maillet, Douglas Eck (mlp)]<br />
−
'''SH''' = [[Stephan H├╝bler]]<br />
+
'''ME2'''  = [https://www.music-ir.org/mirex/abstracts/2009/ME.pdf François Maillet, Douglas Eck (sda)]<br />
 +
'''PS1''' = [https://www.music-ir.org/mirex/abstracts/2009/PS.pdf Tim Pohle, Dominik Schnitzer (2007)]<br />
 +
'''PS2''' = [https://www.music-ir.org/mirex/abstracts/2009/PS.pdf Tim Pohle, Dominik Schnitzer (2009)]<br />
 +
'''SH1''' = [https://www.music-ir.org/mirex/abstracts/2009/SH.pdf Stephan Hübler]<br />
 +
'''SH2''' = [https://www.music-ir.org/mirex/abstracts/2009/SH.pdf Stephan Hübler]<br />
  
 
====Broad Categories====
 
====Broad Categories====
Line 25: Line 33:
 
'''VS''' = Very Similar<br />
 
'''VS''' = Very Similar<br />
  
−
=====Calculating Summary Measures=====
+
=====Understanding Summary Measures=====
−
'''Fine'''<sup>(1)</sup> = Sum of fine-grained human similarity decisions (0-10). <br />
+
'''Fine''' = Has a range from 0 (failure) to 10 (perfection). <br />
−
'''PSum'''<sup>(1)</sup> = Sum of human broad similarity decisions: NS=0, SS=1, VS=2. <br />
+
'''Broad''' = Has a range from 0 (failure) to 2 (perfection) as each query/candidate pair is scored with either NS=0, SS=1 or VS=2. <br />
−
'''WCsum'''<sup>(1)</sup> = 'World Cup' scoring: NS=0, SS=1, VS=3 (rewards Very Similar). <br />
+
 
−
'''SDsum'''<sup>(1)</sup> = 'Stephen Downie' scoring: NS=0, SS=1, VS=4 (strongly rewards Very Similar). <br />
+
==Human Evaluation==
−
'''Greater0'''<sup>(1)</sup> = NS=0, SS=1, VS=1 (binary relevance judgement).<br />
+
===Overall Summary Results===
−
'''Greater1'''<sup>(1)</sup> = NS=0, SS=0, VS=1 (binary relevance judgement using only Very Similar).<br />
 
  
−
<sup>(1)</sup>Normalized to the range 0 to 1.
+
<csv p=3>2009/ams/evalutron/summary_evalutron.csv</csv>
  
−
===Overall Summary Results===
+
===Friedman's Tests===
−
'''NB''': The results for BK2 were interpolated from partial data due to a runtime error.
+
====Friedman's Test (FINE Scores)====
 +
The Friedman test was run in MATLAB against the '''Fine''' summary data over the 100 queries.<br />
 +
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 +
 
 +
<csv p=3>2009/ams/evalutron/evalutron.fine.friedman.tukeyKramerHSD.csv</csv>
  
−
<csv>ams/evalutron/summary_evalutron.csv</csv>
+
https://music-ir.org/mirex/results/2009/ams/evalutron/small.evalutron.fine.friedman.tukeyKramerHSD.png
  
−
===Friedman Test with Multiple Comparisons Results (p=0.05)===
+
====Friedman's Test (BROAD Scores)====
−
The Friedman test was run in MATLAB against the Fine summary data over the 100 queries.<br />
+
The Friedman test was run in MATLAB against the '''BROAD''' summary data over the 100 queries.<br />
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
−
<csv>ams07_sum_friedman_fine.csv</csv>
 
−
<csv>ams07_detail_friedman_fine.csv</csv>
 
  
−
[[Image:2007 ams broad scores friedmans.png]]
+
<csv p=3>2009/ams/evalutron/evalutron.cat.friedman.tukeyKramerHSD.csv</csv>
 +
 
 +
https://music-ir.org/mirex/results/2009/ams/evalutron/small.evalutron.cat.friedman.tukeyKramerHSD.png
 +
 
  
 
===Summary Results by Query===
 
===Summary Results by Query===
 +
====FINE Scores====
 
These are the mean FINE scores per query assigned by Evalutron graders. The FINE scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0.0 and 10.0. A perfect score would be 10. Genre labels have been included for reference.  
 
These are the mean FINE scores per query assigned by Evalutron graders. The FINE scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0.0 and 10.0. A perfect score would be 10. Genre labels have been included for reference.  
  
−
<csv>ams07_fine_by_query_with_genre.csv</csv>
+
<csv p=3>2009/ams/evalutron/fine_scores.csv</csv>
  
 +
====BROAD Scores====
 
These are the mean BROAD scores per query assigned by Evalutron graders. The BROAD scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 (not similar) and 2 (very similar). A perfect score would be 2. Genre labels have been included for reference.  
 
These are the mean BROAD scores per query assigned by Evalutron graders. The BROAD scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 (not similar) and 2 (very similar). A perfect score would be 2. Genre labels have been included for reference.  
−
<csv>ams07_broad_by_query_with_genre.csv</csv>
 
  
−
===Anonymized Metadata===
+
<csv p=3>2009/ams/evalutron/cat_scores.csv</csv>
−
[https://www.music-ir.org/mirex2007/results/anonymizedAudioSim07metaData.csv Anonymized Metadata]<br />
+
 
 +
 
  
 
===Raw Scores===
 
===Raw Scores===
−
The raw data derived from the Evalutron 6000 human evaluations are located on the [[Audio Music Similarity and Retrieval Raw Data]] page.
+
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2009:Audio Music Similarity and Retrieval Raw Data]] page.
 +
 
 +
==Metadata and Distance Space Evaluation==
 +
The following reports provide evaluation statistics based on analysis of the distance space and metadata matches and  include:
 +
* Neighbourhood clustering by artist, album and genre
 +
* Artist-filtered genre clustering
 +
* How often the triangular inequality holds
 +
* Statistics on 'hubs' (tracks similar to many tracks) and orphans (tracks that are not similar to any other tracks at N results).
 +
 
 +
=== Reports ===
 +
 
 +
'''ANO''' = [https://music-ir.org/mirex/results/2009/ams/statistics/ANO/report.txt Anonymous]<br />
 +
'''BF1''' = [https://music-ir.org/mirex/results/2009/ams/statistics/BF1/report.txt Benjamin Fields (chr12)]<br />
 +
'''BF2''' = [https://music-ir.org/mirex/results/2009/ams/statistics/BF2/report.txt Benjamin Fields (mfcc10)]<br />
 +
'''BSWH1''' = [https://music-ir.org/mirex/results/2009/ams/statistics/BSWH1/report.txt Dmitry Bogdanov, Joan Serrà, Nicolas Wack, and Perfecto Herrera (clas)]<br />
 +
'''BSWH2''' = [https://music-ir.org/mirex/results/2009/ams/statistics/BSWH2/report.txt Dmitry Bogdanov, Joan Serrà, Nicolas Wack, and Perfecto Herrera (hybrid)]<br />
 +
'''CL1''' = [https://music-ir.org/mirex/results/2009/ams/statistics/CL1/report.txt Chuan Cao, Ming Li]<br />
 +
'''CL2''' = [https://music-ir.org/mirex/results/2009/ams/statistics/CL2/report.txt Chuan Cao, Ming Li]<br />
 +
'''GT''' = [https://music-ir.org/mirex/results/2009/ams/statistics/GT/report.txt George Tzanetakis]<br />
 +
'''LR''' = [https://music-ir.org/mirex/results/2009/ams/statistics/LR/report.txt Thomas Lidy, Andreas Rauber]]<br />
 +
'''ME1'''  = [https://music-ir.org/mirex/results/2009/ams/statistics/ME1/report.txt François Maillet, Douglas Eck (mlp)]<br />
 +
'''ME2'''  = [https://music-ir.org/mirex/results/2009/ams/statistics/ME2/report.txt François Maillet, Douglas Eck (sda)]<br />
 +
'''PS1''' = [https://music-ir.org/mirex/results/2009/ams/statistics/PS1/report.txt Tim Pohle, Dominik Schnitzer (2007)]<br />
 +
'''PS2''' = [https://music-ir.org/mirex/results/2009/ams/statistics/PS2/report.txt Tim Pohle, Dominik Schnitzer (2009)]<br />
 +
'''SH1''' = [https://music-ir.org/mirex/results/2009/ams/statistics/SH1/report.txt Stephan Hübler]<br />
 +
'''SH2''' = [https://music-ir.org/mirex/results/2009/ams/statistics/SH2/report.txt Stephan Hübler]<br />
 +
 
 +
== Run Times ==
 +
<csv>2009/ams/audiosim.runtime.csv</csv>

Latest revision as of 15:31, 23 July 2010

Introduction

These are the results for the 2009 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.

Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 100 songs were randomly selected from the 10 genre groups (10 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical BROAD score with 3 categories: NS,SS,VS as explained below, and one FINE score (in the range from 0 to 10). A description and analysis is provided below.

The systems read in 30 second audio clips as their raw data. The same 30 second clips were used in the grading stage.


General Legend

Team ID

ANO = Anonymous
BF1 = Benjamin Fields (chr12)
BF2 = Benjamin Fields (mfcc10)
BSWH1 = Dmitry Bogdanov, Joan Serrà, Nicolas Wack, Perfecto Herrera (clas)
BSWH2 = Dmitry Bogdanov, Joan Serrà, Nicolas Wack, Perfecto Herrera (hybrid)
CL1 = Chuan Cao, Ming Li
CL2 = Chuan Cao, Ming Li
GT = George Tzanetakis
LR = Thomas Lidy, Andreas Rauber
ME1 = François Maillet, Douglas Eck (mlp)
ME2 = François Maillet, Douglas Eck (sda)
PS1 = Tim Pohle, Dominik Schnitzer (2007)
PS2 = Tim Pohle, Dominik Schnitzer (2009)
SH1 = Stephan Hübler
SH2 = Stephan Hübler

Broad Categories

NS = Not Similar
SS = Somewhat Similar
VS = Very Similar

Understanding Summary Measures

Fine = Has a range from 0 (failure) to 10 (perfection).
Broad = Has a range from 0 (failure) to 2 (perfection) as each query/candidate pair is scored with either NS=0, SS=1 or VS=2.

Human Evaluation

Overall Summary Results

Measure ANO BF1 BF2 BSWH1 BSWH2 CL1 CL2 GT LR ME1 ME2 PS1 PS2 SH1 SH2
Average FINE Score 5.391 2.401 2.587 5.137 5.734 2.525 5.392 5.343 5.470 2.331 2.585 5.751 6.458 5.042 4.932
Average BROAD Score 1.126 0.416 0.410 1.094 1.232 0.476 1.164 1.126 1.148 0.356 0.418 1.262 1.448 1.012 1.040

download these results as csv

Friedman's Tests

Friedman's Test (FINE Scores)

The Friedman test was run in MATLAB against the Fine summary data over the 100 queries.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

TeamID TeamID Lowerbound Mean Upperbound Significance
PS2 PS1 -0.298 1.845 3.988 FALSE
PS2 BSWH2 -0.278 1.865 4.008 FALSE
PS2 LR 0.502 2.645 4.788 TRUE
PS2 CL2 0.917 3.060 5.203 TRUE
PS2 ANO 0.767 2.910 5.053 TRUE
PS2 GT 0.817 2.960 5.103 TRUE
PS2 BSWH1 1.642 3.785 5.928 TRUE
PS2 SH1 1.617 3.760 5.903 TRUE
PS2 SH2 1.837 3.980 6.123 TRUE
PS2 BF2 6.507 8.650 10.793 TRUE
PS2 ME2 6.557 8.700 10.843 TRUE
PS2 CL1 6.627 8.770 10.913 TRUE
PS2 BF1 6.857 9.000 11.143 TRUE
PS2 ME1 6.952 9.095 11.238 TRUE
PS1 BSWH2 -2.123 0.020 2.163 FALSE
PS1 LR -1.343 0.800 2.943 FALSE
PS1 CL2 -0.928 1.215 3.358 FALSE
PS1 ANO -1.078 1.065 3.208 FALSE
PS1 GT -1.028 1.115 3.258 FALSE
PS1 BSWH1 -0.203 1.940 4.083 FALSE
PS1 SH1 -0.228 1.915 4.058 FALSE
PS1 SH2 -0.008 2.135 4.278 FALSE
PS1 BF2 4.662 6.805 8.948 TRUE
PS1 ME2 4.712 6.855 8.998 TRUE
PS1 CL1 4.782 6.925 9.068 TRUE
PS1 BF1 5.012 7.155 9.298 TRUE
PS1 ME1 5.107 7.250 9.393 TRUE
BSWH2 LR -1.363 0.780 2.923 FALSE
BSWH2 CL2 -0.948 1.195 3.338 FALSE
BSWH2 ANO -1.098 1.045 3.188 FALSE
BSWH2 GT -1.048 1.095 3.238 FALSE
BSWH2 BSWH1 -0.223 1.920 4.063 FALSE
BSWH2 SH1 -0.248 1.895 4.038 FALSE
BSWH2 SH2 -0.028 2.115 4.258 FALSE
BSWH2 BF2 4.642 6.785 8.928 TRUE
BSWH2 ME2 4.692 6.835 8.978 TRUE
BSWH2 CL1 4.762 6.905 9.048 TRUE
BSWH2 BF1 4.992 7.135 9.278 TRUE
BSWH2 ME1 5.087 7.230 9.373 TRUE
LR CL2 -1.728 0.415 2.558 FALSE
LR ANO -1.878 0.265 2.408 FALSE
LR GT -1.828 0.315 2.458 FALSE
LR BSWH1 -1.003 1.140 3.283 FALSE
LR SH1 -1.028 1.115 3.258 FALSE
LR SH2 -0.808 1.335 3.478 FALSE
LR BF2 3.862 6.005 8.148 TRUE
LR ME2 3.912 6.055 8.198 TRUE
LR CL1 3.982 6.125 8.268 TRUE
LR BF1 4.212 6.355 8.498 TRUE
LR ME1 4.307 6.450 8.593 TRUE
CL2 ANO -2.293 -0.150 1.993 FALSE
CL2 GT -2.243 -0.100 2.043 FALSE
CL2 BSWH1 -1.418 0.725 2.868 FALSE
CL2 SH1 -1.443 0.700 2.843 FALSE
CL2 SH2 -1.223 0.920 3.063 FALSE
CL2 BF2 3.447 5.590 7.733 TRUE
CL2 ME2 3.497 5.640 7.783 TRUE
CL2 CL1 3.567 5.710 7.853 TRUE
CL2 BF1 3.797 5.940 8.083 TRUE
CL2 ME1 3.892 6.035 8.178 TRUE
ANO GT -2.093 0.050 2.193 FALSE
ANO BSWH1 -1.268 0.875 3.018 FALSE
ANO SH1 -1.293 0.850 2.993 FALSE
ANO SH2 -1.073 1.070 3.213 FALSE
ANO BF2 3.597 5.740 7.883 TRUE
ANO ME2 3.647 5.790 7.933 TRUE
ANO CL1 3.717 5.860 8.003 TRUE
ANO BF1 3.947 6.090 8.233 TRUE
ANO ME1 4.042 6.185 8.328 TRUE
GT BSWH1 -1.318 0.825 2.968 FALSE
GT SH1 -1.343 0.800 2.943 FALSE
GT SH2 -1.123 1.020 3.163 FALSE
GT BF2 3.547 5.690 7.833 TRUE
GT ME2 3.597 5.740 7.883 TRUE
GT CL1 3.667 5.810 7.953 TRUE
GT BF1 3.897 6.040 8.183 TRUE
GT ME1 3.992 6.135 8.278 TRUE
BSWH1 SH1 -2.168 -0.025 2.118 FALSE
BSWH1 SH2 -1.948 0.195 2.338 FALSE
BSWH1 BF2 2.722 4.865 7.008 TRUE
BSWH1 ME2 2.772 4.915 7.058 TRUE
BSWH1 CL1 2.842 4.985 7.128 TRUE
BSWH1 BF1 3.072 5.215 7.358 TRUE
BSWH1 ME1 3.167 5.310 7.453 TRUE
SH1 SH2 -1.923 0.220 2.363 FALSE
SH1 BF2 2.747 4.890 7.033 TRUE
SH1 ME2 2.797 4.940 7.083 TRUE
SH1 CL1 2.867 5.010 7.153 TRUE
SH1 BF1 3.097 5.240 7.383 TRUE
SH1 ME1 3.192 5.335 7.478 TRUE
SH2 BF2 2.527 4.670 6.813 TRUE
SH2 ME2 2.577 4.720 6.863 TRUE
SH2 CL1 2.647 4.790 6.933 TRUE
SH2 BF1 2.877 5.020 7.163 TRUE
SH2 ME1 2.972 5.115 7.258 TRUE
BF2 ME2 -2.093 0.050 2.193 FALSE
BF2 CL1 -2.023 0.120 2.263 FALSE
BF2 BF1 -1.793 0.350 2.493 FALSE
BF2 ME1 -1.698 0.445 2.588 FALSE
ME2 CL1 -2.073 0.070 2.213 FALSE
ME2 BF1 -1.843 0.300 2.443 FALSE
ME2 ME1 -1.748 0.395 2.538 FALSE
CL1 BF1 -1.913 0.230 2.373 FALSE
CL1 ME1 -1.818 0.325 2.468 FALSE
BF1 ME1 -2.048 0.095 2.238 FALSE

download these results as csv

https://music-ir.org/mirex/results/2009/ams/evalutron/small.evalutron.fine.friedman.tukeyKramerHSD.png

Friedman's Test (BROAD Scores)

The Friedman test was run in MATLAB against the BROAD summary data over the 100 queries.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

TeamID TeamID Lowerbound Mean Upperbound Significance
PS2 PS1 -0.352 1.730 3.812 FALSE
PS2 BSWH2 -0.052 2.030 4.112 FALSE
PS2 CL2 0.657 2.740 4.822 TRUE
PS2 LR 0.682 2.765 4.848 TRUE
PS2 ANO 1.157 3.240 5.322 TRUE
PS2 GT 0.802 2.885 4.968 TRUE
PS2 BSWH1 1.387 3.470 5.553 TRUE
PS2 SH2 1.742 3.825 5.907 TRUE
PS2 SH1 2.252 4.335 6.418 TRUE
PS2 CL1 6.277 8.360 10.443 TRUE
PS2 ME2 6.478 8.560 10.643 TRUE
PS2 BF1 6.402 8.485 10.568 TRUE
PS2 BF2 6.463 8.545 10.627 TRUE
PS2 ME1 6.772 8.855 10.938 TRUE
PS1 BSWH2 -1.782 0.300 2.382 FALSE
PS1 CL2 -1.073 1.010 3.092 FALSE
PS1 LR -1.048 1.035 3.118 FALSE
PS1 ANO -0.573 1.510 3.592 FALSE
PS1 GT -0.927 1.155 3.237 FALSE
PS1 BSWH1 -0.343 1.740 3.822 FALSE
PS1 SH2 0.013 2.095 4.178 TRUE
PS1 SH1 0.522 2.605 4.688 TRUE
PS1 CL1 4.548 6.630 8.713 TRUE
PS1 ME2 4.747 6.830 8.912 TRUE
PS1 BF1 4.673 6.755 8.838 TRUE
PS1 BF2 4.732 6.815 8.898 TRUE
PS1 ME1 5.043 7.125 9.207 TRUE
BSWH2 CL2 -1.373 0.710 2.792 FALSE
BSWH2 LR -1.347 0.735 2.817 FALSE
BSWH2 ANO -0.873 1.210 3.292 FALSE
BSWH2 GT -1.228 0.855 2.938 FALSE
BSWH2 BSWH1 -0.642 1.440 3.522 FALSE
BSWH2 SH2 -0.287 1.795 3.877 FALSE
BSWH2 SH1 0.223 2.305 4.388 TRUE
BSWH2 CL1 4.247 6.330 8.412 TRUE
BSWH2 ME2 4.447 6.530 8.613 TRUE
BSWH2 BF1 4.372 6.455 8.537 TRUE
BSWH2 BF2 4.433 6.515 8.598 TRUE
BSWH2 ME1 4.742 6.825 8.908 TRUE
CL2 LR -2.058 0.025 2.107 FALSE
CL2 ANO -1.583 0.500 2.583 FALSE
CL2 GT -1.938 0.145 2.228 FALSE
CL2 BSWH1 -1.353 0.730 2.812 FALSE
CL2 SH2 -0.998 1.085 3.167 FALSE
CL2 SH1 -0.487 1.595 3.678 FALSE
CL2 CL1 3.538 5.620 7.702 TRUE
CL2 ME2 3.737 5.820 7.902 TRUE
CL2 BF1 3.663 5.745 7.827 TRUE
CL2 BF2 3.723 5.805 7.888 TRUE
CL2 ME1 4.032 6.115 8.197 TRUE
LR ANO -1.607 0.475 2.558 FALSE
LR GT -1.962 0.120 2.203 FALSE
LR BSWH1 -1.377 0.705 2.788 FALSE
LR SH2 -1.022 1.060 3.143 FALSE
LR SH1 -0.512 1.570 3.652 FALSE
LR CL1 3.513 5.595 7.678 TRUE
LR ME2 3.712 5.795 7.878 TRUE
LR BF1 3.638 5.720 7.803 TRUE
LR BF2 3.697 5.780 7.862 TRUE
LR ME1 4.008 6.090 8.172 TRUE
ANO GT -2.438 -0.355 1.728 FALSE
ANO BSWH1 -1.853 0.230 2.312 FALSE
ANO SH2 -1.498 0.585 2.667 FALSE
ANO SH1 -0.988 1.095 3.178 FALSE
ANO CL1 3.038 5.120 7.202 TRUE
ANO ME2 3.237 5.320 7.402 TRUE
ANO BF1 3.163 5.245 7.327 TRUE
ANO BF2 3.223 5.305 7.388 TRUE
ANO ME1 3.533 5.615 7.697 TRUE
GT BSWH1 -1.498 0.585 2.667 FALSE
GT SH2 -1.143 0.940 3.022 FALSE
GT SH1 -0.632 1.450 3.533 FALSE
GT CL1 3.393 5.475 7.558 TRUE
GT ME2 3.592 5.675 7.758 TRUE
GT BF1 3.518 5.600 7.683 TRUE
GT BF2 3.578 5.660 7.742 TRUE
GT ME1 3.888 5.970 8.053 TRUE
BSWH1 SH2 -1.728 0.355 2.438 FALSE
BSWH1 SH1 -1.218 0.865 2.947 FALSE
BSWH1 CL1 2.808 4.890 6.973 TRUE
BSWH1 ME2 3.007 5.090 7.173 TRUE
BSWH1 BF1 2.933 5.015 7.098 TRUE
BSWH1 BF2 2.993 5.075 7.157 TRUE
BSWH1 ME1 3.303 5.385 7.468 TRUE
SH2 SH1 -1.573 0.510 2.592 FALSE
SH2 CL1 2.453 4.535 6.617 TRUE
SH2 ME2 2.652 4.735 6.817 TRUE
SH2 BF1 2.578 4.660 6.742 TRUE
SH2 BF2 2.638 4.720 6.803 TRUE
SH2 ME1 2.947 5.030 7.112 TRUE
SH1 CL1 1.942 4.025 6.107 TRUE
SH1 ME2 2.143 4.225 6.308 TRUE
SH1 BF1 2.067 4.150 6.232 TRUE
SH1 BF2 2.127 4.210 6.293 TRUE
SH1 ME1 2.438 4.520 6.603 TRUE
CL1 ME2 -1.883 0.200 2.283 FALSE
CL1 BF1 -1.958 0.125 2.208 FALSE
CL1 BF2 -1.897 0.185 2.268 FALSE
CL1 ME1 -1.587 0.495 2.578 FALSE
ME2 BF1 -2.158 -0.075 2.007 FALSE
ME2 BF2 -2.098 -0.015 2.067 FALSE
ME2 ME1 -1.788 0.295 2.377 FALSE
BF1 BF2 -2.022 0.060 2.143 FALSE
BF1 ME1 -1.712 0.370 2.453 FALSE
BF2 ME1 -1.772 0.310 2.393 FALSE

download these results as csv

https://music-ir.org/mirex/results/2009/ams/evalutron/small.evalutron.cat.friedman.tukeyKramerHSD.png


Summary Results by Query

FINE Scores

These are the mean FINE scores per query assigned by Evalutron graders. The FINE scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0.0 and 10.0. A perfect score would be 10. Genre labels have been included for reference.

Genre Query ANO BF1 BF2 BSWH1 BSWH2 CL1 CL2 GT LR ME1 ME2 PS1 PS2 SH1 SH2
BAROQUE d005166 3.060 2.120 1.900 2.520 3.700 3.280 5.560 3.560 2.620 0.540 1.860 3.040 3.700 3.960 2.040
BAROQUE d007244 5.260 0.540 2.620 6.060 5.540 4.400 5.940 5.900 6.140 3.500 1.840 4.880 7.100 5.180 6.360
BAROQUE d004490 7.880 3.940 6.680 3.400 6.060 5.340 8.520 7.500 8.300 5.160 3.440 7.800 9.200 7.080 8.280
BAROQUE d009737 7.620 5.180 0.420 4.540 8.940 5.860 6.080 2.480 8.180 1.320 1.060 8.820 8.840 3.440 2.780
BAROQUE d009837 4.140 1.520 0.360 1.700 1.640 2.860 1.000 3.100 3.860 0.700 0.720 3.960 4.680 2.320 3.520
BAROQUE d008174 5.000 3.880 2.260 7.680 7.600 7.140 7.460 7.480 7.020 3.320 1.860 7.700 7.680 5.080 7.580
BAROQUE d011054 7.320 4.160 0.000 4.360 6.140 5.500 5.280 5.460 5.980 0.940 1.000 5.980 7.160 5.780 5.560
BAROQUE d018083 6.940 1.380 2.540 5.600 3.040 2.540 5.940 3.400 6.240 0.460 0.260 3.660 3.580 3.440 4.020
BAROQUE d017335 4.680 0.280 0.960 1.940 4.640 2.340 3.440 5.780 3.380 2.540 0.000 3.560 4.240 3.880 1.980
BAROQUE d019774 5.500 3.600 4.660 4.060 4.000 3.860 4.400 5.560 4.060 3.440 5.160 4.500 6.300 4.960 4.820
BLUES e002545 5.760 6.260 1.320 5.600 7.140 0.280 6.760 6.140 5.160 1.600 4.180 7.340 8.060 4.760 5.200
BLUES e000111 3.200 5.900 4.300 3.800 3.200 1.980 7.400 5.500 3.000 2.900 7.000 6.000 6.660 7.300 6.800
BLUES e001417 6.700 3.320 2.920 4.760 4.980 7.480 8.800 7.660 7.000 6.480 4.940 7.160 6.980 5.020 7.160
BLUES e004211 2.880 2.840 2.680 2.720 5.880 6.840 6.520 4.280 5.380 0.180 2.020 7.840 8.460 6.640 0.120
BLUES e014267 5.400 2.200 0.960 7.040 4.700 0.000 4.880 5.400 2.780 1.380 3.220 4.460 6.640 4.440 2.280
BLUES e013973 8.480 0.700 4.140 6.680 8.100 0.480 9.060 8.560 8.400 4.300 3.560 8.520 8.520 8.660 8.140
BLUES e014486 3.800 3.080 2.280 4.280 5.180 0.160 1.760 5.380 3.160 3.420 1.300 5.920 4.360 4.660 5.000
BLUES e010067 4.160 2.180 2.520 5.640 7.320 4.620 6.620 5.480 5.100 7.020 2.960 4.680 7.960 3.880 4.460
BLUES e012895 7.760 3.360 1.580 5.880 7.560 6.720 7.740 6.460 7.840 2.280 1.740 8.360 8.540 7.700 6.060
BLUES e015354 5.400 2.700 0.000 4.000 5.600 0.000 6.400 4.800 4.800 0.800 2.200 1.200 6.900 5.600 6.800
CLASSICAL d000239 7.580 5.080 0.720 6.400 7.280 7.640 6.160 9.100 8.020 0.580 3.620 8.860 8.760 5.360 6.500
CLASSICAL d000762 6.100 2.040 2.520 7.200 6.840 5.260 6.660 6.660 6.940 0.800 0.720 8.300 8.520 5.400 5.800
CLASSICAL d002538 7.200 0.200 3.800 6.600 7.000 4.200 6.400 6.800 7.000 2.400 3.800 6.800 7.200 6.600 6.800
CLASSICAL d001502 8.700 4.300 7.560 8.600 8.900 8.400 8.820 8.840 9.100 6.740 5.800 8.920 8.800 9.100 8.660
CLASSICAL d012972 7.960 3.960 2.080 8.100 7.860 6.860 7.640 8.820 7.200 3.860 4.640 8.420 7.940 7.660 7.580
CLASSICAL d010713 5.000 3.820 2.500 5.100 5.200 4.200 1.740 3.600 3.100 2.400 4.100 2.800 3.300 3.200 3.100
CLASSICAL d012985 8.680 4.200 1.920 3.120 9.080 5.900 8.040 6.460 8.740 0.420 5.520 8.760 8.920 7.720 8.240
CLASSICAL d019802 6.660 2.060 1.500 5.580 7.380 4.680 5.440 4.680 3.100 2.300 3.540 5.700 6.220 6.420 6.540
CLASSICAL d019790 6.600 5.200 6.000 6.600 7.200 4.800 6.200 5.400 6.200 1.800 3.400 7.000 6.600 6.600 6.800
CLASSICAL d019783 7.800 2.400 4.200 6.800 7.400 7.000 7.800 6.400 7.600 3.600 2.200 6.100 8.400 8.200 7.800
COUNTRY e002580 3.880 1.660 2.300 3.740 3.340 0.000 2.440 4.800 2.900 3.080 1.920 3.760 3.820 2.640 2.500
COUNTRY e003090 3.200 0.800 2.800 2.200 4.000 0.000 2.200 3.800 4.400 1.600 0.800 6.200 7.200 3.000 2.500
COUNTRY e001591 2.000 4.760 3.780 3.380 6.340 1.700 4.880 5.140 2.740 2.640 3.060 7.200 6.680 5.600 5.500
COUNTRY e006811 4.820 2.480 3.860 5.400 5.760 0.860 4.220 3.580 5.040 1.460 2.800 4.900 8.360 6.040 3.460
COUNTRY e003155 0.000 0.060 1.200 0.200 2.800 0.000 3.340 1.720 2.100 0.200 0.500 3.700 5.320 5.220 2.680
COUNTRY e003544 6.200 0.880 2.660 5.200 3.720 0.000 4.040 4.600 4.500 1.100 2.900 5.400 5.240 5.720 6.280
COUNTRY e014385 4.800 0.000 0.600 2.600 3.800 0.000 2.000 2.900 1.700 0.900 1.700 5.300 5.000 1.800 1.820
COUNTRY e011855 6.860 0.740 1.160 4.280 6.340 0.440 4.960 6.200 6.560 1.500 0.820 7.380 5.040 5.480 5.120
COUNTRY e015843 5.540 1.940 2.140 3.620 2.880 0.000 4.400 4.980 4.400 2.180 4.780 5.500 4.760 4.000 5.520
COUNTRY e015137 7.120 2.600 2.860 4.880 6.420 0.840 4.260 6.700 7.740 3.100 2.800 7.160 7.560 5.360 3.360
EDANCE a009068 2.400 1.940 3.840 4.800 5.280 2.320 3.760 3.940 2.680 1.940 2.360 3.520 4.640 3.540 4.520
EDANCE b001840 3.920 0.660 0.300 4.640 4.960 0.100 0.280 5.120 4.120 0.400 0.000 5.020 4.460 1.520 1.200
EDANCE b015999 7.780 2.000 1.700 6.640 6.480 0.000 6.340 8.340 8.980 0.000 0.680 7.680 8.280 7.400 7.300
EDANCE b015503 4.200 1.200 1.000 5.300 6.500 0.200 2.900 2.900 6.400 2.300 3.400 1.000 4.200 4.600 2.300
EDANCE b019464 5.060 2.900 0.760 5.620 4.520 0.000 0.200 4.400 2.680 4.760 5.780 2.520 6.100 4.420 1.480
EDANCE b019570 5.880 0.400 4.600 7.320 6.660 0.900 6.300 7.860 6.500 3.920 3.480 8.220 7.140 6.860 5.220
EDANCE f014939 1.880 0.820 0.860 1.220 0.660 0.260 1.360 1.880 1.980 1.480 1.200 1.560 3.020 1.880 3.020
EDANCE f008160 4.680 2.420 4.480 7.420 7.040 1.200 6.100 6.600 6.180 3.240 2.880 7.200 8.100 6.880 5.460
EDANCE f011114 3.400 3.060 4.000 5.320 5.560 1.360 4.520 2.180 2.080 3.480 4.760 1.580 5.620 2.620 2.500
EDANCE f003748 6.800 1.660 4.240 6.540 5.840 0.260 6.560 6.380 5.140 4.860 1.160 6.800 4.720 4.980 4.800
JAZZ a003703 1.000 3.260 0.180 2.540 1.120 0.000 3.820 0.680 3.620 0.580 0.120 3.060 2.200 0.600 1.020
JAZZ e002394 1.000 0.000 1.000 1.800 1.800 3.200 4.000 6.800 3.600 1.700 1.800 3.800 4.000 0.400 2.500
JAZZ e001113 6.980 2.120 3.480 7.640 8.360 3.520 7.400 4.560 8.220 2.320 2.940 7.780 7.120 7.100 5.180
JAZZ e004292 6.000 3.940 1.300 7.520 8.000 5.800 6.740 7.080 5.700 1.140 2.040 5.600 7.940 3.760 3.720
JAZZ e004662 2.780 0.860 2.160 1.060 3.880 0.800 4.640 2.660 4.300 1.280 0.640 3.580 6.120 1.660 2.680
JAZZ e004944 7.600 2.860 2.740 5.220 5.640 6.600 8.320 8.520 8.660 2.560 2.900 5.280 8.540 8.060 5.080
JAZZ e004070 6.440 4.540 1.440 3.980 4.900 1.480 4.960 5.680 5.760 4.120 5.140 5.180 6.380 4.520 3.440
JAZZ e012026 2.140 4.180 0.160 4.740 5.400 1.000 2.440 3.900 2.640 3.460 0.760 1.920 7.920 2.880 6.280
JAZZ e015744 7.540 2.120 1.560 7.860 8.120 3.440 7.300 7.900 8.140 5.360 4.900 7.700 9.080 6.560 7.180
JAZZ e015566 6.440 2.800 1.700 4.720 6.700 7.020 5.780 5.560 7.720 5.020 2.720 6.840 8.300 5.560 2.940
METAL a003208 7.440 1.620 5.800 6.280 7.220 1.100 7.880 6.980 7.000 3.880 4.260 7.840 5.400 7.820 8.120
METAL b006262 5.920 3.500 3.360 6.520 4.060 0.000 5.420 2.380 3.240 1.260 1.740 5.800 6.920 3.280 5.340
METAL b007445 7.240 3.680 4.500 4.760 6.600 6.580 7.020 7.380 7.820 2.760 4.560 7.140 7.920 7.380 6.260
METAL b010346 6.600 1.780 2.340 5.980 7.060 2.520 6.820 6.160 7.260 3.800 4.800 6.720 6.940 7.320 6.960
METAL b014327 1.760 0.920 3.860 6.000 3.400 0.620 5.960 4.240 2.800 1.560 2.260 5.200 5.880 4.820 5.200
METAL b017546 4.300 2.340 3.120 6.140 5.400 0.960 5.980 4.480 5.920 1.020 0.980 6.420 7.880 5.100 3.680
METAL b019571 6.760 2.000 3.720 5.460 7.540 1.380 6.300 7.600 7.380 3.160 2.200 7.700 7.000 7.080 6.080
METAL f002408 6.220 0.140 5.320 5.480 6.260 1.560 7.120 5.880 6.940 2.960 8.120 8.260 7.540 5.860 6.200
METAL f000530 6.440 1.520 2.020 6.240 7.300 0.000 8.520 7.520 6.860 2.000 4.900 6.780 7.980 7.440 7.520
METAL f005072 5.640 3.420 4.360 5.060 4.800 2.700 5.140 6.640 5.540 1.920 3.740 4.760 5.760 3.620 4.380
RAPHIPHOP a000293 7.480 3.540 2.080 7.780 8.580 0.480 7.540 7.160 5.320 2.580 3.660 8.120 9.020 2.160 5.760
RAPHIPHOP a000827 5.700 4.020 3.880 4.440 5.860 0.120 6.500 6.420 4.440 2.940 2.080 6.260 3.100 3.000 4.220
RAPHIPHOP a000897 5.940 3.560 4.120 6.020 6.580 0.400 5.500 6.740 6.760 2.000 2.160 6.480 6.860 6.220 6.400
RAPHIPHOP a002740 6.560 2.540 3.160 6.040 7.180 0.940 5.640 6.080 6.600 2.500 2.280 7.000 6.640 5.240 6.720
RAPHIPHOP a009059 6.200 1.920 2.540 6.520 6.640 0.020 7.000 2.200 5.040 2.900 2.320 2.280 7.760 3.680 2.460
RAPHIPHOP b010144 6.380 1.820 2.100 7.120 7.500 0.080 8.180 7.280 6.700 3.520 1.860 6.700 8.920 7.820 7.880
RAPHIPHOP b011546 5.680 3.040 0.860 5.960 7.800 0.000 6.480 7.600 5.800 2.680 1.980 7.320 5.640 5.520 6.800
RAPHIPHOP b010454 7.820 1.680 5.060 8.300 8.340 0.000 7.600 7.100 8.900 2.320 3.160 8.140 8.220 6.880 8.300
RAPHIPHOP b017461 3.660 1.780 4.280 7.500 7.100 0.180 7.840 6.900 5.780 4.220 4.100 7.320 7.700 6.600 7.160
RAPHIPHOP b018747 7.300 0.920 0.640 8.280 8.240 0.000 8.060 7.960 7.040 1.920 0.980 7.820 8.240 7.260 7.200
ROCKROLL b001069 6.000 2.760 5.260 5.780 3.600 4.080 6.520 3.520 6.700 1.540 3.820 6.220 7.420 5.860 6.800
ROCKROLL b001751 3.640 1.300 0.800 2.540 1.540 0.940 0.980 1.780 3.120 1.380 2.100 3.660 2.860 3.760 4.100
ROCKROLL b003625 1.600 2.100 3.140 5.340 3.080 0.800 3.040 4.040 2.280 1.160 1.300 4.600 4.220 5.280 5.500
ROCKROLL b004162 5.040 3.180 3.380 3.700 4.760 1.740 5.840 4.260 5.020 2.340 2.200 4.660 5.540 5.420 4.560
ROCKROLL b004353 6.600 1.800 3.600 7.000 6.600 1.000 5.400 4.400 5.400 4.000 2.800 4.200 8.300 6.200 5.000
ROCKROLL b006456 5.400 1.760 8.240 7.800 5.740 0.480 6.880 6.000 6.720 1.660 3.200 8.060 6.020 7.980 6.540
ROCKROLL b009365 5.960 1.020 4.220 4.840 7.940 0.200 4.220 3.920 6.160 2.940 3.180 6.860 7.440 7.600 5.200
ROCKROLL b008878 5.480 2.400 3.160 5.820 4.740 0.840 4.300 5.860 4.940 2.720 2.360 4.220 4.040 3.140 1.700
ROCKROLL b011553 1.740 0.960 1.540 2.400 2.660 0.140 3.020 3.240 3.280 0.120 0.800 1.800 3.560 2.220 1.460
ROCKROLL e017211 2.080 1.580 1.640 1.460 2.820 1.340 1.980 1.700 2.120 1.580 1.340 2.000 2.020 2.140 1.980
ROMANTIC d004429 8.860 6.340 3.840 7.860 8.800 7.920 7.140 8.400 8.820 5.140 3.540 8.700 9.180 8.560 8.860
ROMANTIC d001688 4.020 0.600 1.260 2.160 3.420 2.440 2.660 3.080 3.780 0.060 0.540 3.780 3.100 2.100 2.460
ROMANTIC d004908 5.640 2.220 0.360 8.400 7.060 5.720 5.880 5.940 5.220 1.720 0.740 6.280 7.500 4.800 6.000
ROMANTIC d007929 4.700 1.500 2.060 6.860 7.700 7.180 5.080 3.340 6.740 0.420 2.120 6.480 6.800 4.720 4.640
ROMANTIC d011624 6.400 0.000 1.800 3.900 6.700 3.500 5.800 3.700 6.100 0.400 0.800 7.100 7.480 0.200 3.200
ROMANTIC d016855 4.700 3.900 0.000 3.600 5.800 4.100 4.900 4.300 4.600 0.400 0.900 4.400 4.400 4.400 4.900
ROMANTIC d014946 7.140 2.200 1.800 6.400 6.300 7.060 2.000 5.040 7.360 0.480 2.880 5.880 7.460 5.900 4.720
ROMANTIC d014940 5.680 2.920 0.280 6.460 7.160 4.300 6.160 5.860 7.680 1.460 0.880 6.400 7.380 3.220 4.900
ROMANTIC d018672 2.100 0.200 1.080 0.580 2.300 1.660 1.720 2.540 2.240 0.320 0.000 2.860 2.440 2.320 1.640
ROMANTIC d017019 3.680 3.520 1.320 1.180 3.320 3.540 4.260 4.720 2.960 2.500 1.440 3.520 4.240 3.560 3.020

download these results as csv

BROAD Scores

These are the mean BROAD scores per query assigned by Evalutron graders. The BROAD scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 (not similar) and 2 (very similar). A perfect score would be 2. Genre labels have been included for reference.

Genre Query ANO BF1 BF2 BSWH1 BSWH2 CL1 CL2 GT LR ME1 ME2 PS1 PS2 SH1 SH2
BAROQUE d005166 1.000 0.600 0.600 1.000 1.000 0.800 1.600 1.000 1.000 0.200 0.600 1.000 1.200 1.000 0.600
BAROQUE d007244 1.200 0.000 0.400 1.400 1.200 0.800 1.400 1.600 1.400 0.800 0.200 0.800 1.800 1.200 1.600
BAROQUE d004490 2.000 0.800 1.600 0.800 1.600 1.200 2.000 1.800 2.000 1.200 0.800 1.800 2.000 1.800 2.000
BAROQUE d009737 2.000 1.200 0.000 0.800 2.000 1.600 1.200 0.400 1.800 0.000 0.000 2.000 2.000 0.800 0.400
BAROQUE d009837 1.200 0.400 0.000 0.400 0.400 0.800 0.200 0.800 1.200 0.200 0.000 1.000 1.400 0.600 0.800
BAROQUE d008174 1.200 0.800 0.400 2.000 2.000 1.800 2.000 2.000 1.600 0.600 0.200 2.000 2.000 1.200 2.000
BAROQUE d011054 1.800 1.200 0.000 1.000 1.400 1.400 1.000 1.400 1.400 0.200 0.200 1.600 1.600 1.200 1.200
BAROQUE d018083 2.000 0.400 0.600 1.600 0.600 0.600 1.400 1.000 1.800 0.000 0.000 1.000 1.000 0.600 1.000
BAROQUE d017335 0.400 0.000 0.000 0.000 0.600 0.000 0.400 0.600 0.200 0.200 0.000 0.000 0.400 0.400 0.200
BAROQUE d019774 1.000 0.200 0.800 1.000 1.400 1.200 1.200 1.600 1.400 0.200 1.200 2.000 2.000 1.200 1.800
BLUES e002545 1.400 1.200 0.200 1.400 1.400 0.000 1.600 1.400 1.200 0.200 0.800 1.800 2.000 1.000 1.400
BLUES e000111 0.400 1.200 0.800 0.600 0.400 0.000 1.600 1.200 0.400 0.600 1.400 1.000 1.200 1.400 1.400
BLUES e001417 1.800 0.600 0.400 1.000 1.000 1.800 2.000 2.000 1.600 1.400 0.800 1.600 1.800 1.000 1.600
BLUES e004211 0.400 0.600 0.600 0.600 1.400 1.600 1.600 0.600 0.800 0.000 0.400 1.800 2.000 1.600 0.000
BLUES e014267 1.200 0.600 0.200 1.400 0.800 0.000 1.200 1.200 0.600 0.200 0.600 1.000 1.400 0.800 0.400
BLUES e013973 1.600 0.200 0.600 1.400 1.800 0.000 2.000 2.000 1.600 0.800 0.800 1.800 1.800 2.000 1.800
BLUES e014486 0.800 0.600 0.200 1.200 1.200 0.000 0.400 1.200 0.600 0.800 0.400 1.200 1.200 1.000 1.200
BLUES e010067 0.400 0.000 0.200 1.200 1.600 0.800 1.400 0.800 0.800 1.600 0.400 0.800 2.000 0.400 0.600
BLUES e012895 1.600 0.800 0.200 1.000 1.600 1.600 1.600 1.600 1.800 0.400 0.200 1.800 2.000 1.400 1.200
BLUES e015354 1.200 0.600 0.000 1.000 1.400 0.000 1.400 1.200 1.400 0.200 0.400 0.200 1.600 1.400 2.000
CLASSICAL d000239 1.600 1.000 0.000 1.400 1.600 1.600 1.200 2.000 1.800 0.000 0.800 2.000 2.000 1.000 1.200
CLASSICAL d000762 1.400 0.400 0.400 1.600 1.400 1.000 1.400 1.400 1.600 0.000 0.000 2.000 2.000 1.000 1.000
CLASSICAL d002538 1.200 0.000 0.600 1.000 1.200 0.800 1.200 1.200 1.200 0.400 0.800 1.200 1.400 1.000 1.400
CLASSICAL d001502 2.000 0.800 1.600 2.000 2.000 1.800 2.000 2.000 2.000 1.200 1.000 2.000 2.000 2.000 2.000
CLASSICAL d012972 1.800 0.800 0.000 1.800 2.000 1.600 1.800 2.000 1.600 0.400 0.800 2.000 2.000 1.800 1.800
CLASSICAL d010713 1.200 0.600 0.000 1.400 1.600 1.000 1.400 1.800 1.200 0.200 0.000 1.800 1.600 0.800 1.000
CLASSICAL d012985 2.000 0.800 0.400 0.600 1.800 1.200 1.800 1.200 2.000 0.000 1.000 2.000 1.800 1.400 2.000
CLASSICAL d019802 1.200 0.400 0.200 0.800 1.600 0.600 1.000 0.800 0.400 0.400 0.400 1.200 1.000 1.200 1.400
CLASSICAL d019790 1.000 1.000 1.200 1.400 1.400 0.800 1.000 1.000 1.000 0.200 0.600 1.200 1.400 1.000 1.400
CLASSICAL d019783 1.800 0.200 0.600 1.400 1.800 1.600 1.800 1.200 1.800 0.600 0.000 1.200 2.000 2.000 1.800
COUNTRY e002580 1.000 0.200 0.400 0.800 0.600 0.000 0.400 1.000 0.400 0.600 0.200 0.800 1.000 0.400 0.400
COUNTRY e003090 0.600 0.200 0.600 0.400 1.000 0.000 0.800 0.800 1.000 0.400 0.200 1.000 1.600 0.800 0.800
COUNTRY e001591 0.000 0.800 0.600 0.200 1.400 0.000 1.000 1.000 0.200 0.600 0.400 1.600 1.600 1.200 1.200
COUNTRY e006811 1.200 0.400 0.600 1.200 1.200 0.000 0.800 0.600 1.200 0.200 0.400 1.000 2.000 1.400 0.400
COUNTRY e003155 0.000 0.000 0.200 0.000 0.600 0.000 0.800 0.400 0.400 0.000 0.000 0.800 1.200 1.200 0.600
COUNTRY e003544 1.400 0.200 0.400 1.200 0.600 0.000 1.000 0.800 0.800 0.000 0.400 1.400 1.200 1.400 1.400
COUNTRY e014385 1.000 0.000 0.000 0.600 1.000 0.000 0.400 0.400 0.200 0.200 0.200 1.200 1.200 0.200 0.400
COUNTRY e011855 1.600 0.000 0.000 0.600 1.800 0.000 1.000 1.000 1.400 0.200 0.000 2.000 1.200 1.400 1.200
COUNTRY e015843 1.600 0.200 0.400 1.000 0.800 0.000 1.000 1.200 1.000 0.400 1.200 1.600 1.200 0.800 1.200
COUNTRY e015137 1.600 0.600 0.600 1.000 1.200 0.200 0.600 1.600 1.600 0.600 0.400 1.600 1.800 1.200 0.800
EDANCE a009068 0.000 0.200 0.600 1.000 1.000 0.200 0.600 0.800 0.200 0.200 0.200 0.600 0.800 0.600 0.800
EDANCE b001840 0.400 0.000 0.000 0.600 0.600 0.000 0.000 0.800 0.800 0.000 0.000 0.800 0.600 0.000 0.000
EDANCE b015999 1.800 0.400 0.200 1.400 1.400 0.000 1.200 1.800 2.000 0.000 0.000 1.800 1.800 1.600 1.600
EDANCE b015503 0.600 0.200 0.000 1.000 1.200 0.000 0.400 0.400 1.000 0.200 0.400 0.000 0.400 0.400 0.200
EDANCE b019464 1.200 0.600 0.000 1.600 1.000 0.000 0.000 1.000 0.600 0.800 1.400 0.600 1.600 1.000 0.200
EDANCE b019570 1.200 0.000 0.600 1.600 1.400 0.000 1.200 1.800 1.000 0.600 0.600 2.000 1.400 1.400 1.200
EDANCE f014939 0.200 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.200 0.000 0.000 0.000 0.400 0.000 0.600
EDANCE f008160 1.000 0.200 1.000 1.800 1.600 0.000 1.400 1.600 1.400 0.400 0.600 1.800 2.000 1.600 1.400
EDANCE f011114 0.600 0.400 0.600 1.000 1.200 0.000 1.000 0.200 0.000 0.400 0.800 0.000 1.000 0.400 0.400
EDANCE f003748 1.600 0.200 0.800 1.600 1.200 0.000 1.200 1.200 1.200 1.000 0.200 1.400 1.000 1.000 1.000
JAZZ a003703 0.200 0.600 0.000 0.600 0.200 0.000 1.000 0.200 0.600 0.200 0.000 0.800 0.400 0.000 0.200
JAZZ e002394 0.200 0.000 0.200 0.400 0.200 0.800 1.000 1.400 0.600 0.200 0.400 0.800 1.000 0.000 0.600
JAZZ e001113 1.400 0.400 0.800 1.800 1.800 0.800 1.600 1.000 1.800 0.400 0.400 1.600 1.600 1.600 1.000
JAZZ e004292 1.000 0.800 0.000 1.600 1.600 1.200 1.600 1.400 1.000 0.000 0.200 1.000 1.600 0.400 0.800
JAZZ e004662 0.400 0.000 0.000 0.000 0.600 0.000 0.600 0.000 0.600 0.000 0.000 0.400 1.200 0.000 0.200
JAZZ e004944 2.000 0.400 0.400 1.200 1.200 1.600 2.000 2.000 2.000 0.200 0.600 1.200 2.000 2.000 1.000
JAZZ e004070 1.400 1.000 0.000 0.600 1.000 0.200 1.000 1.000 1.200 0.600 0.800 1.000 1.400 0.800 0.600
JAZZ e012026 0.400 0.800 0.000 1.000 1.000 0.200 0.600 0.600 0.600 0.600 0.200 0.400 2.000 0.400 1.600
JAZZ e015744 1.400 0.200 0.200 1.800 1.800 0.400 1.200 1.600 1.800 1.000 0.800 1.800 2.000 1.200 1.200
JAZZ e015566 1.400 0.600 0.400 1.000 1.600 1.600 1.200 1.000 1.600 1.200 0.400 1.600 2.000 1.000 0.400
METAL a003208 1.400 0.000 0.800 1.000 1.200 0.000 1.800 1.400 1.200 0.400 0.400 1.600 0.800 1.600 1.800
METAL b006262 1.200 0.800 1.000 1.400 0.600 0.000 1.400 0.400 0.800 0.200 0.400 1.600 1.600 0.800 1.000
METAL b007445 2.000 0.400 0.800 0.600 1.800 1.400 1.800 2.000 2.000 0.200 0.800 2.000 2.000 1.600 1.400
METAL b010346 1.600 0.000 0.000 1.400 1.800 0.200 1.400 1.400 1.800 0.400 1.000 1.600 1.800 1.800 1.400
METAL b014327 0.000 0.000 0.400 0.800 0.200 0.000 0.800 0.600 0.200 0.000 0.200 0.600 1.000 0.600 0.800
METAL b017546 0.800 0.400 0.400 1.400 1.200 0.000 1.200 0.800 1.400 0.000 0.000 1.600 1.800 1.000 0.600
METAL b019571 1.600 0.200 1.000 1.200 1.800 0.200 1.400 1.600 1.400 0.400 0.400 1.600 1.600 1.400 1.400
METAL f002408 1.000 0.000 1.200 1.200 1.000 0.400 1.600 1.000 1.400 0.600 1.800 1.800 1.400 1.200 1.400
METAL f000530 1.000 0.200 0.200 1.200 1.400 0.000 1.600 1.400 1.200 0.400 0.800 1.200 1.600 1.600 1.600
METAL f005072 0.600 0.400 0.600 0.800 0.600 0.200 0.800 1.200 0.600 0.000 0.400 0.400 1.000 0.400 0.200
RAPHIPHOP a000293 1.600 0.600 0.200 1.800 2.000 0.000 1.800 1.600 1.000 0.600 0.600 2.000 2.000 0.200 1.000
RAPHIPHOP a000827 1.200 0.800 0.600 1.000 1.200 0.000 1.400 1.600 1.000 0.600 0.400 1.400 0.600 0.200 0.800
RAPHIPHOP a000897 1.600 0.600 1.000 1.400 1.800 0.000 1.400 2.000 2.000 0.200 0.200 1.800 2.000 1.600 1.400
RAPHIPHOP a002740 1.400 0.400 0.400 1.200 1.600 0.000 1.400 1.200 1.400 0.200 0.200 1.600 1.600 1.200 1.600
RAPHIPHOP a009059 1.200 0.400 0.200 1.000 1.400 0.000 1.600 0.000 1.000 0.200 0.400 0.200 1.600 0.600 0.400
RAPHIPHOP b010144 1.200 0.400 0.200 1.400 1.800 0.000 2.000 1.800 1.400 1.000 0.200 1.600 2.000 1.800 1.800
RAPHIPHOP b011546 1.400 0.600 0.000 1.400 1.800 0.000 1.600 1.600 1.600 0.600 0.400 2.000 1.400 1.200 1.800
RAPHIPHOP b010454 1.600 0.400 1.200 2.000 1.800 0.000 1.800 1.400 2.000 0.200 0.600 1.800 1.800 1.600 2.000
RAPHIPHOP b017461 0.800 0.400 1.000 2.000 1.800 0.000 1.800 1.800 1.400 1.000 1.000 1.800 2.000 1.600 2.000
RAPHIPHOP b018747 1.800 0.200 0.000 2.000 2.000 0.000 2.000 2.000 2.000 0.400 0.000 2.000 2.000 1.800 1.800
ROCKROLL b001069 1.000 0.400 0.800 1.000 0.600 0.600 1.400 0.600 1.400 0.200 0.600 1.200 1.600 1.000 1.400
ROCKROLL b001751 1.000 0.600 0.000 0.600 0.200 0.000 0.200 0.600 0.800 0.400 0.800 0.800 0.800 0.800 1.000
ROCKROLL b003625 0.200 0.200 0.600 1.000 0.600 0.000 0.600 0.600 0.400 0.000 0.000 0.800 1.000 1.200 1.200
ROCKROLL b004162 1.000 0.200 0.200 0.800 1.000 0.000 1.000 1.000 0.800 0.200 0.200 1.000 1.000 0.800 0.800
ROCKROLL b004353 1.600 0.000 0.000 1.800 1.600 0.000 1.000 0.400 0.800 0.200 0.000 0.200 2.000 1.200 0.600
ROCKROLL b006456 1.400 0.200 2.000 1.600 1.200 0.000 1.600 1.400 1.600 0.000 0.600 2.000 1.400 2.000 1.600
ROCKROLL b009365 1.000 0.200 1.000 1.000 2.000 0.000 0.600 0.600 1.000 0.400 0.400 1.400 1.600 1.600 1.000
ROCKROLL b008878 1.400 0.200 0.400 1.400 1.000 0.000 0.800 1.400 1.000 0.400 0.200 1.000 0.800 0.400 0.200
ROCKROLL b011553 0.400 0.400 0.400 0.600 0.800 0.000 0.800 0.800 0.800 0.000 0.200 0.400 0.800 0.600 0.200
ROCKROLL e017211 0.000 0.000 0.000 0.000 0.200 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
ROMANTIC d004429 2.000 1.400 0.600 1.800 2.000 1.800 1.600 2.000 2.000 1.000 0.600 2.000 2.000 2.000 2.000
ROMANTIC d001688 0.600 0.000 0.000 0.200 0.400 0.200 0.400 0.200 0.600 0.000 0.000 0.600 0.200 0.000 0.000
ROMANTIC d004908 1.200 0.400 0.000 2.000 1.600 1.200 1.400 1.400 1.200 0.400 0.200 1.400 1.600 0.800 1.200
ROMANTIC d007929 1.000 0.400 0.400 1.600 1.400 1.600 1.000 1.000 1.600 0.000 0.600 1.400 1.600 1.200 1.000
ROMANTIC d011624 1.600 0.000 0.400 1.000 1.800 0.800 1.200 1.000 1.600 0.000 0.000 1.600 1.800 0.000 0.600
ROMANTIC d016855 0.800 0.800 0.000 1.000 1.400 0.800 1.200 1.000 1.000 0.000 0.200 0.800 1.000 0.800 1.000
ROMANTIC d014946 1.400 0.400 0.400 1.400 1.400 1.600 0.400 1.000 1.600 0.000 0.600 1.400 1.600 1.400 1.200
ROMANTIC d014940 1.200 0.400 0.000 1.600 1.600 0.800 1.600 1.200 1.800 0.200 0.200 1.800 1.800 0.600 1.000
ROMANTIC d018672 0.200 0.000 0.200 0.000 0.200 0.000 0.000 0.200 0.200 0.000 0.000 0.400 0.200 0.400 0.000
ROMANTIC d017019 0.400 0.600 0.000 0.000 0.400 0.600 0.600 0.800 0.200 0.200 0.000 0.400 0.600 0.000 0.200

download these results as csv


Raw Scores

The raw data derived from the Evalutron 6000 human evaluations are located on the 2009:Audio Music Similarity and Retrieval Raw Data page.

Metadata and Distance Space Evaluation

The following reports provide evaluation statistics based on analysis of the distance space and metadata matches and include:

  • Neighbourhood clustering by artist, album and genre
  • Artist-filtered genre clustering
  • How often the triangular inequality holds
  • Statistics on 'hubs' (tracks similar to many tracks) and orphans (tracks that are not similar to any other tracks at N results).

Reports

ANO = Anonymous
BF1 = Benjamin Fields (chr12)
BF2 = Benjamin Fields (mfcc10)
BSWH1 = Dmitry Bogdanov, Joan Serrà, Nicolas Wack, and Perfecto Herrera (clas)
BSWH2 = Dmitry Bogdanov, Joan Serrà, Nicolas Wack, and Perfecto Herrera (hybrid)
CL1 = Chuan Cao, Ming Li
CL2 = Chuan Cao, Ming Li
GT = George Tzanetakis
LR = Thomas Lidy, Andreas Rauber]
ME1 = François Maillet, Douglas Eck (mlp)
ME2 = François Maillet, Douglas Eck (sda)
PS1 = Tim Pohle, Dominik Schnitzer (2007)
PS2 = Tim Pohle, Dominik Schnitzer (2009)
SH1 = Stephan Hübler
SH2 = Stephan Hübler

Run Times

Participant Machine Runtime (dd:hh:mm)
ANO ALE Feat/Dist 00:00:22/00:00:04
BF1 ALE 10:00:00
BF2 ALE 10:00:00
BSWH1 ALE Feat/Dist 00:20:06/00:00:22
BSWH2 ALE Feat/Dist 00:20:06/00:01:12
CL1 FAST3 Lost - HD Failure
CL2 FAST3 Lost - HD Failure
GT ALE Feat/Dist 00:00:23/00:00:20
LR ALE Feat/Dist 00:05:09/00:02:05
ME1 ALE Feat/Dist 00:16:10/00:12:37
ME2 ALE Feat/Dist 00:16:08/02:23:27
PS1 ALE Feat/Dist 00:00:42/00:03:51
PS2 ALE Feat/Dist 00:03:40/00:05:56
SH1 ALE 00:04:06
SH2 ALE 00:05:50

download these results as csv

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2009:Audio_Music_Similarity_and_Retrieval_Results&oldid=7365"