2006:Audio Music Similarity and Retrieval Results - Revision history

IMIRSELBot: Robot: Automated text replacement (-http://www.music-ir.org/evaluation/MIREX/2006_abstracts/ +http://www.music-ir.org/mirex/abstracts/2006/)

2010-05-14T01:32:00Z

Robot: Automated text replacement (-http://www.music-ir.org/evaluation/MIREX/2006_abstracts/ +http://www.music-ir.org/mirex/abstracts/2006/)

IMIRSELBot: Robot: Automated text replacement (-]*)> +2006/)

2010-05-14T00:33:30Z

Robot: Automated text replacement (-<csv([^>]*)> +<csv\1>2006/)

IMIRSELBot: Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

2010-05-13T18:11:47Z

Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

IMIRSELBot: Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

2010-05-13T18:03:45Z

Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

Kriswest: /* A statistic for evaluation and use in selection & optimization experiments */

2006-10-10T18:48:53Z

‎A statistic for evaluation and use in selection & optimization experiments

Kriswest: /* A statistic for evaluation and use in selection & optimization experiments */

2006-10-10T18:48:05Z

‎A statistic for evaluation and use in selection & optimization experiments

Kriswest at 18:45, 10 October 2006

2006-10-10T18:45:44Z

Kriswest: /* Results from Automatic Evaluation */

2006-10-10T18:39:37Z

‎Results from Automatic Evaluation

Kriswest: /* Results from Automatic Evaluation */

2006-10-10T18:37:49Z

‎Results from Automatic Evaluation

Kriswest: /* Results from Automatic Evaluation */

2006-10-10T17:38:21Z

‎Results from Automatic Evaluation

@@ Line 14: / Line 14: @@
 ====General Legend====
 =====Team ID=====
-'''EP''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/AS_pampalk.pdf Elias Pampalk]<br />
+'''EP''' = [https://www.music-ir.org/mirex/abstracts/2006/AS_pampalk.pdf Elias Pampalk]<br />
-'''TP''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/AS_pohle.pdf Tim Pohle]<br />
+'''TP''' = [https://www.music-ir.org/mirex/abstracts/2006/AS_pohle.pdf Tim Pohle]<br />
 '''VS''' = Vitor Soares<br />
-'''LR''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/AS_lidy.pdf Thomas Lidy and Andreas Rauber]<br />
+'''LR''' = [https://www.music-ir.org/mirex/abstracts/2006/AS_lidy.pdf Thomas Lidy and Andreas Rauber]<br />
 '''KWT''' = Kris West (Trans)<br />
 '''KWL''' = Kris West (Likely)<br />

@@ Line 38: / Line 38: @@
 ===Overall Summary Results===
-<csv>mirex06_as_overalllist.csv</csv>
+<csv>2006/mirex06_as_overalllist.csv</csv>
 http://staff.aist.go.jp/elias.pampalk/papers/mirex06/friedman.png
@@ Line 46: / Line 46: @@
 ====Audio Music Similarity and Retrieval Runtime Data====
-<csv>as06_runtime.csv</csv>
+<csv>2006/as06_runtime.csv</csv>
 For a description of the computers the submission ran on see [[2006:MIREX_2006_Equipment]].
@@ Line 53: / Line 53: @@
 The Friedman test was run in MATLAB against the Fine summary data over the 60 queries.<br />
 Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
-<csv>AV_sum_friedman.csv</csv>
+<csv>2006/AV_sum_friedman.csv</csv>
-<csv>AV_fine_result.csv</csv>
+<csv>2006/AV_fine_result.csv</csv>
 ===Summary Results by Query===
-<csv>mirex06_as_uberlist.csv</csv>
+<csv>2006/mirex06_as_uberlist.csv</csv>
 ===Raw Scores===
@@ Line 63: / Line 63: @@
 ===Query Meta Data===
-<csv>as06_queries.csv</csv>
+<csv>2006/as06_queries.csv</csv>
@@ Line 69: / Line 69: @@
 ==Results from Automatic Evaluation==
-<csv>as06_nonhuman_results.csv</csv>
+<csv>2006/as06_nonhuman_results.csv</csv>

← Older revision		Revision as of 18:11, 13 May 2010
Line 73:		Line 73:

	=== Other Results from Automatic Evaluation===		=== Other Results from Automatic Evaluation===
−	See [[Audio Music Similarity and Retrieval Other Automatic Evaluation Results]] page.	+	See [[2006:Audio Music Similarity and Retrieval Other Automatic Evaluation Results]] page.

@@ Line 1: / Line 1: @@
 [[Category: Results]]
 ==Introduction==
-These are the results for the 2006 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the [[Audio Music Similarity and Retrieval]] page.
+These are the results for the 2006 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the [[2006:Audio Music Similarity and Retrieval]] page.
 Each system was given 5000 songs chosen from "uspop", "uscrap" and "cover song" collections. Each system then returned a 5000x5000 distance matrix. 60 songs were randomly selected as queries and the first 5 most highly ranked songs out of the 5000 were extracted for each query (after filtering out the query itself, returned results from the same artist and members of the cover song collection). Then, for each query, the returned results from all participants were grouped and were evaluated by human graders, each query being evaluated by 3 different graders with two scores (using the Evalutron 6000 system). Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 10). An automated statistical evaluation based on a metadata catalog was also conducted. A desciption and analysis is provided below.
@@ Line 48: / Line 48: @@
 <csv>as06_runtime.csv</csv>
-For a description of the computers the submission ran on see [[MIREX_2006_Equipment]].
+For a description of the computers the submission ran on see [[2006:MIREX_2006_Equipment]].
 ===Friedman Test with Multiple Comparisons Results (p=0.05)===
@@ Line 60: / Line 60: @@
 ===Raw Scores===
-The raw data derived from the Evalutron 6000 human evaluations are located on the [[Audio Music Similarity and Retrieval Raw Data]] page.
+The raw data derived from the Evalutron 6000 human evaluations are located on the [[2006:Audio Music Similarity and Retrieval Raw Data]] page.
 ===Query Meta Data===

@@ Line 162: / Line 162: @@
 As each statitic was found to be correlated with the results of the listening test, any *may* be used   to evaluate performance and to guide model optimisation or feature selection/weighting experiments. However, unfiltered genre and artist identification statistics are known to allow overfitting to produce over-optimistic performance estimates. In a model optimisation or feature selection experiment these statistics will be more likely to indicate '''Audio-similarity''' performance rather than actual '''Music-similarity''' performance and may lead to the selection of sub-optimal features or models. The artist-filtered genre neighbourhood can be used to avoid this effect.
-The results from MIREX 2006 do not show a significant drop in performance using the artist-filtered genre statistic as would normally be expected. This may be due to the excessively skewed distribution of examples in the database (roughly 50% of edxamples are labelled as Rock/Pop, while a further 25% are Rap & Hip-Hop). Hence, the difference between the results produced and the random baseline are not significantly highly emphasized. Normalising this statistic by the prior probabilities of examples in the database (taking the mean of the diagonal of the artist-filtered genre confusion matrix) equally weights the contribution of each class to the final statistic and prevents performance on a single class dominating the statistic. This normalised statistic shows a drastic reduction in the performance estimates for each system and increases the relative distance between each of the systems in the evaluation.
+The results from MIREX 2006 do not show a significant drop in performance using the artist-filtered genre statistic as would normally be expected. This may be due to the excessively skewed distribution of examples in the database (roughly 50% of examples are labelled as Rock/Pop, while a further 25% are Rap & Hip-Hop). Hence, the difference between the results produced and the random baseline are not well emphasized. Normalising this statistic by the prior probabilities of examples in the database (taking the mean of the diagonal of the artist-filtered genre confusion matrix) equally weights the contribution of each class to the final statistic and prevents performance on a single class dominating the statistic. This normalised statistic shows a drastic reduction in the performance estimates for each system and increases the relative distance between each of the systems in the evaluation.
 [[User:Kriswest|Kriswest]]

@@ Line 3: / Line 3: @@
 These are the results for the 2006 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the [[Audio Music Similarity and Retrieval]] page.
-Each system was given 5000 songs chosen from "uspop", "uscrap" and "cover song" collections. Each system then returned a 5000x5000 distance matrix. 60 songs were randomly selected as queries and the first 5 most highly ranked songs out of the 5000 were extracted for each query (after filtering out the query itself, returned results from the same artist and members of the cover song collection). Then, for each query, the returned results from all participants were grouped and were evaluated by human graders, each query being evaluated by 3 different graders with two scores (using the Evalutron 6000 system). Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 10).
+Each system was given 5000 songs chosen from "uspop", "uscrap" and "cover song" collections. Each system then returned a 5000x5000 distance matrix. 60 songs were randomly selected as queries and the first 5 most highly ranked songs out of the 5000 were extracted for each query (after filtering out the query itself, returned results from the same artist and members of the cover song collection). Then, for each query, the returned results from all participants were grouped and were evaluated by human graders, each query being evaluated by 3 different graders with two scores (using the Evalutron 6000 system). Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 10). An automated statistical evaluation based on a metadata catalog was also conducted. A desciption and analysis is provided below.
-====Summary Data on Human Evaluations (Evalutron 6000)====
+===Summary Data on Human Evaluations (Evalutron 6000)===
 '''Number of evaluators''' = 24<br />
 '''Number of evaluation per query/candidate pair''' = 3<br />
@@ Line 12: / Line 12: @@
 '''Number of randomly selected queries''' = 60<br />
-===General Legend===
+====General Legend====
-====Team ID====
+=====Team ID=====
 '''EP''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/AS_pampalk.pdf Elias Pampalk]<br />
 '''TP''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/AS_pohle.pdf Tim Pohle]<br />
@@ Line 21: / Line 21: @@
 '''KWL''' = Kris West (Likely)<br />
-====Broad Categories====
+=====Broad Categories====
 '''NS''' = Not Similar<br />
 '''SS''' = Somewhat Similar<br />
 '''VS''' = Very Similar<br />
-===Calculating Summary Measures===
+=====Calculating Summary Measures=====
 '''Fine'''<sup>(1)</sup> = Sum of fine-grained human similarity decisions (0-10). <br />
 '''PSum'''<sup>(1)</sup> = Sum of human broad similarity decisions: NS=0, SS=1, VS=2. <br />
@@ Line 36: / Line 36: @@
 <sup>(1)</sup>Normalized to the range 0 to 1.
-==Overall Summary Results==
+===Overall Summary Results===
 <csv>mirex06_as_overalllist.csv</csv>
@@ Line 44: / Line 44: @@
 This figure shows the official ranking of the submissions computed using a Friedman test. The blue lines indicate significance boundaries at the p=0.05 level. As can be seen, the differences are not significant. For a more detailed description and discussion see [http://staff.aist.go.jp/elias.pampalk/papers/pam_mirex06.pdf].
-===Audio Music Similarity and Retrieval Runtime Data===
+====Audio Music Similarity and Retrieval Runtime Data====
 <csv>as06_runtime.csv</csv>
@@ Line 50: / Line 50: @@
 For a description of the computers the submission ran on see [[MIREX_2006_Equipment]].
-==Friedman Test with Multiple Comparisons Results (p=0.05)==
+===Friedman Test with Multiple Comparisons Results (p=0.05)===
 The Friedman test was run in MATLAB against the Fine summary data over the 60 queries.<br />
 Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
@@ Line 56: / Line 56: @@
 <csv>AV_fine_result.csv</csv>
-==Summary Results by Query==
+===Summary Results by Query===
 <csv>mirex06_as_uberlist.csv</csv>
-==Raw Scores==
+===Raw Scores===
 The raw data derived from the Evalutron 6000 human evaluations are located on the [[Audio Music Similarity and Retrieval Raw Data]] page.
-==Query Meta Data==
+===Query Meta Data===
 <csv>as06_queries.csv</csv>
 ==Results from Automatic Evaluation==

@@ Line 138: / Line 138: @@
 * Establish optimal granularity or range of granularities for a genre catalogue to be used in this type of evaluation (8, 32 or 256 classes?) and integrate a confusion-cost matrix to reduce the penalisation of confusion between similar genres of music (e.g Punk and Heavy Metal) relative to confusion between highly dissimilar genres (e. Classical and Heavy Metal).
 [[User:Kriswest|Kriswest]]
 === Evaluation Tools in Music-2-Knowledge (M2K) ===
@@ Line 152: / Line 153: @@
 === Comments ===
-The evaluation statistics for the MIREX 2006 Audio music similarity contest seem to support the contention that genre, artist and artist-filtered genre neighbourhood statistics are correlated with the human perception of the performance of music similarity estimators as they all reproduce the ranking produced by the human evaluation. However, the differences between systems in that evaluation are not statistically significant, so no firm conclusion can be made. Average distance statistics produce a different ranking but are intended to correlate with visualisation performance and not search. [[User:Kriswest|Kriswest]]
+The evaluation statistics for the MIREX 2006 Audio music similarity contest seem to support the contention that genre, artist and artist-filtered genre neighbourhood statistics are correlated with the human perception of the performance of music similarity estimators as they all reproduce the ranking produced by the human evaluation. However, the differences between systems in that evaluation are not statistically significant, so no firm conclusion can be made. Average distance statistics produce a different ranking but are intended to correlate with visualisation performance and not search.

@@ Line 152: / Line 152: @@
 === Comments ===
 === References ===
 # [http://gatekeeper.research.compaq.com/pub/compaq/CRL/publications/logan/icme2001_logan.pdf Logan and Salomon (ICME 2001), '''A Music Similarity Function Based On Signal Analysis'''].<br>One of the first papers on this topic. Reports a small scale listening test (2 users) which rate items in a playlists as similar or not similar to the query song. In addition automatic evaluation is reported: percentage of top 5, 10, 20 most similar songs in the same genre/artist/album as query.
 # [http://www.ofai.at/~elias.pampalk/publications/pampalk06thesis.pdf E. Pampalk, '''Computational Models of Music Similarity and their Application in Music Information Retrieval.'''] <br>PhD thesis, Vienna University of Technology, Austria, March 2006

@@ Line 128: / Line 128: @@
 *** removing this one example from the collection of 5000 tracks makes it appear that the system does not suffer from indexing anomolies.
 ** What will be the number and coverage of ΓÇÿhubsΓÇÖ in a 100,000 song DB?
@@ Line 136: / Line 137: @@
 * Determine a safe minimum size for a test collection to be used to predict performance on an ΓÇÿindustrial-sizedΓÇÖ collection
 * Establish optimal granularity or range of granularities for a genre catalogue to be used in this type of evaluation (8, 32 or 256 classes?) and integrate a confusion-cost matrix to reduce the penalisation of confusion between similar genres of music (e.g Punk and Heavy Metal) relative to confusion between highly dissimilar genres (e. Classical and Heavy Metal).
+[[User:Kriswest|Kriswest]]
 === Evaluation Tools in Music-2-Knowledge (M2K) ===