2007:Audio Music Similarity and Retrieval Results

From MIREX Wiki
Revision as of 06:10, 13 September 2007 by JDownie (talk | contribs)

TEMPLATE: NEEDS TO BE UPDATED

Introduction

These are the results for the 2007 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.

Each system was given 5000 songs chosen from "uspop", "uscrap" and "cover song" collections. Each system then returned a 5000x5000 distance matrix. 60 songs were randomly selected as queries and the first 5 most highly ranked songs out of the 5000 were extracted for each query (after filtering out the query itself, returned results from the same artist and members of the cover song collection). Then, for each query, the returned results from all participants were grouped and were evaluated by human graders, each query being evaluated by 3 different graders with two scores (using the Evalutron 6000 system). Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 10). An automated statistical evaluation based on a metadata catalog was also conducted. A desciption and analysis is provided below.

Summary Data on Human Evaluations (Evalutron 6000)

Number of evaluators = 24
Number of evaluation per query/candidate pair = 3
Number of queries per grader = 7~8
Size of the candidate lists = Maximum 30 (with no overlap)
Number of randomly selected queries = 60

General Legend

Team ID

EP = Elias Pampalk
TP = Tim Pohle
VS = Vitor Soares
LR = Thomas Lidy and Andreas Rauber
KWT = Kris West (Trans)
KWL = Kris West (Likely)

=Broad Categories

NS = Not Similar
SS = Somewhat Similar
VS = Very Similar

Calculating Summary Measures

Fine(1) = Sum of fine-grained human similarity decisions (0-10).
PSum(1) = Sum of human broad similarity decisions: NS=0, SS=1, VS=2.
WCsum(1) = 'World Cup' scoring: NS=0, SS=1, VS=3 (rewards Very Similar).
SDsum(1) = 'Stephen Downie' scoring: NS=0, SS=1, VS=4 (strongly rewards Very Similar).
Greater0(1) = NS=0, SS=1, VS=1 (binary relevance judgement).
Greater1(1) = NS=0, SS=0, VS=1 (binary relevance judgement using only Very Similar).

(1)Normalized to the range 0 to 1.

Overall Summary Results

file /nema-raid/www/mirex/results/mirex06_as_overalllist.csv not found

http://staff.aist.go.jp/elias.pampalk/papers/mirex06/friedman.png

This figure shows the official ranking of the submissions computed using a Friedman test. The blue lines indicate significance boundaries at the p=0.05 level. As can be seen, the differences are not significant. For a more detailed description and discussion see [1].

Audio Music Similarity and Retrieval Runtime Data

file /nema-raid/www/mirex/results/as06_runtime.csv not found

For a description of the computers the submission ran on see MIREX_2006_Equipment.

Friedman Test with Multiple Comparisons Results (p=0.05)

The Friedman test was run in MATLAB against the Fine summary data over the 60 queries.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/AV_sum_friedman.csv not found file /nema-raid/www/mirex/results/AV_fine_result.csv not found

Summary Results by Query

file /nema-raid/www/mirex/results/mirex06_as_uberlist.csv not found

Raw Scores

The raw data derived from the Evalutron 6000 human evaluations are located on the Audio Music Similarity and Retrieval Raw Data page.

Query Meta Data

file /nema-raid/www/mirex/results/as06_queries.csv not found


Results from Automatic Evaluation

file /nema-raid/www/mirex/results/as06_nonhuman_results.csv not found


Other Results from Automatic Evaluation

See Audio Music Similarity and Retrieval Other Automatic Evaluation Results page.


Introduction to automatic evaluation

Automated evaluation of music similarity techniques based on a metadata catalogue has several advantages:

  • It does not require costly human ΓÇÿgradersΓÇÖ
  • Allows testing of incremental changes in indexing algorithms
  • Can achieve complete coverage over the test collection
  • Provides a target for machine-learning, feature-selection and optimisation experiments
  • Can predict the visualisation performance of an indexing technique
  • Can identify indexing ΓÇÿanomoliesΓÇÖ in the indices tested

Automated ΓÇÿpseudo-objectiveΓÇÖ evaluation of music similarity estimation techniques was introduced by Logan & Saloman [1] and were shown to be highly correlated with careful human-based evaluations by Pampalk [2]. The results of this contest support the conclusions of Pampalk [2] although further work is required to fully understand the evaluation statistics.


Description of evaluation statistics

The evaluation statistics

Neighbourhood clustering (artist, genre, album)
average % of the top N results for each query in the collection with the same same label
Artist-filtered genre neighbourhood
average % of the top N results for each query belonging to the same genre label, ignoring matches from the same artist (ensures that results reflect musical not audio similarity)
Mean Artist-filtered genre neighbourhood
normalised form of the above statistic equally weighting each genre, penalising lop-sided performance.
Normalised average distance between examples
average distance between examples with the same label, indicates degree of clustering and potential for visual organisation of a collection
Always similar (hubs)
largest # of times an example appears in top N results for other queries, a result that appears too often will adversely affect performance without affecting other statistics
Never similar (orphans)
% of examples that never appear in a top N result list and cannot be retrieved by search
Triangular inequality (metric space)
indicates whether the function produces a metric distance space and therefore what visualisation techniques may be applied to it


Normalisation

Each of the neighbourhood statistics, described above, has been normalised by the number of examples of each class (a genre, album or artist) that is available in the test database. E.g. if the collection contained 20 tracks by a particular artist and a particular system retrieved 10 of those examples in its top 50 results it would normally achieve an artist neighbourhood score of 20%, while the normalised form of the metric would report a score of 50% (of the available matches were retrieved). Such normalisation is intended to avoid bias introduced into the results by skewed distribution of the examples according to each label set.

The mean artist-filtered genre neighbourhood is a normalised form of the artist-filter genre neighbourhood metric, which gives equal weight to performance of a system on each genre class. This version of the statistic is intended to match the prior probabilities or distribtuion of examples according to genre labels used as queries in the human listening test (where an equal number of examples from each class was selected - stratified random sampling) instead of the prior probabilities or distribution of examples appearing in the database.


Music-similarity evaluation issues

Care must be taken with all evaluations of audio musical similarity estimation techniques as there is a great potential for over-fitting in these experiments and for over-optimistic estimates of the performance of a system on novel test data to be produced.

The metadata catalog used to conduct automated evaluations should be as accurate as possible. However, this technique seems relatively robust to a degree of noise in the catalogue, parhaps due to its coarse granularity.

Small test collections do not allow us to accurately predict performance on larger test collections, for example:

  • Indexing anomalies (ΓÇÿhubsΓÇÖ and ΓÇÿorphansΓÇÖ) cannot yet be understood.
    • a single ΓÇÿhubΓÇÖ was found in the results of one system
      • appeared in nearly 2/5 of result lists
      • removing this one example from the collection of 5000 tracks makes it appear that the system does not suffer from indexing anomolies.
    • What will be the number and coverage of ΓÇÿhubsΓÇÖ in a 100,000 song DB?

Kriswest


Directions for further work on evaluating audio music similarity

  • Establish whether stratified sampling used in the human evals is optimal for producing results that reflect human perception of the quality of music indexes or whether the database should be sampled randomly.
    • will influence selection of a statistic for use in automated evaluations or optimisation exps (artist-filtered genre or the mean artist-filtered genre).
  • Explain the indexing anomalies in some techniques.
  • Determine a safe minimum size for a test collection to be used to predict performance on an ΓÇÿindustrial-sizedΓÇÖ collection
  • Establish optimal granularity or range of granularities for a genre catalogue to be used in this type of evaluation (8, 32 or 256 classes?) and integrate a confusion-cost matrix to reduce the penalisation of confusion between similar genres of music (e.g Punk and Heavy Metal) relative to confusion between highly dissimilar genres (e. Classical and Heavy Metal).

Kriswest


Evaluation Tools in Music-2-Knowledge (M2K)

The tools used to produce the evaluation statistics for MIREX 2006 will be released as part of M2K 1.2 (forthcoming). These tools provide services to:

  • import collection metadata and distance matrices
  • generate a stratified query set
  • extract artist-filtered results (for use in human evaluation exps)
  • calculate any of the evaluation statistics described above.

These tools may be used on the command-line by implementing the MIREX distance matrix file format, with M2K in the Data-2-Knowledge toolkit (D2K) or integrated into existing Java code with the new M2K API.

To obtain a copy of the evaluation tools prior to the M2K 1.2 release, contact Kris West


Comments

The evaluation statistics for the MIREX 2006 Audio music similarity contest seem to support the contention that genre, artist and artist-filtered genre neighbourhood statistics are correlated with the human perception of the performance of music similarity estimators as they all reproduce the ranking produced by the human evaluation. However, the differences between systems in that evaluation are not statistically significant, so no firm conclusion can be made. Average distance statistics produce a different ranking but are intended to correlate with visualisation performance and not search. Kriswest


A statistic for evaluation and use in selection & optimization experiments

As each statitic was found to be correlated with the results of the listening test, any *may* be used to evaluate performance and to guide model optimisation or feature selection/weighting experiments. However, unfiltered genre and artist identification statistics are known to allow overfitting to produce over-optimistic performance estimates. In a model optimisation or feature selection experiment these statistics will be more likely to indicate Audio-similarity performance rather than actual Music-similarity performance and may lead to the selection of sub-optimal features or models. The artist-filtered genre neighbourhood can be used to avoid this effect.

The results from MIREX 2006 do not show a significant drop in performance using the artist-filtered genre statistic as would normally be expected. This may be due to the excessively skewed distribution of examples in the database (roughly 50% of examples are labelled as Rock/Pop, while a further 25% are Rap & Hip-Hop). Hence, the difference between the results produced and the random baseline are not well emphasized. Normalising this statistic by the prior probabilities of examples in the database (taking the mean of the diagonal of the artist-filtered genre confusion matrix) equally weights the contribution of each class to the final statistic and prevents performance on a single class dominating the statistic. This normalised statistic shows a drastic reduction in the performance estimates for each system and increases the relative distance between each of the systems in the evaluation. Kriswest

References

  1. Logan and Salomon (ICME 2001), A Music Similarity Function Based On Signal Analysis.
    One of the first papers on this topic. Reports a small scale listening test (2 users) which rate items in a playlists as similar or not similar to the query song. In addition automatic evaluation is reported: percentage of top 5, 10, 20 most similar songs in the same genre/artist/album as query.
  2. E. Pampalk, Computational Models of Music Similarity and their Application in Music Information Retrieval.
    PhD thesis, Vienna University of Technology, Austria, March 2006
Retrieved from "https://music-ir.org/mirex/w/index.php?title=2007:Audio_Music_Similarity_and_Retrieval_Results&oldid=3182"