|
|
| Line 1: |
Line 1: |
| | [[Category: Results]] | | [[Category: Results]] |
| − | NEEDS TO BE UPDATED: TEMPLATE ONLY
| |
| | ==Introduction== | | ==Introduction== |
| − | These are the results for the 2007 running of the Audio Cover Song Identification task. For background information about this task set please refer to the [[Audio Cover Song]] page. | + | These are the results for the 2007 running of the Audio Artist Identification task. For background information about this task set please refer to the [[Audio Artist Identification]] page. |
| − | | |
| − | Each system was given a collection of 1000 songs which included of 30 different classes (sets) of cover songs where each class/set was represented by 11 different versions of a particular song. Each of the 330 cover songs were used as queries and the systems were required to return 10 results for each query. Systems were evaluated on the number of the songs from the same class/set as the query that were returned in the list of 10 results for each query. '''Special note''': Some systems (e.g., KWL, KWT, LR and TP) participated as a by-product of the [[Audio Music Similarity and Retrieval]] task, that is, these systems were not specifically written to detect cover song variants. Other systems (e.g., CS, DE, KL1 and KL2) were specifically written to detect cover song variants.
| |
| − | | |
| | | | |
| | ===General Legend=== | | ===General Legend=== |
| | ====Team ID==== | | ====Team ID==== |
| − | '''CS''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/CS_sailer.pdf Christian Sailer and Karin Dressler<sup>(1)</sup>]<br /> | + | '''MM''' = [https://www.music-ir.org/evaluation/MIREX/2007_abstracts/AI_mandel.pdf Michael Mandel]<br /> |
| − | '''DE''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/CS_ellis.pdf Daniel P. W. Ellis <sup>(1)</sup>]<br /> | + | '''TL''' = [https://www.music-ir.org/evaluation/MIREX/2007_abstracts/AI_lidy.pdf Thomas Lidy, Andreas Rauber, Antonio Pertusa and Jose Manuel Inesta]<br /> |
| − | '''KL1''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/CS_lee.pdf Kyogu Lee 1 <sup>(1)</sup>]<br /> | + | '''GT''' = [https://www.music-ir.org/evaluation/MIREX/2007_abstracts/AI_tzanetakis.pdf George Tzanetakis]<br /> |
| − | '''KL2''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/CS_lee.pdf Kyogu Lee 2 <sup>(1)</sup>]<br /> | + | '''KL''' = [https://www.music-ir.org/evaluation/MIREX/2007_abstracts/AI_lee.pdf Kyogu Lee]<br /> |
| − | '''KWL''' = Kris West (Likely) <sup>(2)</sup><br />
| + | '''IM''' = [https://www.music-ir.org/evaluation/MIREX/2007_abstracts/AI_imirsel.pdf IMIRSEL]<br /> |
| − | '''KWT''' = Kris West (Trans) <sup>(2)</sup><br />
| |
| − | '''LR''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/AS_lidy.pdf Thomas Lidy and Andreas Rauber] <sup>(2)</sup><br />
| |
| − | '''TP''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/AS_pohle.pdf Tim Pohle] <sup>(2)</sup><br /> | |
| − | | |
| − | <sup>(1)</sup> Denotes submissions specifically designed to detect cover song variants.<br/>
| |
| − | <sup>(2)</sup> Denotes submissions not-specifically designed to detect cover song variants (See [[Audio Music Similarity and Retrieval Results]]).
| |
| − | | |
| − | ===Calculating Summary Measures===
| |
| − | '''MRR''' = Mean Reciprocal Rank. Reciprocal rank is the reciprocal of the rank of the first correctly identified cover for each query (1/rank). These values are averaged for each cover song group as well as overall.<br />
| |
| − | | |
| − | ==Overall Summary Results==
| |
| − | <csv>coversong_overall.csv</csv>
| |
| − | | |
| − | ===Audio Cover Song Identification Runtime Data===
| |
| − | | |
| − | <csv>ac06_runtime.csv</csv>
| |
| − | | |
| − | ==Friedman Test with Multiple Comparisons Results (p=0.05)==
| |
| − | The Friedman test was run in MATLAB against the MRR summary data over the 30 song groups.<br /> Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
| |
| − | | |
| − | <csv>cover_song_sum_friedman.csv</csv>
| |
| − | <csv>audiocover_friedman_mrr.csv</csv>
| |
| − | | |
| − | ==Mean Reciprocal Rank (MRR) Results==
| |
| − | <csv>coversong_mrr.csv</csv>
| |
| − | | |
| − | ==Raw Results==
| |
| − | The total number of correctly identified covers for each query, and the maximum number of covers identified for each cover song group.
| |
| − | <csv>coversong_results.csv</csv>
| |
