MIREX Wiki - User contributions [en]

MIREX Impact Citations

2011-05-09T11:37:32Z

Dbogdanov: /* Submitted Citations */

=Introduction=

Dear MIREX Participant:

We are gathering evidence of impact for MIREX. We would appreciate it greatly if you would post citations to papers (yours or others) that have used MIREX data and/or results. Any acceptable citation format is OK. DOIs or URL to accessible copies especially appreciated. I have started out with some samples from IMIRSEL.

In a similar vein, we are also gathering citations to MIREX-related comments as further evidence of impact for MIREX on the [[MIREX Impact Statements]] page.

We are collecting these statements and citations as evidence influence and success to submit to funding agencies, research administrators and/or future collaborators.

Thank you very much.

J. Stephen Downie 
Director, IMIRSEL 

=Submitted Citations=

# J.S. Downie, D. Byrd, and T. Crawford. "Ten Years of ISMIR: Reflections on Challenges and Opportunities", in the 10th International Symposium on Music Information Retrieval (ISMIR), Kobe, 2009, pp. 13-18. Available: http://ismir2009.ismir.net/proceedings/keynote1.pdf
# M. Bay, A.F. Ehmann and J.S. Downie, "Evaluation of Multiple-F0 Estimation and Tracking Systems", in the 10th International Symposium on Music Information Retrieval (ISMIR), Kobe, 2009, pp. 315-320. Available: http://ismir2009.ismir.net/proceedings/PS2-21.pdf
# E. Law, K. West, M. Mandel, M. Bay and, J. S. Downie, "Evaluation of Algorithms Using Games: The Case of Music Tagging", in the 10th International Symposium on Music Information Retrieval (ISMIR), Kobe, 2009, pp. 387-392. Available: http://ismir2009.ismir.net/proceedings/OS5-5.pdf
# J. S. Downie, M. Bay, A. F. Ehmann and M. C. Jones, "Audio Cover Song Identification: MIREX 2006-2007 Results and analysis", in the 9th International Conference on Music Information Retrieval (ISMIR 2008), Philadelphia, 2008, pp. 468-473. Available: http://ismir2008.ismir.net/papers/ISMIR2008_265.pdf
# X. Hu, J. S. Downie, C. Laurier, M. Bay and A. F. Ehmann, "The 2007 MIREX Audio Mood Classification Task: Lessons learned", in the 9th International Conference on Music Information Retrieval (ISMIR 2008), Philadelphia, 2008, pp. 462-467. Available: http://ismir2008.ismir.net/papers/ISMIR2008_263.pdf
# J. S. Downie, A. F. Ehmann and J. H. Lee, "The Music Information Retrieval Evaluation eXchange (MIREX): Community-led formal evaluations", in the Digital Humanities 2008 , Oulu Finland, 2008, pp. 239-240.
# M. McVicar and T. De Bie, "Exploiting Online Resources to Improve Chord Recognition Accuracy", late-breaking abstract in the 11th International Conference on Music Information Retrieval (ISMIR 2010), Utrecht, Netherlands, 2010.
# M. McVicar and T. De Bie, "Enhancing chord recognition accuracy using web resources", 3rd ACM Workshop on Machine Learning and Music, October 25, 2010, Firenze, Italy.
# M. McVicar, Y. Ni, R. Santos-Rodriguez, T. De Bie, "Using online chord databases to enhance chord recognition", Journal of New Music Research, Special issue on Music and Machine Learning (in press).
#R. Zhou , M. Mattavelli and G. Zoia, "Music Onset Detection Based on Resonator Time-frequency Image", IEEE Transactions On Audio, Speech And Language Processing, vol. 16, num. 8, 2008, p. 1685-1695.
#Ruohua Zhou, Joshua D. Reiss, Marco Mattavelli and Giorgio Zoia, "A Computationally Efficient Method for Polyphonic Pitch Estimation", EURASIP Journal on Advances in Signal Processing, Volume 2009 (2009), Article ID 729494, 11 pages.
# M. Haro and P. Herrera, “From Low-level to Song-level Percussion Descriptors of Polyphonic Music,” in 10th International Society for Music Information Retrieval Conference (ISMIR 2009), Kobe, Japan, 2009. Available: http://ismir2009.ismir.net/proceedings/PS2-9.pdf
# J. Urbano, J. Morato, M. Marrero and D. Martín, "Crowdsourcing Preference Judgments for Evaluation of Music Similarity Tasks", ACM SIGIR Workshop on Crowdsourcing for Search Evaluation, pp. 9-16, 2010. [http://julian-urbano.info/publications/012-crowdsourcing-preference-judgments-evaluation-music-similarity-tasks/012-paper.pdf PDF]
# J. Urbano, M. Marrero, D. Martín and J. Lloréns, "Improving the Generation of Ground Truths based on Partially Ordered Lists", International Society for Music Information Retrieval Conference, pp. 285-290, 2010. [http://julian-urbano.info/publications/010-improving-generation-ground-truths-based-partially-ordered-lists/010-paper.pdf PDF]
# J. Urbano, J. Lloréns, J. Morato and S. Sánchez-Cuadrado, "Using the Shape of Music to Compute the Similarity between Symbolic Musical Pieces", International Symposium on Computer Music Modeling and Retrieval, pp. 385-396, 2010. [http://julian-urbano.info/publications/009-using-shape-music-compute-similarity-between-symbolic-musical-pieces/009-paper.pdf PDF]
# D. Bogdanov, J. Serrà, N. Wack, P. Herrera and X. Serra. Unifying low-level and high-level music similarity measures. IEEE Transactions on Multimedia. In press. [http://mtg.upf.edu/node/1689]

2010:Audio Music Similarity and Retrieval Results

2010-08-05T11:53:44Z

Dbogdanov: /* Reports */

== Introduction ==
These are the results for the 2010 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.

Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 100 songs were randomly selected from the 10 genre groups (10 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical '''BROAD''' score with 3 categories: NS,SS,VS as explained below, and one '''FINE''' score (in the range from 0 to 100). A description and analysis is provided below.

The systems read in 30 second audio clips as their raw data. The same 30 second clips were used in the grading stage.

=== General Legend ===

==== Team ID ====
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BWL1
| MTG-AMS || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/BWL1.pdf PDF] || [http://mtg.upf.edu Dmitry Bogdanov], [http://mtg.upf.edu Joan Serrà], [http://mtg.upf.edu Nicolas Wack], [http://mtg.upf.edu Perfecto Herrera]
|-
! PS1
| PS09 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/PS1.pdf PDF] || [http://www.cp.jku.at/ Tim Pohle], [http://www.cp.jku.at/ Dominik Schnitzer]
|-
! PSS1
| PSS10 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/PSS1.pdf PDF] || [http://www.cp.jku.at/ Tim Pohle], [http://www.cp.jku.at Klaus Seyerlehner], [http://www.cp.jku.at/ Dominik Schnitzer]
|-
! RZ1
| RND || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RZ1.pdf PDF] || [http://www.cp.jku.at Rainer Zufall]
|-
! SSPK2
| cbmr_sim || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/SSPK2.pdf PDF] || [http://www.cp.jku.at Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Tim Pohle], [http://www.cp.jku.at Peter Knees]
|-
! TLN1
| MarsyasSimilarity || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TNL1.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://sness.net Steven Ness], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange]
|-
! TLN2
| Post-Processing 1 of Marsyas similarity results || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TLN1.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange], [http://sness.net Steven Ness]
|-
! TLN3
| Post-Processing 2 of Marsyas similarity results || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TLN2.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange], [http://sness.net Steven Ness]
|}

====Broad Categories====
'''NS''' = Not Similar 
'''SS''' = Somewhat Similar 
'''VS''' = Very Similar 

=====Understanding Summary Measures=====
'''Fine''' = Has a range from 0 (failure) to 100 (perfection). 
'''Broad''' = Has a range from 0 (failure) to 2 (perfection) as each query/candidate pair is scored with either NS=0, SS=1 or VS=2. 

==Human Evaluation==
===Overall Summary Results===

<csv p=3>2010/ams/AMS2010summary_evalutron.csv</csv>
 
'''Note:RZ1''' is the random result for comparing purpose.

===Friedman's Tests===
====Friedman's Test (FINE Scores)====
The Friedman test was run in MATLAB against the '''Fine''' summary data over the 100 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=3>2010/ams/evalutron.fine.friedman.tukeyKramerHSD.csv</csv>

[[File:2010AMS.evalutron.fine.friedman.tukeyKramerHSD.png|500px]]

====Friedman's Test (BROAD Scores)====
The Friedman test was run in MATLAB against the '''BROAD''' summary data over the 100 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=3>2010/ams/evalutron.cat.friedman.tukeyKramerHSD.csv</csv>

[[File:2010AMS.evalutron.cat.friedman.tukeyKramerHSD.png|500px]]

===Summary Results by Query===
====FINE Scores====
These are the mean FINE scores per query assigned by Evalutron graders. The FINE scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 and 100. A perfect score would be 100. Genre labels have been included for reference.

<csv p=1>2010/ams/fine_scores.csv</csv>

====BROAD Scores====
These are the mean BROAD scores per query assigned by Evalutron graders. The BROAD scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 (not similar) and 2 (very similar). A perfect score would be 2. Genre labels have been included for reference.

<csv p=1>2010/ams/cat_scores.csv</csv>

===Raw Scores===
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2010:Audio Music Similarity and Retrieval Raw Data]] page.

==Metadata and Distance Space Evaluation==
The following reports provide evaluation statistics based on analysis of the distance space and metadata matches and include:
* Neighbourhood clustering by artist, album and genre
* Artist-filtered genre clustering
* How often the triangular inequality holds
* Statistics on 'hubs' (tracks similar to many tracks) and orphans (tracks that are not similar to any other tracks at N results).

=== Reports ===

'''BWL1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/BWL1/report.txt Dmitry Bogdanov, Joan Serrà, Nicolas Wack, Perfecto Herrera] 
'''PS1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/PS1/report.txt Tim Pohle, Dominik Schnitzer] 
'''PSS1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/PSS1/report.txt Tim Pohle, Klaus Seyerlehner, Dominik Schnitzer] 
'''RZ1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/RZ1/report.txt Rainer Zufall] 
'''SSPK2''' = [https://music-ir.org/mirex/results/2010/ams/statistics/SSPK2/report.txt Klaus Seyerlehner, Markus Schedl, Tim Pohle, Peter Knees] 
'''TLN1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN1/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
'''TLN2''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN2/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
'''TLN3''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN3/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 

== Run Times ==
<csv>2010/ams/audiosim.runtime.csv</csv>

2010:Audio Music Similarity and Retrieval Results

2010-08-05T11:53:24Z

Dbogdanov: /* Reports */

== Introduction ==
These are the results for the 2010 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.

Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 100 songs were randomly selected from the 10 genre groups (10 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical '''BROAD''' score with 3 categories: NS,SS,VS as explained below, and one '''FINE''' score (in the range from 0 to 100). A description and analysis is provided below.

The systems read in 30 second audio clips as their raw data. The same 30 second clips were used in the grading stage.

=== General Legend ===

==== Team ID ====
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BWL1
| MTG-AMS || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/BWL1.pdf PDF] || [http://mtg.upf.edu Dmitry Bogdanov], [http://mtg.upf.edu Joan Serrà], [http://mtg.upf.edu Nicolas Wack], [http://mtg.upf.edu Perfecto Herrera]
|-
! PS1
| PS09 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/PS1.pdf PDF] || [http://www.cp.jku.at/ Tim Pohle], [http://www.cp.jku.at/ Dominik Schnitzer]
|-
! PSS1
| PSS10 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/PSS1.pdf PDF] || [http://www.cp.jku.at/ Tim Pohle], [http://www.cp.jku.at Klaus Seyerlehner], [http://www.cp.jku.at/ Dominik Schnitzer]
|-
! RZ1
| RND || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RZ1.pdf PDF] || [http://www.cp.jku.at Rainer Zufall]
|-
! SSPK2
| cbmr_sim || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/SSPK2.pdf PDF] || [http://www.cp.jku.at Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Tim Pohle], [http://www.cp.jku.at Peter Knees]
|-
! TLN1
| MarsyasSimilarity || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TNL1.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://sness.net Steven Ness], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange]
|-
! TLN2
| Post-Processing 1 of Marsyas similarity results || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TLN1.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange], [http://sness.net Steven Ness]
|-
! TLN3
| Post-Processing 2 of Marsyas similarity results || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TLN2.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange], [http://sness.net Steven Ness]
|}

====Broad Categories====
'''NS''' = Not Similar 
'''SS''' = Somewhat Similar 
'''VS''' = Very Similar 

=====Understanding Summary Measures=====
'''Fine''' = Has a range from 0 (failure) to 100 (perfection). 
'''Broad''' = Has a range from 0 (failure) to 2 (perfection) as each query/candidate pair is scored with either NS=0, SS=1 or VS=2. 

==Human Evaluation==
===Overall Summary Results===

<csv p=3>2010/ams/AMS2010summary_evalutron.csv</csv>
 
'''Note:RZ1''' is the random result for comparing purpose.

===Friedman's Tests===
====Friedman's Test (FINE Scores)====
The Friedman test was run in MATLAB against the '''Fine''' summary data over the 100 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=3>2010/ams/evalutron.fine.friedman.tukeyKramerHSD.csv</csv>

[[File:2010AMS.evalutron.fine.friedman.tukeyKramerHSD.png|500px]]

====Friedman's Test (BROAD Scores)====
The Friedman test was run in MATLAB against the '''BROAD''' summary data over the 100 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=3>2010/ams/evalutron.cat.friedman.tukeyKramerHSD.csv</csv>

[[File:2010AMS.evalutron.cat.friedman.tukeyKramerHSD.png|500px]]

===Summary Results by Query===
====FINE Scores====
These are the mean FINE scores per query assigned by Evalutron graders. The FINE scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 and 100. A perfect score would be 100. Genre labels have been included for reference.

<csv p=1>2010/ams/fine_scores.csv</csv>

====BROAD Scores====
These are the mean BROAD scores per query assigned by Evalutron graders. The BROAD scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 (not similar) and 2 (very similar). A perfect score would be 2. Genre labels have been included for reference.

<csv p=1>2010/ams/cat_scores.csv</csv>

===Raw Scores===
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2010:Audio Music Similarity and Retrieval Raw Data]] page.

==Metadata and Distance Space Evaluation==
The following reports provide evaluation statistics based on analysis of the distance space and metadata matches and include:
* Neighbourhood clustering by artist, album and genre
* Artist-filtered genre clustering
* How often the triangular inequality holds
* Statistics on 'hubs' (tracks similar to many tracks) and orphans (tracks that are not similar to any other tracks at N results).

=== Reports ===

'''BWL1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/BWL1/report.txt Dmitry Bogdanov, Joan Serrà, Nicolas Wack, Perfecto Herrera] 
'''PS1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/PS1/report.txt Tim Pohle, Dominik Schnitzer] 
'''PSS1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/PSS1/report.txt Tim Pohle, Klaus Seyerlehner, Dominik Schnitzer] 
'''RZ1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/RZ1/report.txt Dmitry Rainer Zufall] 
'''SSPK2''' = [https://music-ir.org/mirex/results/2010/ams/statistics/SSPK2/report.txt Klaus Seyerlehner, Markus Schedl, Tim Pohle, Peter Knees] 
'''TLN1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN1/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
'''TLN2''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN2/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
'''TLN3''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN3/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 

== Run Times ==
<csv>2010/ams/audiosim.runtime.csv</csv>

2010:Audio Music Similarity and Retrieval Results

2010-08-05T10:55:22Z

Dbogdanov: /* Team ID */

== Introduction ==
These are the results for the 2010 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.

Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 100 songs were randomly selected from the 10 genre groups (10 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical '''BROAD''' score with 3 categories: NS,SS,VS as explained below, and one '''FINE''' score (in the range from 0 to 100). A description and analysis is provided below.

The systems read in 30 second audio clips as their raw data. The same 30 second clips were used in the grading stage.

=== General Legend ===

==== Team ID ====
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BWL1
| MTG-AMS || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/BWL1.pdf PDF] || [http://mtg.upf.edu Dmitry Bogdanov], [http://mtg.upf.edu Joan Serrà], [http://mtg.upf.edu Nicolas Wack], [http://mtg.upf.edu Perfecto Herrera]
|-
! PS1
| PS09 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/PS1.pdf PDF] || [http://www.cp.jku.at/ Tim Pohle], [http://www.cp.jku.at/ Dominik Schnitzer]
|-
! PSS1
| PSS10 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/PSS1.pdf PDF] || [http://www.cp.jku.at/ Tim Pohle], [http://www.cp.jku.at Klaus Seyerlehner], [http://www.cp.jku.at/ Dominik Schnitzer]
|-
! RZ1
| RND || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RZ1.pdf PDF] || [http://www.cp.jku.at Rainer Zufall]
|-
! SSPK2
| cbmr_sim || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/SSPK2.pdf PDF] || [http://www.cp.jku.at Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Tim Pohle], [http://www.cp.jku.at Peter Knees]
|-
! TLN1
| MarsyasSimilarity || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TNL1.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://sness.net Steven Ness], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange]
|-
! TLN2
| Post-Processing 1 of Marsyas similarity results || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TLN1.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange], [http://sness.net Steven Ness]
|-
! TLN3
| Post-Processing 2 of Marsyas similarity results || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TLN2.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange], [http://sness.net Steven Ness]
|}

====Broad Categories====
'''NS''' = Not Similar 
'''SS''' = Somewhat Similar 
'''VS''' = Very Similar 

=====Understanding Summary Measures=====
'''Fine''' = Has a range from 0 (failure) to 100 (perfection). 
'''Broad''' = Has a range from 0 (failure) to 2 (perfection) as each query/candidate pair is scored with either NS=0, SS=1 or VS=2. 

==Human Evaluation==
===Overall Summary Results===

<csv p=3>2010/ams/AMS2010summary_evalutron.csv</csv>
 
'''Note:RZ1''' is the random result for comparing purpose.

===Friedman's Tests===
====Friedman's Test (FINE Scores)====
The Friedman test was run in MATLAB against the '''Fine''' summary data over the 100 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=3>2010/ams/evalutron.fine.friedman.tukeyKramerHSD.csv</csv>

[[File:2010AMS.evalutron.fine.friedman.tukeyKramerHSD.png|500px]]

====Friedman's Test (BROAD Scores)====
The Friedman test was run in MATLAB against the '''BROAD''' summary data over the 100 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=3>2010/ams/evalutron.cat.friedman.tukeyKramerHSD.csv</csv>

[[File:2010AMS.evalutron.cat.friedman.tukeyKramerHSD.png|500px]]

===Summary Results by Query===
====FINE Scores====
These are the mean FINE scores per query assigned by Evalutron graders. The FINE scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 and 100. A perfect score would be 100. Genre labels have been included for reference.

<csv p=1>2010/ams/fine_scores.csv</csv>

====BROAD Scores====
These are the mean BROAD scores per query assigned by Evalutron graders. The BROAD scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 (not similar) and 2 (very similar). A perfect score would be 2. Genre labels have been included for reference.

<csv p=1>2010/ams/cat_scores.csv</csv>

===Raw Scores===
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2010:Audio Music Similarity and Retrieval Raw Data]] page.

==Metadata and Distance Space Evaluation==
The following reports provide evaluation statistics based on analysis of the distance space and metadata matches and include:
* Neighbourhood clustering by artist, album and genre
* Artist-filtered genre clustering
* How often the triangular inequality holds
* Statistics on 'hubs' (tracks similar to many tracks) and orphans (tracks that are not similar to any other tracks at N results).

=== Reports ===

'''BWL1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/BWL1/report.txt Dmitry Bogdanov, Nicolas Wack, Cyril Laurier] 
'''PS1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/PS1/report.txt Tim Pohle, Dominik Schnitzer] 
'''PSS1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/PSS1/report.txt Tim Pohle, Klaus Seyerlehner, Dominik Schnitzer] 
'''RZ1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/RZ1/report.txt Dmitry Rainer Zufall] 
'''SSPK2''' = [https://music-ir.org/mirex/results/2010/ams/statistics/SSPK2/report.txt Klaus Seyerlehner, Markus Schedl, Tim Pohle, Peter Knees] 
'''TLN1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN1/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
'''TLN2''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN2/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
'''TLN3''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN3/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
== Run Times ==
<csv>2010/ams/audiosim.runtime.csv</csv>

2010:Audio Music Similarity and Retrieval Results

2010-08-05T10:54:53Z

Dbogdanov: /* Team ID */

== Introduction ==
These are the results for the 2010 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.

Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 100 songs were randomly selected from the 10 genre groups (10 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical '''BROAD''' score with 3 categories: NS,SS,VS as explained below, and one '''FINE''' score (in the range from 0 to 100). A description and analysis is provided below.

The systems read in 30 second audio clips as their raw data. The same 30 second clips were used in the grading stage.

=== General Legend ===

==== Team ID ====
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BWL1
| MTG-AMS || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/BWL1.pdf PDF] || [http://mtg.upf.edu Dmitry Bogdanov], [http://mtg.upf.edu Nicolas Wack], [http://mtg.upf.edu Perfecto Herrera]
|-
! PS1
| PS09 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/PS1.pdf PDF] || [http://www.cp.jku.at/ Tim Pohle], [http://www.cp.jku.at/ Dominik Schnitzer]
|-
! PSS1
| PSS10 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/PSS1.pdf PDF] || [http://www.cp.jku.at/ Tim Pohle], [http://www.cp.jku.at Klaus Seyerlehner], [http://www.cp.jku.at/ Dominik Schnitzer]
|-
! RZ1
| RND || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RZ1.pdf PDF] || [http://www.cp.jku.at Rainer Zufall]
|-
! SSPK2
| cbmr_sim || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/SSPK2.pdf PDF] || [http://www.cp.jku.at Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Tim Pohle], [http://www.cp.jku.at Peter Knees]
|-
! TLN1
| MarsyasSimilarity || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TNL1.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://sness.net Steven Ness], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange]
|-
! TLN2
| Post-Processing 1 of Marsyas similarity results || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TLN1.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange], [http://sness.net Steven Ness]
|-
! TLN3
| Post-Processing 2 of Marsyas similarity results || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/TLN2.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis], [http://recherche.ircam.fr/equipes/analyse-synthese/home.html Mathieu Lagrange], [http://sness.net Steven Ness]
|}

====Broad Categories====
'''NS''' = Not Similar 
'''SS''' = Somewhat Similar 
'''VS''' = Very Similar 

=====Understanding Summary Measures=====
'''Fine''' = Has a range from 0 (failure) to 100 (perfection). 
'''Broad''' = Has a range from 0 (failure) to 2 (perfection) as each query/candidate pair is scored with either NS=0, SS=1 or VS=2. 

==Human Evaluation==
===Overall Summary Results===

<csv p=3>2010/ams/AMS2010summary_evalutron.csv</csv>
 
'''Note:RZ1''' is the random result for comparing purpose.

===Friedman's Tests===
====Friedman's Test (FINE Scores)====
The Friedman test was run in MATLAB against the '''Fine''' summary data over the 100 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=3>2010/ams/evalutron.fine.friedman.tukeyKramerHSD.csv</csv>

[[File:2010AMS.evalutron.fine.friedman.tukeyKramerHSD.png|500px]]

====Friedman's Test (BROAD Scores)====
The Friedman test was run in MATLAB against the '''BROAD''' summary data over the 100 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=3>2010/ams/evalutron.cat.friedman.tukeyKramerHSD.csv</csv>

[[File:2010AMS.evalutron.cat.friedman.tukeyKramerHSD.png|500px]]

===Summary Results by Query===
====FINE Scores====
These are the mean FINE scores per query assigned by Evalutron graders. The FINE scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 and 100. A perfect score would be 100. Genre labels have been included for reference.

<csv p=1>2010/ams/fine_scores.csv</csv>

====BROAD Scores====
These are the mean BROAD scores per query assigned by Evalutron graders. The BROAD scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 (not similar) and 2 (very similar). A perfect score would be 2. Genre labels have been included for reference.

<csv p=1>2010/ams/cat_scores.csv</csv>

===Raw Scores===
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2010:Audio Music Similarity and Retrieval Raw Data]] page.

==Metadata and Distance Space Evaluation==
The following reports provide evaluation statistics based on analysis of the distance space and metadata matches and include:
* Neighbourhood clustering by artist, album and genre
* Artist-filtered genre clustering
* How often the triangular inequality holds
* Statistics on 'hubs' (tracks similar to many tracks) and orphans (tracks that are not similar to any other tracks at N results).

=== Reports ===

'''BWL1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/BWL1/report.txt Dmitry Bogdanov, Nicolas Wack, Cyril Laurier] 
'''PS1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/PS1/report.txt Tim Pohle, Dominik Schnitzer] 
'''PSS1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/PSS1/report.txt Tim Pohle, Klaus Seyerlehner, Dominik Schnitzer] 
'''RZ1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/RZ1/report.txt Dmitry Rainer Zufall] 
'''SSPK2''' = [https://music-ir.org/mirex/results/2010/ams/statistics/SSPK2/report.txt Klaus Seyerlehner, Markus Schedl, Tim Pohle, Peter Knees] 
'''TLN1''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN1/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
'''TLN2''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN2/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
'''TLN3''' = [https://music-ir.org/mirex/results/2010/ams/statistics/TLN3/report.txt George Tzanetakis, Mathieu Lagrange, Steven Ness] 
== Run Times ==
<csv>2010/ams/audiosim.runtime.csv</csv>

2010:Audio Similarity 2010 Graders

2010-07-15T13:23:36Z

Dbogdanov: /* Sign Up Area */

=AMS 2010 Graders=

Welcome to the AMS grader sign-up page. Please give us your name and email contact information. If you obscure your email, please make it relatively obvious to us how to parse the address.

Template: Name. Location. <Email>

Sample: J. Stephen Downie. Illinois, USA. <jdownie@illinois.edu>

=Special Comments=
We are under extraordinary time constraints this year because ISMIR 2010
is being held starting 9 August and we need to have all the final
results calculated and posted by our 2 August target date (fingers crossed).

We hope to open the the Evalutron 6000 (E6K) v.2 grading system by 20
July. To meet our 2 August goal, we must have all the AMS and SMS
similarity grades entered into the E6K by 27 July. YES, THIS GIVES US
ONLY ONE WEEK! So, if you are kind enough to sign up to be a grader,
please understand that we really need you complete your assigned grading
by 27 July.

If you are a SMS or AMS participant, we ask that you do what you can to
encourage adults over 18 years of age to be graders.

We are looking for 50 graders for AMS this year. Each grader will be responsible for two query lists. This year the query lists seem to be moderate in length so tackling two queries lists should not be too onerous. For safety's sake, we would like to see say, an extra five or so names on the sign up sheet below. The "extra names" on the sign up sheet will be considered "back up" graders. We will assign grading tasks in the order of the names as they appear below.

=Sign Up Area=
#Martin Ariel Hartmann. Buenos Aires, Argentina. <martin.hartmann@jyu.fi>
#Sally Jo Cunningham. Hamilton, New Zealand. <sallyjo@cs.waikato.ac.nz>
#Matt Hoffman. New York, USA. <mdhoffma@cs.princeton.edu>
#Thierry Bertin-Mahieux. New York, USA. <tb2332@columbia.edu>
#Dominik Schnitzer, Vienna, Austria <dominik.schnitzer (dumdidum) ofai.at>
#Ming Li, Beijing, China, <liming . ioa [AT] gmail dot com>
#Charlie Inskip, London UK <c.inskip@city.ac.uk>
#Ana M. Barbancho, Malaga, Spain < abp [AT] ic dot uma dot es>
#Arthur Flexer, Vienna, Austria <arthur.flexer [AT] ofai.at>
#Simone Sammartino, Málaga, Spain <ssammartino@ic.uma.es>
#Cristina de la Bandera Cascales, Málaga, Spain <cdelabandera@ic.uma.es>
#Peter Knees, Linz, Austria <peter.knees (AT) jku.at>
#Steve Tjoa. Maryland, USA. <kiemyang at umd edu>
#Michael Mandel, Montreal, QC <mandelm at iro umontreal ca>
#Perfecto Herrera, Barcelona, Spain <perfecto dot herrera [AT] upf dot es>
#Aggelos Gkiokas, Athens, Greece <agkiokas@ilsp.gr>
#Dmitry Bogdanov, Barcelona, Spain <dmitry dot bogdanov [AT] upf dot es>