MIREX Wiki - User contributions [en]

2013:Symbolic Melodic Similarity Results

2013-10-31T15:44:20Z

Jurbano: edited authors in JUx submissions

==Introduction==
These are the results for the 2013 running of the Symbolic Melodic Similarity task set. For background information about this task set please refer to the [[2013:Symbolic Melodic Similarity]] page.

Each system was given a query and returned the 10 most melodically similar songs from those taken from the Essen Collection (5274 pieces in the MIDI format; see [http://www.esac-data.org/ ESAC Data Homepage] for more information). For each query, we made four classes of error-mutations, thus the set comprises the following query classes:

* 0. No errors
* 1. One note deleted
* 2. One note inserted
* 3. One interval enlarged
* 4. One interval compressed

For each query (and its 4 mutations), the returned results (candidates) from all systems were then grouped together (query set) for evaluation by the human graders. The graders were provide with only heard perfect version against which to evaluate the candidates and did not know whether the candidates came from a perfect or mutated query. Each query/candidate set was evaluated by 1 individual grader. Using the Evalutron 6000 system, the graders gave each query/candidate pair two types of scores. Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 100).

====Evalutron 6000 Summary Data====
'''Number of evaluators''' = 6 
'''Number of evaluations per query/candidate pair''' = 1 
'''Number of queries per grader''' = 1 
'''Total number of unique query/candidate pairs graded''' = 388 
'''Average number of query/candidate pairs evaluated per grader: 67 
'''Number of queries''' = 6 (perfect) with each perfect query error-mutated 4 different ways = 30 

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="540" | Contributors
|-

! JU1
| ShapeH || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/JU1.pdf PDF] ||[http://julian-urbano.info Julián Urbano]
|-

! JU2
| ShapeTime || style="text-align: center;" |[https://www.music-ir.org/mirex/abstracts/2013/JU2.pdf PDF] || [http://julian-urbano.info Julián Urbano]
|-

! JU3
| Time || style="text-align: center;" |[https://www.music-ir.org/mirex/abstracts/2013/JU3.pdf PDF] ||[http://julian-urbano.info Julián Urbano]
|-

! RTBB1
| ATIC_SMS_2013 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/RTBB1.pdf PDF] || [http://www.ic.uma.es Carles Roig], [http://www.atic.uma.es Lorenzo J. Tardón],[http://www.ic.uma.es Ana María Barbancho], [http://www.atic.uma.es Isabel Barbancho]
|-

! YHKH1

| sys-IRM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/YHKH1.pdf PDF] ||[http://music.iit.tsukuba.ac.jp/ Sakurako Yazawa], [http://music.iit.tsukuba.ac.jp/ Yuhei Hasegawa ],[http://music.iit.tsukuba.ac.jp/ Kouhei Kanamori], [http://music.iit.tsukuba.ac.jp/ Masatoshi Hamanaka ]
|-

|}

====Broad Categories====
'''NS''' = Not Similar 
'''SS''' = Somewhat Similar 
'''VS''' = Very Similar 

====Table Headings ====
'''ADR''' = Average Dynamic Recall 
'''NRGB''' = Normalize Recall at Group Boundaries 
'''AP''' = Average Precision (non-interpolated) 
'''PND''' = Precision at N Documents 

===Calculating Summary Measures===
'''Fine'''(1) = Sum of fine-grained human similarity decisions (0-100). 
'''PSum'''(1) = Sum of human broad similarity decisions: NS=0, SS=1, VS=2. 
'''WCsum'''(1) = 'World Cup' scoring: NS=0, SS=1, VS=3 (rewards Very Similar). 
'''SDsum'''(1) = 'Stephen Downie' scoring: NS=0, SS=1, VS=4 (strongly rewards Very Similar). 
'''Greater0'''(1) = NS=0, SS=1, VS=1 (binary relevance judgment). 
'''Greater1'''(1) = NS=0, SS=0, VS=1 (binary relevance judgment using only Very Similar). 

(1)Normalized to the range 0 to 1.

==Summary Results==
===Overall Scores (Includes Perfect and Error Candidates)===
<csv>2013/sms/report_all.csv</csv>

===Scores by Query Error Types===
====No Errors====
<csv>2013/sms/No_Errors.csv</csv>
====Note Deletions====
<csv>2013/sms/Deleted.csv</csv>
====Note Insertions====
<csv>2013/sms/Inserted.csv</csv>
====Enlarged Intervals====
<csv>2013/sms/Enlarged.csv</csv>
====Compressed Intervals====
<csv>2013/sms/Compressed.csv</csv>

===Friedman Test with Multiple Comparisons Results (p=0.05)===
The Friedman test was run in MATLAB against the Fine summary data over the 30 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
<csv>2013/sms/sum_friedman_fine.csv</csv>

<csv>2013/sms/sms_fine_scores_friedman.csv</csv>

[[Image:2013 sms fine scores friedmans.png]]

[[Category: Results]]

2012:Audio Similarity 2012 Graders

2012-09-09T18:34:30Z

Jurbano:

=AMS 2012 Graders=

Welcome to the AMS grader sign-up page. Please give us your name and email contact information. If you obscure your email, please make it relatively obvious to us how to parse the address.

Template: Name. Location. <Email>

Sample: J. Stephen Downie. Illinois, USA. <jdownie@illinois.edu>

==Special Comments==
We are under the usual time constraints this year. ISMIR 2012 begins 8
October. We need to have all the MIREX results calculated and posted by
our 1 October target date (fingers crossed)!

We hope to open the Evalutron 6000 (E6K) v.2 grading system by
Friday, 14 September. To meet our 1 October goal, we must have all the
AMS and SMS similarity grades entered into the E6K by Wednesday, 26 September. So, if you are kind enough to sign up to be a grader, PLEASE understand
that we REALLY need you complete your assigned grading by 26 September.

If you are a SMS or AMS participant, we ask that you do what you can to
encourage adults over 18 years of age to be graders.

We are looking for 50 graders for AMS this year. If we make our quota of 50 graders, each grader will be responsible for two query lists. If we fall short, and get around 34 graders, we will be asking each grader to grade 3 queries. In this worst case scenario, we still expect the grading process to take between 2.5 to 3 hours (or less) for each grader.

This year the query lists seem to be moderate in length so tackling two queries lists should not be too onerous. For safety's sake, we would like to see say, an extra 5 or so names on the sign up sheet below. The "extra names" on the sign up sheet will be considered "back up" graders. We will assign grading tasks in the order of the names as they appear below.

=Sign Up Area=

# Minje Kim. Savoy, IL, USA. <minje.kim at gmail com>
# Jia-Min Ren, National Tsing Hua University, Hsinchu, Taiwan. <jmren at mirlab.org>
# Nick Seaver, UC Irvine, CA, USA. <nseaver at uci.edu>
# Julián Urbano, Spain <jurbano at inf.uc3m.es>

2011:Symbolic Melodic Similarity Results

2011-11-04T17:30:09Z

Jurbano:

==Introduction==
These are the results for the 2011 running of the Symbolic Melodic Similarity task set. For background information about this task set please refer to the [[2011:Symbolic Melodic Similarity]] page.

Each system was given a query and returned the 10 most melodically similar songs from those taken from the Essen Collection (5274 pieces in the MIDI format; see [http://www.esac-data.org/ ESAC Data Homepage] for more information). For each query, we made four classes of error-mutations, thus the set comprises the following query classes:

* 0. No errors
* 1. One note deleted
* 2. One note inserted
* 3. One interval enlarged
* 4. One interval compressed

For each query (and its 4 mutations), the returned results (candidates) from all systems were then grouped together (query set) for evaluation by the human graders. The graders were provide with only heard perfect version against which to evaluate the candidates and did not know whether the candidates came from a perfect or mutated query. Each query/candidate set was evaluated by 1 individual grader. Using the Evalutron 6000 system, the graders gave each query/candidate pair two types of scores. Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 100).

====Evalutron 6000 Summary Data====
'''Number of evaluators''' = 6 
'''Number of evaluations per query/candidate pair''' = 1 
'''Number of queries per grader''' = 1 
'''Total number of candidates returned''' = 3900 
'''Total number of unique query/candidate pairs graded''' = 895 
'''Average number of query/candidate pairs evaluated per grader: 149 
'''Number of queries''' = 6 (perfect) with each perfect query error-mutated 4 different ways = 30 

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="540" | Contributors
|-

! LJY1
| LEE1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/LJY1.pdf PDF] || [http://slsp.kaist.ac.kr Juwan Lee],[http://slsp.kaist.ac.kr Seokhwan Jo],[http://slsp.kaist.ac.kr Chang D. Yoo]
|-

! LJY2
| LEE2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/LJY2.pdf PDF] || [http://slsp.kaist.ac.kr Juwan Lee],[http://slsp.kaist.ac.kr Seokhwan Jo],[http://slsp.kaist.ac.kr Chang D. Yoo]
|-

! UL1
| Shape || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/UL1.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns],[http://sites.google.com/site/jorgemorato/ Jorge Morato], [http://www.inf.uc3m.es/en/component/comprofiler/userprofile/sscuadra Sonia Sánchez-Cuadrado]
|-

! UL2
| Pitch || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/UL2.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns],[http://sites.google.com/site/jorgemorato/ Jorge Morato], [http://www.inf.uc3m.es/en/component/comprofiler/userprofile/sscuadra Sonia Sánchez-Cuadrado]
|-

! UL3
| Time || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/UL3.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns],[http://sites.google.com/site/jorgemorato/ Jorge Morato], [http://www.inf.uc3m.es/en/component/comprofiler/userprofile/sscuadra Sonia Sánchez-Cuadrado]
|-

! WK1
| Tir'a'Mir - binary || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK1.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK2
| Tir'a'Mir - melodic cosine || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK2.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK3
| Tir'a'Mir - cosine combine || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK3.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK4
| Tir'a'Mir - tfidf || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK4.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK5
| Tir'a'Mir - melodic bm25 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK5.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK6
| Tir'a'Mir - bm25 combine || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK6.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

|}

====Broad Categories====
'''NS''' = Not Similar 
'''SS''' = Somewhat Similar 
'''VS''' = Very Similar 

====Table Headings ====
'''ADR''' = Average Dynamic Recall 
'''NRGB''' = Normalize Recall at Group Boundaries 
'''AP''' = Average Precision (non-interpolated) 
'''PND''' = Precision at N Documents 

===Calculating Summary Measures===
'''Fine'''(1) = Sum of fine-grained human similarity decisions (0-100). 
'''PSum'''(1) = Sum of human broad similarity decisions: NS=0, SS=1, VS=2. 
'''WCsum'''(1) = 'World Cup' scoring: NS=0, SS=1, VS=3 (rewards Very Similar). 
'''SDsum'''(1) = 'Stephen Downie' scoring: NS=0, SS=1, VS=4 (strongly rewards Very Similar). 
'''Greater0'''(1) = NS=0, SS=1, VS=1 (binary relevance judgment). 
'''Greater1'''(1) = NS=0, SS=0, VS=1 (binary relevance judgment using only Very Similar). 

(1)Normalized to the range 0 to 1.

==Summary Results==
===Overall Scores (Includes Perfect and Error Candidates)===
<csv>2011/sms/Overall.csv</csv>

===Scores by Query Error Types===

<csv>2011/sms/No_Errors.csv</csv>

<csv>2011/sms/Deleted.csv</csv>

<csv>2011/sms/Inserted.csv</csv>

<csv>2011/sms/Enlarged.csv</csv>

<csv>2011/sms/Compressed.csv</csv>

===Friedman Test with Multiple Comparisons Results (p=0.05)===
The Friedman test was run in MATLAB against the Fine summary data over the 30 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
<csv>2011/sms/sum_friedman_fine.csv</csv>

<csv>2011/sms/sms_fine_scores_friedman.csv</csv>

[[Image:2011 sms fine scores friedmans.png]]

TBA

==Raw Scores==
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2011:Symbolic Melodic Similarity Raw Data]] page.

===Run Times===
TBA

[[Category: Results]]

2011:MIREX 2011 Poster List

2011-10-18T19:39:12Z

Jurbano:

==MIREX 2011 Poster Session Planning List==
The MIREX 2011 Poster Session will be held Thursday, 27 October: 12:00-13:50. We will be holding the MIREX plenary meeting 11:00-12:00 on the same day.

Our hosts in Miami need to know the number of posters so they can set up the room. Please add you name and the task(s) dealt with in your poster.

We had many groups/individuals submit across tasks. You can choose to create one ISMIR poster bringing all your data together or can split up your data across, say, two or three posters. If you have questions, please contact me at jdownie@illinois.edu or the MIREX mailing list about task poster options.

===Poster Guidelines===
As in the past, MIREX will follow the ISMIR poster guidelines. All poster sessions, excluding the late-breaking/demo session on Friday, the 28th will be held in the Regency Ballroom. The dimension of a poster board is 36in by 48in (or 48in by 36in if used in landscape mode). A presenter will be responsible for removing his/her poster at the conclusion of its session.

===Possible Printing Option===
A FedEx/Kinko (4441 Collins Avenue) shop is near the conference hotel (4833 Collins Avenue) about four blocks away. The turnaround time is about four hours for walk-in orders. A google map shows the hotel and the shop can be found [http://maps.google.com/maps/ms?f=q&source=s_q&hl=en&geocode=&aq=&vpsrc=6&gl=us&ie=UTF8&hq=fedex+kinkos+miami+beach+collins&hnear=&t=m&msa=0&msid=207227323913999833158.0004af7ffbb36f333fa3b&sll=25.733107,-80.197449&sspn=0.376694,0.473785&ll=25.8172,-80.123162&spn=0.023527,0.029612&z=15 here].

==Add your author names here, once for each poster along with "title of some sort" and (Task(s) covered)==
# IMIRSEL: ''MIREX 2011 Overview, Part I'' (Train Test Tasks)
# IMIRSEL: ''MIREX 2011 Overview, Part II'' (All Other Tasks)
# Franz de Leon, Kirk Martinez: WAIS @ MIREX 2011 (AMS and Genre Classifacation Tasks)
# Brian McFee and Gert Lanckriet: ''Audio similarity via metric learning'' (AMS)
# J. Urbano, J. Lloréns, J. Morato and S. Sánchez-Cuadrado: ''MIREX 2011 Symbolic Melodic Similarity: Sequence Alignment with Geometric Representations''

==Below are some examples from MIREX 2009==

# Matt Hoffman: ''Using CBA to Automatically Tag Songs'' (Audio tag classification/retrieval)
# Suman Ravuri, Dan Ellis: ''The Hydra System of Cover Song Classification'' (Cover Song Identification)
# Joan Serra, Massimiliano Zanin, Ralph G Andrzejak: ''Cover song retrieval by recurrence quantification and unsupervised set detection'' (Cover Song Identification)
# MTG Team: "Music Type Groupers (MTG): Generic Music Classification Algorithms" (Audio Genre Classification, Mood Classification, Artist Identification, Classical Composer Identification)
# R. Jang: "Poster #2" (placeholder to get the auto-counter to increment)
# R. Jang: "Poster #3" (placeholder to get the auto-counter to increment)

MIREX Impact Citations

2011-05-06T22:34:52Z

Jurbano:

=Introduction=

Dear MIREX Participant:

We are gathering evidence of impact for MIREX. We would appreciate it greatly if you would post citations to papers (yours or others) that have used MIREX data and/or results. Any acceptable citation format is OK. DOIs or URL to accessible copies especially appreciated. I have started out with some samples from IMIRSEL.

In a similar vein, we are also gathering citations to MIREX-related comments as further evidence of impact for MIREX on the [[MIREX Impact Statements]] page.

We are collecting these statements and citations as evidence influence and success to submit to funding agencies, research administrators and/or future collaborators.

Thank you very much.

J. Stephen Downie 
Director, IMIRSEL 

=Submitted Citations=

# J.S. Downie, D. Byrd, and T. Crawford. "Ten Years of ISMIR: Reflections on Challenges and Opportunities", in the 10th International Symposium on Music Information Retrieval (ISMIR), Kobe, 2009, pp. 13-18. Available: http://ismir2009.ismir.net/proceedings/keynote1.pdf
# M. Bay, A.F. Ehmann and J.S. Downie, "Evaluation of Multiple-F0 Estimation and Tracking Systems", in the 10th International Symposium on Music Information Retrieval (ISMIR), Kobe, 2009, pp. 315-320. Available: http://ismir2009.ismir.net/proceedings/PS2-21.pdf
# E. Law, K. West, M. Mandel, M. Bay and, J. S. Downie, "Evaluation of Algorithms Using Games: The Case of Music Tagging", in the 10th International Symposium on Music Information Retrieval (ISMIR), Kobe, 2009, pp. 387-392. Available: http://ismir2009.ismir.net/proceedings/OS5-5.pdf
# J. S. Downie, M. Bay, A. F. Ehmann and M. C. Jones, "Audio Cover Song Identification: MIREX 2006-2007 Results and analysis", in the 9th International Conference on Music Information Retrieval (ISMIR 2008), Philadelphia, 2008, pp. 468-473. Available: http://ismir2008.ismir.net/papers/ISMIR2008_265.pdf
# X. Hu, J. S. Downie, C. Laurier, M. Bay and A. F. Ehmann, "The 2007 MIREX Audio Mood Classification Task: Lessons learned", in the 9th International Conference on Music Information Retrieval (ISMIR 2008), Philadelphia, 2008, pp. 462-467. Available: http://ismir2008.ismir.net/papers/ISMIR2008_263.pdf
# J. S. Downie, A. F. Ehmann and J. H. Lee, "The Music Information Retrieval Evaluation eXchange (MIREX): Community-led formal evaluations", in the Digital Humanities 2008 , Oulu Finland, 2008, pp. 239-240.
# M. McVicar and T. De Bie, "Exploiting Online Resources to Improve Chord Recognition Accuracy", late-breaking abstract in the 11th International Conference on Music Information Retrieval (ISMIR 2010), Utrecht, Netherlands, 2010.
# M. McVicar and T. De Bie, "Enhancing chord recognition accuracy using web resources", 3rd ACM Workshop on Machine Learning and Music, October 25, 2010, Firenze, Italy.
# M. McVicar, Y. Ni, R. Santos-Rodriguez, T. De Bie, "Using online chord databases to enhance chord recognition", Journal of New Music Research, Special issue on Music and Machine Learning (in press).
#R. Zhou , M. Mattavelli and G. Zoia, "Music Onset Detection Based on Resonator Time-frequency Image", IEEE Transactions On Audio, Speech And Language Processing, vol. 16, num. 8, 2008, p. 1685-1695.
#Ruohua Zhou, Joshua D. Reiss, Marco Mattavelli and Giorgio Zoia, "A Computationally Efficient Method for Polyphonic Pitch Estimation", EURASIP Journal on Advances in Signal Processing, Volume 2009 (2009), Article ID 729494, 11 pages.
# M. Haro and P. Herrera, “From Low-level to Song-level Percussion Descriptors of Polyphonic Music,” in 10th International Society for Music Information Retrieval Conference (ISMIR 2009), Kobe, Japan, 2009. Available: http://ismir2009.ismir.net/proceedings/PS2-9.pdf
# J. Urbano, J. Morato, M. Marrero and D. Martín, "Crowdsourcing Preference Judgments for Evaluation of Music Similarity Tasks", ACM SIGIR Workshop on Crowdsourcing for Search Evaluation, pp. 9-16, 2010. [http://julian-urbano.info/publications/012-crowdsourcing-preference-judgments-evaluation-music-similarity-tasks/012-paper.pdf PDF]
# J. Urbano, M. Marrero, D. Martín and J. Lloréns, "Improving the Generation of Ground Truths based on Partially Ordered Lists", International Society for Music Information Retrieval Conference, pp. 285-290, 2010. [http://julian-urbano.info/publications/010-improving-generation-ground-truths-based-partially-ordered-lists/010-paper.pdf PDF]
# J. Urbano, J. Lloréns, J. Morato and S. Sánchez-Cuadrado, "Using the Shape of Music to Compute the Similarity between Symbolic Musical Pieces", International Symposium on Computer Music Modeling and Retrieval, pp. 385-396, 2010. [http://julian-urbano.info/publications/009-using-shape-music-compute-similarity-between-symbolic-musical-pieces/009-paper.pdf PDF]

2010:Symbolic Melodic Similarity Results

2010-08-02T09:29:29Z

Jurbano: JU's authors

==Introduction==
These are the results for the 2010 running of the Symbolic Melodic Similarity task set. For background information about this task set please refer to the [[2010:Symbolic Melodic Similarity]] page.

Each system was given a query and returned the 10 most melodically similar songs from those taken from the Essen Collection (5274 pieces in the MIDI format; see [http://www.esac-data.org/ ESAC Data Homepage] for more information). For each query, we made four classes of error-mutations, thus the set comprises the following query classes:

* 0. No errors
* 1. One note deleted
* 2. One note inserted
* 3. One interval enlarged
* 4. One interval compressed

For each query (and its 4 mutations), the returned results (candidates) from all systems were then grouped together (query set) for evaluation by the human graders. The graders were provide with only heard perfect version against which to evaluate the candidates and did not know whether the candidates came from a perfect or mutated query. Each query/candidate set was evaluated by 1 individual grader. Using the Evalutron 6000 system, the graders gave each query/candidate pair two types of scores. Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 100).

====Evalutron 6000 Summary Data====
'''Number of evaluators''' = 6 
'''Number of evaluations per query/candidate pair''' = 1 
'''Number of queries per grader''' = 1 
'''Total number of candidates returned''' = 3900 
'''Total number of unique query/candidate pairs graded''' = 895 
'''Average number of query/candidate pairs evaluated per grader: 149 
'''Number of queries''' = 6 (perfect) with each perfect query error-mutated 4 different ways = 30 

===General Legend===
== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: #efefef;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="540" | Contributors
|-
! HFRA1
| SMS simbals || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/HFRA1.pdf PDF] || [http://simbals.labri.fr Pierre Hanna], [http://simbals.labri.fr Pascal Ferraro], [http://simbals.labri.fr Matthias Robine], [http://simbals.labri.fr Julien Allali]
|-
! JU1
| SMS-Domain || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/JU1.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns], [http://www.kr.inf.uc3m.es Jorge Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]
|-
! JU2
| SMS-PitchDeriv || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/JU2.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns], [http://www.kr.inf.uc3m.es Jorge Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]
|-
! JU3
| SMS-ParamDeriv || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/JU3.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns], [http://www.kr.inf.uc3m.es Jorge Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]
|-
! JU4
| SMS-Shape || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/JU4.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns], [http://www.kr.inf.uc3m.es Jorge Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]
|-
! LL1
| CbrahmsS2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/LL1.pdf PDF] || [http://cs.helsinki.fi Mika Laitinen], [http://cs.helsinki.fi Kjell Lemström]
|-
! LL2
| CbrahmsW2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/LL2.pdf PDF] || [http://cs.helsinki.fi Mika Laitinen], [http://cs.helsinki.fi Kjell Lemström]
|-
! RI1
| UAC || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RI1.pdf PDF] || [http://grfia.dlsi.ua.es/cm/ David Rizo], [http://grfia.dlsi.ua.es/cm José Manuel Iñesta]
|-
! RI2
| UAT || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RI2.pdf PDF] || [http://grfia.dlsi.ua.es/cm/ David Rizo], [http://grfia.dlsi.ua.es/cm José Manuel Iñesta]
|-
! RI3
| UAT3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RI3.pdf PDF] || [http://grfia.dlsi.ua.es/cm/ David Rizo], [http://grfia.dlsi.ua.es/cm José Manuel Iñesta]
|-
! RI4
| UAPR || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RI4.pdf PDF] || [http://grfia.dlsi.ua.es/cm/ David Rizo], [http://grfia.dlsi.ua.es/cm José Manuel Iñesta]
|-
! SU1
| NGR5 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/SU1.pdf PDF] || [http://ade.ishs.net Iman Suyoto], [http://www.cs.rmit.edu.au/~alu Alexandra Uitdenbogerd]
|-
! SU2
| PIOI || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/SU2.pdf PDF] || [http://ade.ishs.net Iman Suyoto], [http://www.cs.rmit.edu.au/~alu Alexandra Uitdenbogerd]
|}

====Broad Categories====
'''NS''' = Not Similar 
'''SS''' = Somewhat Similar 
'''VS''' = Very Similar 

====Table Headings ====
'''ADR''' = Average Dynamic Recall 
'''NRGB''' = Normalize Recall at Group Boundaries 
'''AP''' = Average Precision (non-interpolated) 
'''PND''' = Precision at N Documents 

===Calculating Summary Measures===
'''Fine'''(1) = Sum of fine-grained human similarity decisions (0-100). 
'''PSum'''(1) = Sum of human broad similarity decisions: NS=0, SS=1, VS=2. 
'''WCsum'''(1) = 'World Cup' scoring: NS=0, SS=1, VS=3 (rewards Very Similar). 
'''SDsum'''(1) = 'Stephen Downie' scoring: NS=0, SS=1, VS=4 (strongly rewards Very Similar). 
'''Greater0'''(1) = NS=0, SS=1, VS=1 (binary relevance judgment). 
'''Greater1'''(1) = NS=0, SS=0, VS=1 (binary relevance judgment using only Very Similar). 

(1)Normalized to the range 0 to 1.

==Summary Results==
===Run Times===
<csv>2010/sms/sms_runtimes.csv</csv>
===Overall Scores (Includes Perfect and Error Candidates)===
<csv>2010/sms/Overall.csv</csv>

===Scores by Query Error Types===

<csv>2010/sms/No_Errors.csv</csv>

<csv>2010/sms/Deleted.csv</csv>

<csv>2010/sms/Inserted.csv</csv>

<csv>2010/sms/Enlarged.csv</csv>

<csv>2010/sms/Compressed.csv</csv>

===Friedman Test with Multiple Comparisons Results (p=0.05)===
The Friedman test was run in MATLAB against the Fine summary data over the 30 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
<csv>2010/sms/sum_friedman_fine.csv</csv>
<csv>2007/sms/detail_friedman_fine.csv</csv>

[[Image:2010 sms fine scores friedmans.png]]

==Raw Scores==
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2010:Symbolic Melodic Similarity Raw Data]] page.

[[Category: Results]]

2010:MIREX 2010 Poster List

2010-07-30T12:36:43Z

Jurbano:

==MIREX 2010 Poster Session Planning List==
The MIREX 2010 Poster Session will be held Wednesday, 11 August: 16:00 - 17:45. We will be holding the MIREX plenary meeting 13:00-14:00 as a working lunch on the same day.

Our hosts in Utrecht need to know the number of posters so they can set up the room. Please add you name and the task(s) dealt with in your poster.

We had many groups/individuals submit across tasks. You can choose to create one ISMIR poster bringing all your data together or can split up your data across, say, two or three posters. If you have questions, please contact me at jdownie@illinois.edu or the MIREX mailing list about task poster options.

As a reminder, the MIREX posters need to follow the [http://ismir2010.ismir.net/information-for-authors/information-for-presenters/ ISMIR 2010 poster guidelines] (i.e., A0, portrait orientation).

==Add your author names here, once for each poster along with "title of some sort" and (Task(s) covered)==
# IMIRSEL: ''MIREX 2010 Overview, Part I'' (Train Test Tasks)
# IMIRSEL: ''MIREX 2010 Overview, Part II'' (All Other Tasks)
# Andreas Arzt and Gerhard Widmer: "Real-time Music Tracking using Tempo-aware On-line Dynamic Time Warping" (Real-time Audio to Score Alignment (a.k.a Score Following))
# Pasi Saari and Olivier Lartillot: "SubEnsemble - Classification framework based on the Ensemble Approach and Feature Selection" (Train Test Tasks)
# Gabriel Sargent, Frédéric Bimbot and Emmanuel Vincent: "Structural segmentation of songs using multi-criteria generalized likelihood ratio and regularity constraints" (Structural Segmentation Task)
# Emmanouil Benetos and Simon Dixon: "Multiple fundamental frequency estimation using spectral structure and temporal evolution rules" (Multiple Fundamental Frequency Estimation & Tracking Task)
# J. Urbano, J. Lloréns, J. Morato and S. Sánchez-Cuadrado: ''Local Alignment with Geometric Representations'' (Symbolic Melodic Similarity)

==Below are some examples from MIREX 2009==

# Matt Hoffman: ''Using CBA to Automatically Tag Songs'' (Audio tag classification/retrieval)
# Suman Ravuri, Dan Ellis: ''The Hydra System of Cover Song Classification'' (Cover Song Identification)
# Joan Serra, Massimiliano Zanin, Ralph G Andrzejak: ''Cover song retrieval by recurrence quantification and unsupervised set detection'' (Cover Song Identification)
# MTG Team: "Music Type Groupers (MTG): Generic Music Classification Algorithms" (Audio Genre Classification, Mood Classification, Artist Identification, Classical Composer Identification)
# R. Jang: "Poster #2" (placeholder to get the auto-counter to increment)
# R. Jang: "Poster #3" (placeholder to get the auto-counter to increment)

2010:MIREX 2010 Poster List

2010-07-30T01:01:17Z

Jurbano:

==MIREX 2010 Poster Session Planning List==
The MIREX 2010 Poster Session will be held Wednesday, 11 August: 16:00 - 17:45. We will be holding the MIREX plenary meeting 13:00-14:00 as a working lunch on the same day.

Our hosts in Utrecht need to know the number of posters so they can set up the room. Please add you name and the task(s) dealt with in your poster.

We had many groups/individuals submit across tasks. You can choose to create one ISMIR poster bringing all your data together or can split up your data across, say, two or three posters. If you have questions, please contact me at jdownie@illinois.edu or the MIREX mailing list about task poster options.

As a reminder, the MIREX posters need to follow the [http://ismir2010.ismir.net/information-for-authors/information-for-presenters/ ISMIR 2010 poster guidelines] (i.e., A0, portrait orientation).

==Add your author names here, once for each poster along with "title of some sort" and (Task(s) covered)==
# IMIRSEL: ''MIREX 2010 Overview, Part I'' (Train Test Tasks)
# IMIRSEL: ''MIREX 2010 Overview, Part II'' (All Other Tasks)

==Below are some examples from MIREX 2009==

# Matt Hoffman: ''Using CBA to Automatically Tag Songs'' (Audio tag classification/retrieval)
# Suman Ravuri, Dan Ellis: ''The Hydra System of Cover Song Classification'' (Cover Song Identification)
# Joan Serra, Massimiliano Zanin, Ralph G Andrzejak: ''Cover song retrieval by recurrence quantification and unsupervised set detection'' (Cover Song Identification)
# MTG Team: "Music Type Groupers (MTG): Generic Music Classification Algorithms" (Audio Genre Classification, Mood Classification, Artist Identification, Classical Composer Identification)
# R. Jang: "Poster #2" (placeholder to get the auto-counter to increment)
# R. Jang: "Poster #3" (placeholder to get the auto-counter to increment)
# J. Urbano, J. Lloréns, J. Morato and S. Sánchez-Cuadrado: ''Local Alignment with Geometric Representations'' (Symbolic Melodic Similarity)

2010:MIREX 2010 Poster List

2010-07-30T00:57:30Z

Jurbano:

==MIREX 2010 Poster Session Planning List==
The MIREX 2010 Poster Session will be held Wednesday, 11 August: 16:00 - 17:45. We will be holding the MIREX plenary meeting 13:00-14:00 as a working lunch on the same day.

Our hosts in Utrecht need to know the number of posters so they can set up the room. Please add you name and the task(s) dealt with in your poster.

We had many groups/individuals submit across tasks. You can choose to create one ISMIR poster bringing all your data together or can split up your data across, say, two or three posters. If you have questions, please contact me at jdownie@illinois.edu or the MIREX mailing list about task poster options.

As a reminder, the MIREX posters need to follow the [http://ismir2010.ismir.net/information-for-authors/information-for-presenters/ ISMIR 2010 poster guidelines] (i.e., A0, portrait orientation).

==Add your author names here, once for each poster along with "title of some sort" and (Task(s) covered)==
# IMIRSEL: ''MIREX 2010 Overview, Part I'' (Train Test Tasks)
# IMIRSEL: ''MIREX 2010 Overview, Part II'' (All Other Tasks)

==Below are some examples from MIREX 2009==

# Matt Hoffman: ''Using CBA to Automatically Tag Songs'' (Audio tag classification/retrieval)
# Suman Ravuri, Dan Ellis: ''The Hydra System of Cover Song Classification'' (Cover Song Identification)
# Joan Serra, Massimiliano Zanin, Ralph G Andrzejak: ''Cover song retrieval by recurrence quantification and unsupervised set detection'' (Cover Song Identification)mbolic
# MTG Team: "Music Type Groupers (MTG): Generic Music Classification Algorithms" (Audio Genre Classification, Mood Classification, Artist Identification, Classical Composer Identification)
# R. Jang: "Poster #2" (placeholder to get the auto-counter to increment)
# R. Jang: "Poster #3" (placeholder to get the auto-counter to increment)
# J. Urbano, J. Lloréns, J. Morato and S. Sánchez-Cuadrado: ''Local Alignment with Geometric Representations'' (Symbolic Melodic Similarity)

2010:Audio Similarity 2010 Graders

2010-07-17T15:04:13Z

Jurbano:

=AMS 2010 Graders=

Welcome to the AMS grader sign-up page. Please give us your name and email contact information. If you obscure your email, please make it relatively obvious to us how to parse the address.

Template: Name. Location. <Email>

Sample: J. Stephen Downie. Illinois, USA. <jdownie@illinois.edu>

=Special Comments=
We are under extraordinary time constraints this year because ISMIR 2010
is being held starting 9 August and we need to have all the final
results calculated and posted by our 2 August target date (fingers crossed).

We hope to open the the Evalutron 6000 (E6K) v.2 grading system by 20
July. To meet our 2 August goal, we must have all the AMS and SMS
similarity grades entered into the E6K by 27 July. YES, THIS GIVES US
ONLY ONE WEEK! So, if you are kind enough to sign up to be a grader,
please understand that we really need you complete your assigned grading
by 27 July.

If you are a SMS or AMS participant, we ask that you do what you can to
encourage adults over 18 years of age to be graders.

We are looking for 50 graders for AMS this year. Each grader will be responsible for two query lists. This year the query lists seem to be moderate in length so tackling two queries lists should not be too onerous. For safety's sake, we would like to see say, an extra five or so names on the sign up sheet below. The "extra names" on the sign up sheet will be considered "back up" graders. We will assign grading tasks in the order of the names as they appear below.

=Sign Up Area=
#Martin Ariel Hartmann. Buenos Aires, Argentina. <martin.hartmann@jyu.fi>
#Sally Jo Cunningham. Hamilton, New Zealand. <sallyjo@cs.waikato.ac.nz>
#Matt Hoffman. New York, USA. <mdhoffma@cs.princeton.edu>
#Thierry Bertin-Mahieux. New York, USA. <tb2332@columbia.edu>
#Dominik Schnitzer, Vienna, Austria <dominik.schnitzer (dumdidum) ofai.at>
#Ming Li, Beijing, China, <liming . ioa [AT] gmail dot com>
#Charlie Inskip, London UK <c.inskip@city.ac.uk>
#Ana M. Barbancho, Malaga, Spain < abp [AT] ic dot uma dot es>
#Arthur Flexer, Vienna, Austria <arthur.flexer [AT] ofai.at>
#Simone Sammartino, Málaga, Spain <ssammartino@ic.uma.es>
#Cristina de la Bandera Cascales, Málaga, Spain <cdelabandera@ic.uma.es>
#Peter Knees, Linz, Austria <peter.knees (AT) jku.at>
#Steve Tjoa. Maryland, USA. <kiemyang at umd edu>
#Michael Mandel, Montreal, QC <mandelm at iro umontreal ca>
#Perfecto Herrera, Barcelona, Spain <perfecto dot herrera [AT] upf dot es>
#Aggelos Gkiokas, Athens, Greece <agkiokas@ilsp.gr>
#Dmitry Bogdanov, Barcelona, Spain <dmitry dot bogdanov [AT] upf dot es>
#Ewa Lukasik, Poznan, Poland <ewa.lukasik@cs.put.poznan.pl>
#Tim Pohle, Linz, Austria <tim dot pohle at jku.at >
#Tom Collins. Milton Keynes, UK. <t dot e dot collins at open dot ac dot uk>
#Julián Urbano, Madrid, Spain < jurbano at inf dot uc3m dot es >