2012:Audio Music Similarity and Retrieval Results - Revision history

Kahyun Choi: /* Friedman's Test (BROAD Scores) */

2013-08-20T01:40:35Z

‎Friedman's Test (BROAD Scores)

Kahyun Choi: /* Friedman's Test (FINE Scores) */

2013-08-20T01:40:23Z

‎Friedman's Test (FINE Scores)

Kahyun Choi: /* Overall Summary Results */

2013-08-20T01:40:09Z

‎Overall Summary Results

Kahyun Choi: /* Introduction */

2013-08-20T01:39:57Z

‎Introduction

AndreasEhmann: /* Reports */

2012-10-04T21:13:22Z

‎Reports

AndreasEhmann: /* Team ID */

2012-10-04T21:12:26Z

‎Team ID

Kahyun Choi: /* Introduction */

2012-10-04T21:02:14Z

‎Introduction

Kahyun Choi: /* Team ID */

2012-10-04T21:01:08Z

‎Team ID

Kahyun Choi: /* Team ID */

2012-10-04T20:58:58Z

‎Team ID

Kahyun Choi: /* Reports */

2012-10-04T20:57:43Z

‎Reports

@@ Line 74: / Line 74: @@
 ====Friedman's Test (BROAD Scores)====
-The Friedman test was run in MATLAB against the '''BROAD''' summary data over the 100 queries.<br />
+The Friedman test was run in MATLAB against the '''BROAD''' summary data over the 50 queries.<br />
 Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

@@ Line 66: / Line 66: @@
 ===Friedman's Tests===
 ====Friedman's Test (FINE Scores)====
-The Friedman test was run in MATLAB against the '''Fine''' summary data over the 100 queries.<br />
+The Friedman test was run in MATLAB against the '''Fine''' summary data over the 50 queries.<br />
 Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

← Older revision		Revision as of 01:40, 20 August 2013
Line 63:		Line 63:

	<csv p=3>2012/ams/summary_evalutron.csv</csv>		<csv p=3>2012/ams/summary_evalutron.csv</csv>
−
−	~~<br />~~
−	~~'''Note:RZ1''' is the random result for comparing purpose.~~

	===Friedman's Tests===		===Friedman's Tests===

@@ Line 2: / Line 2: @@
 These are the results for the 2012 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.
-Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 100 songs were randomly selected from the 10 genre groups (10 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical '''BROAD''' score with 3 categories: NS,SS,VS as explained below, and one '''FINE''' score (in the range from 0 to 100). A description and analysis is provided below.
+Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 50 songs were randomly selected from the 10 genre groups (5 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical '''BROAD''' score with 3 categories: NS,SS,VS as explained below, and one '''FINE''' score (in the range from 0 to 100). A description and analysis is provided below.
 The systems read in 30 second audio clips as their raw data. The same 30 second clips were used in the grading stage.

@@ Line 116: / Line 116: @@
 '''RW4''' = [https://music-ir.org/mirex/results/2012/ams/statistics/RW4/report.txt Jia-Min Ren,Ming-Ju Wu,Jyh-Shing Roger Jang]<br />
 '''SSKP1''' = [https://music-ir.org/mirex/results/2012/ams/statistics/SSKP1/report.txt Klaus Seyerlehner, Markus Schedl, Peter Knees, Tim Pohle]<br />
-'''SSKP2''' = [https://music-ir.org/mirex/results/2012/ams/statistics/SSKS2/report.txt Klaus Seyerlehner, Markus Schedl, Peter Knees, Tim Pohle]<br />
+'''SSKP2''' = [https://music-ir.org/mirex/results/2012/ams/statistics/SSKS2/report.txt Klaus Seyerlehner, Markus Schedl, Peter Knees, Reinhard Sonnleitner]<br />

@@ Line 46: / Line 46: @@
          |-
 ! SSKS2
-	| cbmr_sim_2011 ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SSKS2.pdf PDF] || [http://www.seyerlehner.info Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Peter Knees], [http://www.cp.jku.at/ Tim Pohle]
+	| cbmr_sim_2011 ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SSKS2.pdf PDF] || [http://www.seyerlehner.info Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Peter Knees], [http://www.cp.jku.at/ Reinhard Sonnleitner]
          |-
 	|}

@@ Line 1: / Line 1: @@
 == Introduction ==
-These are the results for the 2011 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.
+These are the results for the 2012 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.
 Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 100 songs were randomly selected from the 10 genre groups (10 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical '''BROAD''' score with 3 categories: NS,SS,VS as explained below, and one '''FINE''' score (in the range from 0 to 100). A description and analysis is provided below.

@@ Line 38: / Line 38: @@
          ! PS1
 	| PS09 ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/PS1.pdf PDF] || [http://www.ofai.at/~dominik.schnitzer Dominik Schnitzer], [http://www.cp.jku.at/ Tim Pohle]
          |-
          ! RW4

@@ Line 17: / Line 17: @@
 	! width="80" style="text-align: center;" | Abstract
 	! width="440" | Contributors
 	|-
          ! DM6
 	| DM6 ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/DM6.pdf PDF] || [http://www.iam.ecs.soton.ac.uk/ Franz	de Leon], [http://www.iam.ecs.soton.ac.uk/ Kirk	Martinez]
          |-
          ! GT3
 	| MarsyasSimilarity ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/GT3.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis]
@@ Line 44: / Line 38: @@
          ! PS1
 	| PS09 ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/PS1.pdf PDF] || [http://www.ofai.at/~dominik.schnitzer Dominik Schnitzer], [http://www.cp.jku.at/ Tim Pohle]
          |-
          ! PS1
@@ Line 58: / Line 48: @@
 	| cbmr_sim_2010 ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SSPK1.pdf PDF] || [http://www.seyerlehner.info Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Peter Knees], [http://www.cp.jku.at/ Tim Pohle]
          |-
-! SSKP2
+! SSKS2
-	| cbmr_sim_2011 ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SSPK2.pdf PDF] || [http://www.seyerlehner.info Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Peter Knees], [http://www.cp.jku.at/ Tim Pohle]
+	| cbmr_sim_2011 ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SSKS2.pdf PDF] || [http://www.seyerlehner.info Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Peter Knees], [http://www.cp.jku.at/ Tim Pohle]
          |-
 	|}

@@ Line 120: / Line 120: @@
 === Reports ===
 '''DM6''' = [https://music-ir.org/mirex/results/2012/ams/statistics/DM6/report.txt Franz de Leon, Kirk Martinez]<br />
 '''GT3''' = [https://music-ir.org/mirex/results/2012/ams/statistics/GT3/report.txt George Tzanetakis]<br />
 '''JR2''' = [https://music-ir.org/mirex/results/2012/ams/statistics/JR2/report.txt Jia-Min Ren, Jyh-Shing Roger Jang]<br />
 '''NHHL1''' = [https://music-ir.org/mirex/results/2012/ams/statistics/NHHL1/report.txt Byeong-jun Han, Kyogu Lee,Juhan Nam,Jorge Herrera]<br />
 '''NHHL2''' = [https://music-ir.org/mirex/results/2012/ams/statistics/NHHL2/report.txt Byeong-jun Han, Kyogu Lee,Juhan Nam,Jorge Herrera]<br />
 '''PS1''' = [https://music-ir.org/mirex/results/2012/ams/statistics/PS1/report.txt Dominik Schnitzer, Tim Pohle]<br />
 '''RW4''' = [https://music-ir.org/mirex/results/2012/ams/statistics/RW4/report.txt Jia-Min Ren,Ming-Ju Wu,Jyh-Shing Roger Jang]<br />
 '''SSKP1''' = [https://music-ir.org/mirex/results/2012/ams/statistics/SSKP1/report.txt Klaus Seyerlehner, Markus Schedl, Peter Knees, Tim Pohle]<br />
 '''SSKP2''' = [https://music-ir.org/mirex/results/2012/ams/statistics/SSKS2/report.txt Klaus Seyerlehner, Markus Schedl, Peter Knees, Tim Pohle]<br />