2006:Evalutron6000 Issues - Revision history

Xiao: /* Official Description of the Evalutron 6000 System and Protocol */

2010-06-03T16:50:26Z

‎Official Description of the Evalutron 6000 System and Protocol

IMIRSELBot: Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

2010-05-13T18:12:07Z

Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

IMIRSELBot: Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

2010-05-13T18:04:45Z

Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2006:\1)

Tomlidy: /* Potential Audio Simularity "Graders" (MIREX 2006) */

2006-09-05T16:34:07Z

‎Potential Audio Simularity "Graders" (MIREX 2006)

Admin at 17:19, 4 September 2006

2006-09-04T17:19:55Z

Admin at 17:19, 4 September 2006

2006-09-04T17:19:30Z

Matic at 06:15, 4 September 2006

2006-09-04T06:15:45Z

Dswilliamson: /* Informed Consent Text Template */

2006-09-01T17:58:56Z

‎Informed Consent Text Template

Dswilliamson: /* Potential Audio Simularity "Graders" (MIREX 2006) */

2006-09-01T17:50:14Z

‎Potential Audio Simularity "Graders" (MIREX 2006)

Youngmoo: /* Potential Audio Simularity "Graders" (MIREX 2006) */

2006-09-01T16:29:24Z

‎Potential Audio Simularity "Graders" (MIREX 2006)

@@ Line 17: / Line 17: @@
 The research protocol for this protocol is IRB# 07066.
-'''Task Description:''' For the ΓÇ£similarity-basedΓÇ¥ class of MIR/MDL tasks, the submitted music retrieval systems are being tested whether (or not) they can find one or more songs ΓÇ£similarΓÇ¥ to the standardized ΓÇ£queryΓÇ¥ songs that were selected by IMIRSEL. For each ΓÇ£query song,ΓÇ¥ each retrieval system under evaluation gives IMIRSEL a list of songs that it ΓÇ£thinksΓÇ¥ are similar to the ΓÇ£query songΓÇ¥. These songs are called ΓÇ£candidate songsΓÇ¥ (or simply, ΓÇ£candidatesΓÇ¥). IMIRSEL gathers up these lists of candidates and then makes a ΓÇ£master listΓÇ¥ of all the candidates for each query song. At this point, we then turn to volunteers drawn from the MIR/MDL research community to act as similarity ΓÇ£graders.ΓÇ¥ The gradersΓÇÖ job is to determine whether each query/candidate pair is similar (or not). Once we have collected and aggregated the similarity judgments for each query/candidate pair, we can then go back and determine, for each system, how many similar (and not similar) songs were retrieved for each query. Once the aggregated ground-truth data has been establish, the evaluation procedures concerning the success (or failures) of the systems becomes automatic (i.e., no more human involvement) as the scoring of each system is done algorithmically by a series of computer programmes.
+'''Task Description:''' For the "similarity-based" class of MIR/MDL tasks, the submitted music retrieval systems are being tested whether (or not) they can find one or more songs "similar" to the standardized "query" songs that were selected by IMIRSEL. For each "query song," each retrieval system under evaluation gives IMIRSEL a list of songs that it "thinks" are similar to the "query song". These songs are called "candidate songs" (or simply, "candidates"). IMIRSEL gathers up these lists of candidates and then makes a "master list" of all the candidates for each query song. At this point, we then turn to volunteers drawn from the MIR/MDL research community to act as similarity "graders." The graders' job is to determine whether each query/candidate pair is similar (or not). Once we have collected and aggregated the similarity judgments for each query/candidate pair, we can then go back and determine, for each system, how many similar (and not similar) songs were retrieved for each query. Once the aggregated ground-truth data has been establish, the evaluation procedures concerning the success (or failures) of the systems becomes automatic (i.e., no more human involvement) as the scoring of each system is done algorithmically by a series of computer programmes.
@@ Line 30: / Line 30: @@
 **1.b.i.	Data security and integrity
-**1.b.ii.	Affording the graders the ability to ΓÇ£come and goΓÇ¥ from the system as they wish because the system uses the username to record which query/candidate pairs have already been evaluated by each user. This also allows graders to modify earlier judgments if they want.
+**1.b.ii.	Affording the graders the ability to "come and go" from the system as they wish because the system uses the username to record which query/candidate pairs have already been evaluated by each user. This also allows graders to modify earlier judgments if they want.
 **1.b.iii.	Affording us the ability to delete specific grader scores should a grader decide to withdraw or a security breach is detected, etc.
@@ Line 52: / Line 52: @@
 **2.b.i.	The ability to hear, stop, start, rewind, etc. each candidate song
-**2.b.ii.	An input mechanism to indicate and record the graderΓÇÖs similarity judgment for each candidate
+**2.b.ii.	An input mechanism to indicate and record the grader's similarity judgment for each candidate
 .	Use a robust, password-protected, PHP/MySql database system to:
@@ Line 71: / Line 71: @@
 '''Date and Duration Issues:''' This protocol is designed to generate similarity ground-truth data for the evaluation of MIR/MDL systems under the conditions that pertain to the MIREX evaluation tasks. These tasks are ever-evolving as new test collections are generated and new algorithms are developed. Because of this constant evolution, every so often, at unpredictable intervals, IMIRSEL will need to collect new ground-truth data sets under this protocol. Thus, in general, this protocol has no fixed end date.
-'''Measures:''' Currently a query/candidate song pair is deemed to be similar by simple majority vote of the graders (e.g., if  two out of three graders say the pair is similar then we will deem that pair to be similar). Future evaluation runs might be set up to include the ability to gather ΓÇ£relativeΓÇ¥ similarity scores (i.e., on a scale from, say, 1 to 10, etc.) and/or ΓÇ£comparativeΓÇ¥ scores (i.e., song A is more/less similar to the query than song B, etc.). None of the aforementioned possible scoring variants materially affects the underlying purpose of the protocol under consideration which is, again, the establishment of similarity values between query/candidate pairs.
+'''Measures:''' Currently a query/candidate song pair is deemed to be similar by simple majority vote of the graders (e.g., if  two out of three graders say the pair is similar then we will deem that pair to be similar). Future evaluation runs might be set up to include the ability to gather "relative" similarity scores (i.e., on a scale from, say, 1 to 10, etc.) and/or "comparative" scores (i.e., song A is more/less similar to the query than song B, etc.). None of the aforementioned possible scoring variants materially affects the underlying purpose of the protocol under consideration which is, again, the establishment of similarity values between query/candidate pairs.
 '''Derivative aggregate data:''' (i.e., not specific to any individual grader) concerning the data collection process will be analyzed for possible system improvements and generalizations. Such aggregate data would include, for example, descriptive measures of number of pairs judged, ratios of similar to not similar judgments, estimates of judgment times, etc.
-'''About Graders and Data Integrity:''' The volunteer graders will be drawn from the MIR/MDL researcher community and NOT the general public. We want to use only those graders that have a vested interest in making honest similarity assessments. MIR/MDL researchers have this vested interest in honesty as the creation of valid ground-truth data and the subsequent evaluation of their MIR/MDL systems based upon valid similarity ground-truth provide for them a scientifically valid basis for future research and development. To further ensure data integrity, the protocol is set up to be '''ΓÇ£double-blindΓÇ¥''' in the same manner that peer-reviewing is double-blind. In a very real sense, ''<b>this protocol is peer-reviewing</b>''. Because IMIRSEL runs the submitted systems in-house against music collections unknown to the submitters, '''AND''' our protocol aggregates lists of query/candidate pairs from across all submissions, none of the graders knows from which system a particular query/candidate comes (i.e., ''it could even from their own system and they would have no way of knowing!''). The submitters of the systems under evaluation also have no way of knowing which graders provided similarity judgments for their systems as all the grading information is stripped of grader identifiers and then aggregated into a collective set of similarity values.
+'''About Graders and Data Integrity:''' The volunteer graders will be drawn from the MIR/MDL researcher community and NOT the general public. We want to use only those graders that have a vested interest in making honest similarity assessments. MIR/MDL researchers have this vested interest in honesty as the creation of valid ground-truth data and the subsequent evaluation of their MIR/MDL systems based upon valid similarity ground-truth provide for them a scientifically valid basis for future research and development. To further ensure data integrity, the protocol is set up to be '''"double-blind"''' in the same manner that peer-reviewing is double-blind. In a very real sense, ''<b>this protocol is peer-reviewing</b>''. Because IMIRSEL runs the submitted systems in-house against music collections unknown to the submitters, '''AND''' our protocol aggregates lists of query/candidate pairs from across all submissions, none of the graders knows from which system a particular query/candidate comes (i.e., ''it could even from their own system and they would have no way of knowing!''). The submitters of the systems under evaluation also have no way of knowing which graders provided similarity judgments for their systems as all the grading information is stripped of grader identifiers and then aggregated into a collective set of similarity values.
 '''Informed Consent Issues:'''

@@ Line 1: / Line 1: @@
 =Introduction=
-This page is intended to provide a detailed explication of the Evalutron 6000 setup for the [[2006:Audio Music Similarity and Retrieval]] and [[Symbolic Melodic Similarity]] tasks.
+This page is intended to provide a detailed explication of the Evalutron 6000 setup for the [[2006:Audio Music Similarity and Retrieval]] and [[2006:Symbolic Melodic Similarity]] tasks.
 Please read it through carefully.
@@ Line 7: / Line 7: @@
 #[[2006:Evalutron6000_Walkthrough_For_Symbolic_Melodic_Similarity]]
-#[[Evalutron6000_Walkthrough_For_Audio_Music_Similarity_and_Retrieval]]
+#[[2006:Evalutron6000_Walkthrough_For_Audio_Music_Similarity_and_Retrieval]]

@@ Line 1: / Line 1: @@
 =Introduction=
-This page is intended to provide a detailed explication of the Evalutron 6000 setup for the [[Audio Music Similarity and Retrieval]] and [[Symbolic Melodic Similarity]] tasks.
+This page is intended to provide a detailed explication of the Evalutron 6000 setup for the [[2006:Audio Music Similarity and Retrieval]] and [[Symbolic Melodic Similarity]] tasks.
 Please read it through carefully.
@@ Line 6: / Line 6: @@
 Also, please carefully read through the two task-specific Evalutron Walkthroughs:
-#[[Evalutron6000_Walkthrough_For_Symbolic_Melodic_Similarity]]
+#[[2006:Evalutron6000_Walkthrough_For_Symbolic_Melodic_Similarity]]
 #[[Evalutron6000_Walkthrough_For_Audio_Music_Similarity_and_Retrieval]]
@@ Line 20: / Line 20: @@
-'''Location:''' IMIRSEL has developed a web-based similarity judgment collection system currently called the Evalutron 6000. The Evalutron 6000 is located on the servers of the Graduate School of Library and Information Science, UIUC. A "walkthrough" with screen shots of the system prototype  located at the [[Evalutron6000 Walkthrough]] page. The Evalutron 6000 is set up to:
+'''Location:''' IMIRSEL has developed a web-based similarity judgment collection system currently called the Evalutron 6000. The Evalutron 6000 is located on the servers of the Graduate School of Library and Information Science, UIUC. A "walkthrough" with screen shots of the system prototype  located at the [[2006:Evalutron6000 Walkthrough]] page. The Evalutron 6000 is set up to:

@@ Line 128: / Line 128: @@
 #Audrey Laplante, McGill
 #Wietse Balkema
 #Robert Neumayer, TU Vienna
 #Stephen Cox, UEA

@@ Line 6: / Line 6: @@
 Also, please carefully read through the two task-specific Evalutron Walkthroughs:
-#[Evalutron6000_Walkthrough_For_Symbolic_Melodic_Similarity]
+#[[Evalutron6000_Walkthrough_For_Symbolic_Melodic_Similarity]]
-#[Evalutron6000_Walkthrough_For_Audio_Music_Similarity_and_Retrieval]
+#[[Evalutron6000_Walkthrough_For_Audio_Music_Similarity_and_Retrieval]]

@@ Line 4: / Line 4: @@
 Please read it through carefully.
-Also, please carefully read through the [[Evalutron6000 Walkthrough]].
+Also, please carefully read through the two task-specific Evalutron Walkthroughs:
 =Official Description of the Evalutron 6000 System and Protocol=

@@ Line 134: / Line 134: @@
 #Youngmoo Kim, Drexel
 #Donald S. Williamson, Drexel
 =Symbolic Melodic Simularity Assumptions (MIREX 2006)=

@@ Line 216: / Line 216: @@
 {Choose one}<br>
-( ) Yes, I Agree<br>
+(x ) Yes, I Agree<br>
 ( ) No, I Disagree<br>

@@ Line 132: / Line 132: @@
 #Elias Pampalk
 #Tim Pohle, Johannes Kepler Univ. Linz
 =Symbolic Melodic Simularity Assumptions (MIREX 2006)=