2006:Evalutron6000 Walkthrough For Symbolic Melodic Similarity - Revision history

IMIRSELBot: Bot: Fixing image links

2010-05-14T03:34:00Z

Bot: Fixing image links

IMIRSELBot: Robot: Automated text replacement (-\[\[Image:\s*([A-Z]) +[[Image:2006_\1)

2010-05-14T03:29:07Z

Robot: Automated text replacement (-\[\[Image:\s*([A-Z]) +[[Image:2006_\1)

Admin: /* Grading Expectations and "Reasonableness" */

2006-09-08T13:23:50Z

‎Grading Expectations and "Reasonableness"

Admin at 17:16, 4 September 2006

2006-09-04T17:16:46Z

Admin: /* Evaluation Pages: Queries and Candidate Lists */

2006-09-04T16:35:48Z

‎Evaluation Pages: Queries and Candidate Lists

Admin: /* Task Description: Focusing on Melody */

2006-09-04T16:34:42Z

‎Task Description: Focusing on Melody

Admin: /* Evaluation Pages: Queries and Candidate Lists */

2006-09-04T16:31:36Z

‎Evaluation Pages: Queries and Candidate Lists

Mertbay: /* Four Important Issues Before Proceeding */

2006-09-01T23:08:49Z

‎Four Important Issues Before Proceeding

CameronJones: /* Registration */

2006-09-01T23:06:49Z

‎Registration

Mertbay: /* Four Important Issues Before Proceeding */

2006-09-01T23:06:06Z

‎Four Important Issues Before Proceeding

@@ Line 37: / Line 37: @@
 When first visiting the Evalutron 6000 homepage, you will see a page similar to this (Fig. 1).
-[[Image:2006_E6ksms_home_page_scaled.png]]
+[[Image:2006_e6ksms_home_page_scaled.png]]
 '''Figure 1. Evalutron 6000 start page.'''
@@ Line 55: / Line 55: @@
 If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.
-[[Image:2006_E6ksms_registration_page_scaled.png]]
+[[Image:2006_e6ksms_registration_page_scaled.png]]
 '''Figure 2. Evalutron 6000 registration page.'''
@@ Line 62: / Line 62: @@
 After completing the Registration, the system will ask you to sign-in with your newly created username and password. After signing-in, you can begin evaluation (Fig. 3). To start the evaluation process, click on the "Start Evaluation" link on the homepage. You will be presented with several options for which media player you want to use. We have invested considerable effort coding various interfaces to be compatible with the maximum number of browsers, platforms, and players. Most users should be able to use the Flash MP3 Player option -- which we've found to work the best. The Windows Media Player and QuickTime options should also work in most modern browsers. We have noted that the Media Player does introduce some start-up lags when starting a new query set which we believe is caused by the way Media Player downloads the MP3 files. We have also noted that the combination of Internet Explorer and QuickTime is simply not working.
-[[Image:2006_E6ksms_options_page_scaled.png]]
+[[Image:2006_e6ksms_options_page_scaled.png]]
 '''Figure 3. Audio player selection page.'''
@@ Line 69: / Line 69: @@
 Once you have selected a player configuration, you'll see a query evaluation page (Fig. 4). On this page, you will see the query player and the list of candidate players. The queries and candidates drawn from the RISM collection (Queries #1 through #6) are fairly short - most candidates are about 4-5 seconds long. '''''Nota Bene''''' (The following description of the Mixed and Karaoke candidates has been updated, 3 September 2006). The length of candidates for the polyphonic task, drawn from the "Mixed" and "Karaoke" collections (Queries #7 through #17) is now more consistently in the 30 second range. We discovered that the timing information returned was not a reliable indicator of where the "best match" was found in each of the candidates. To correct for this, we took the "best match" starting information and then "padded" the candiate on both sides of the "best match" start point. Thus, the first 10 or so seconds of each candidate might, or might not, represent a strong "hit". The last 10 or so might also start "wandering away" from the "best match" region. Therfore, we recommend that you listen to the entire candidate before giving your grade and to be mindful that the "best match" section in each candidate is a bit slippery.
-[[Image:2006_E6ksms_evaluation_page_scaled.png]]
+[[Image:2006_e6ksms_evaluation_page_scaled.png]]
 '''Figure 4. Sample evaluation page.'''
@@ Line 79: / Line 79: @@
 The procedure for listening to a candidate is the same as listening to a query -- click the "Play Candidate" buttons to load the clip into the player and listen to it (Fig. 4). Once you have a feeling for how similar the candidate is to the query, click the "Not Similar," "Somewhat Similar" or "Very Similar" radio buttons to the right of the candidate. Note: you can continue to replay the query during candidate playback.
-[[Image:2006_E6ksms_select_category.png]]
+[[Image:2006_e6ksms_select_category.png]]
 '''Figure 5. Close up image of Broad Category selection buttons.'''
@@ Line 85: / Line 85: @@
 Depending on how you graded the candidate, you should see the candidate box react to your grade, indicating that the vote has been logged in the database (Fig. 6). If you indicated the song was "Somewhat Similar" the box will turn yellow and state "SAVED". If you indicated the song was "Very Similar" the box will turn green, "Not Similar" the box will turn red, and state "SAVED".
-[[Image:2006_E6ksms_select_category_saved.png]]
+[[Image:2006_e6ksms_select_category_saved.png]]
 '''Figure 6. Close up image of Broad Category selection buttons with "Somewhat Similar" selected and "SAVED" automatically.'''
@@ Line 91: / Line 91: @@
 We also need each grader to assign a fine-grained score for the similarity of the candidate to the query on a scale of 0-10 (Fig. 7). More information about this process below.
-[[Image:2006_E6ksms_select_score.png]]
+[[Image:2006_e6ksms_select_score.png]]
 '''Figure 7. Close up image of the Fine Score selection scale.'''
@@ Line 97: / Line 97: @@
 The system will automatically record the score when you let go of the scaler (Fig. 8).
-[[Image:2006_E6ksms_select_score_saved.png]]
+[[Image:2006_e6ksms_select_score_saved.png]]
 '''Figure 8. Close up image of a Fine Score selection scale with a random score "SAVED".'''
@@ Line 103: / Line 103: @@
 You can also manually enter the score in the box, but in this case you MUST click the button labeled "Click to Save" to record your score (Fig. 9). After the score is saved, the scaler will automatically move to the correct position and the label on the button will change to "Saved" (Fig. 8).
-[[Image:2006_E6ksms_select_score_before_save.PNG]]
+[[Image:2006_e6ksms_select_score_before_save.png]]
 '''Figure 9. Close up image of a Fine Score selection scale before "SAVED".'''
@@ Line 112: / Line 112: @@
-[[Image:2006_E6ksms_my_assignment_scaled.png]]
+[[Image:2006_e6ksms_my_assignment_scaled.png]]
 '''Figure 10. My Assignment page. '''
@@ Line 118: / Line 118: @@
 You can also see a list of all of the queries you have been assigned at the bottom of each Candidate List page. You can return to any query by clicking on the button for that query here (Fig. 11). You can re-evaluate any candidate for any query at any time, up to the closing of the evaluation system.
-[[Image:2006_Eval6_queries_detail.png]]
+[[Image:2006_eval6_queries_detail.png]]
 '''Figure 11. Close up image of Query List buttons which allow for revisiting of completed query/candidate sets. '''

@@ Line 37: / Line 37: @@
 When first visiting the Evalutron 6000 homepage, you will see a page similar to this (Fig. 1).
-[[Image:E6ksms_home_page_scaled.png]]
+[[Image:2006_E6ksms_home_page_scaled.png]]
 '''Figure 1. Evalutron 6000 start page.'''
@@ Line 55: / Line 55: @@
 If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.
-[[Image:E6ksms_registration_page_scaled.png]]
+[[Image:2006_E6ksms_registration_page_scaled.png]]
 '''Figure 2. Evalutron 6000 registration page.'''
@@ Line 62: / Line 62: @@
 After completing the Registration, the system will ask you to sign-in with your newly created username and password. After signing-in, you can begin evaluation (Fig. 3). To start the evaluation process, click on the "Start Evaluation" link on the homepage. You will be presented with several options for which media player you want to use. We have invested considerable effort coding various interfaces to be compatible with the maximum number of browsers, platforms, and players. Most users should be able to use the Flash MP3 Player option -- which we've found to work the best. The Windows Media Player and QuickTime options should also work in most modern browsers. We have noted that the Media Player does introduce some start-up lags when starting a new query set which we believe is caused by the way Media Player downloads the MP3 files. We have also noted that the combination of Internet Explorer and QuickTime is simply not working.
-[[Image:E6ksms_options_page_scaled.png]]
+[[Image:2006_E6ksms_options_page_scaled.png]]
 '''Figure 3. Audio player selection page.'''
@@ Line 69: / Line 69: @@
 Once you have selected a player configuration, you'll see a query evaluation page (Fig. 4). On this page, you will see the query player and the list of candidate players. The queries and candidates drawn from the RISM collection (Queries #1 through #6) are fairly short - most candidates are about 4-5 seconds long. '''''Nota Bene''''' (The following description of the Mixed and Karaoke candidates has been updated, 3 September 2006). The length of candidates for the polyphonic task, drawn from the "Mixed" and "Karaoke" collections (Queries #7 through #17) is now more consistently in the 30 second range. We discovered that the timing information returned was not a reliable indicator of where the "best match" was found in each of the candidates. To correct for this, we took the "best match" starting information and then "padded" the candiate on both sides of the "best match" start point. Thus, the first 10 or so seconds of each candidate might, or might not, represent a strong "hit". The last 10 or so might also start "wandering away" from the "best match" region. Therfore, we recommend that you listen to the entire candidate before giving your grade and to be mindful that the "best match" section in each candidate is a bit slippery.
-[[Image:E6ksms_evaluation_page_scaled.png]]
+[[Image:2006_E6ksms_evaluation_page_scaled.png]]
 '''Figure 4. Sample evaluation page.'''
@@ Line 79: / Line 79: @@
 The procedure for listening to a candidate is the same as listening to a query -- click the "Play Candidate" buttons to load the clip into the player and listen to it (Fig. 4). Once you have a feeling for how similar the candidate is to the query, click the "Not Similar," "Somewhat Similar" or "Very Similar" radio buttons to the right of the candidate. Note: you can continue to replay the query during candidate playback.
-[[Image:E6ksms_select_category.png]]
+[[Image:2006_E6ksms_select_category.png]]
 '''Figure 5. Close up image of Broad Category selection buttons.'''
@@ Line 85: / Line 85: @@
 Depending on how you graded the candidate, you should see the candidate box react to your grade, indicating that the vote has been logged in the database (Fig. 6). If you indicated the song was "Somewhat Similar" the box will turn yellow and state "SAVED". If you indicated the song was "Very Similar" the box will turn green, "Not Similar" the box will turn red, and state "SAVED".
-[[Image:E6ksms_select_category_saved.png]]
+[[Image:2006_E6ksms_select_category_saved.png]]
 '''Figure 6. Close up image of Broad Category selection buttons with "Somewhat Similar" selected and "SAVED" automatically.'''
@@ Line 91: / Line 91: @@
 We also need each grader to assign a fine-grained score for the similarity of the candidate to the query on a scale of 0-10 (Fig. 7). More information about this process below.
-[[Image:E6ksms_select_score.png]]
+[[Image:2006_E6ksms_select_score.png]]
 '''Figure 7. Close up image of the Fine Score selection scale.'''
@@ Line 97: / Line 97: @@
 The system will automatically record the score when you let go of the scaler (Fig. 8).
-[[Image:E6ksms_select_score_saved.png]]
+[[Image:2006_E6ksms_select_score_saved.png]]
 '''Figure 8. Close up image of a Fine Score selection scale with a random score "SAVED".'''
@@ Line 103: / Line 103: @@
 You can also manually enter the score in the box, but in this case you MUST click the button labeled "Click to Save" to record your score (Fig. 9). After the score is saved, the scaler will automatically move to the correct position and the label on the button will change to "Saved" (Fig. 8).
-[[Image:E6ksms_select_score_before_save.PNG]]
+[[Image:2006_E6ksms_select_score_before_save.PNG]]
 '''Figure 9. Close up image of a Fine Score selection scale before "SAVED".'''
@@ Line 112: / Line 112: @@
-[[Image:E6ksms_my_assignment_scaled.png]]
+[[Image:2006_E6ksms_my_assignment_scaled.png]]
 '''Figure 10. My Assignment page. '''
@@ Line 118: / Line 118: @@
 You can also see a list of all of the queries you have been assigned at the bottom of each Candidate List page. You can return to any query by clicking on the button for that query here (Fig. 11). You can re-evaluate any candidate for any query at any time, up to the closing of the evaluation system.
-[[Image:Eval6_queries_detail.png]]
+[[Image:2006_Eval6_queries_detail.png]]
 '''Figure 11. Close up image of Query List buttons which allow for revisiting of completed query/candidate sets. '''

← Older revision		Revision as of 13:23, 8 September 2006
Line 11:		Line 11:

	Evalutron 6000 design and use details follow below and should clarify for you the use of the Broad Category and Fine Score input systems.		Evalutron 6000 design and use details follow below and should clarify for you the use of the Broad Category and Fine Score input systems.
		+
		+	====Clarification====
		+	A recent question about the Evalutron scoring system prompted the following response from me. In case others were confused, I think it best to share what I said with the rest of the grading community.
		+
		+
		+	'''My response''':
		+	The "Broad Category" and the "Fine Score" are meant to express the same grade only in different ways. Think of exams you might have taken: Some professors might give an A or B or F (i.e., Broad Category). On the same exam, another professor might want to express the same grade as 99%, 78% or 20% (i.e., Fine Score).
		+
		+	We need the graders to be "two professors in one" . We are asking each grader to be a "broad grading" professor and also a "fine grading" professor. On our fine scale, 0 is meant to represent complete failure and 10 a perfectly similar success. With the set of paired "Broad" and "Fine" scores we can can do different kinds of post-Evalutron analyses with each kind of score. Also, we want to be able to take both kinds of scores and analyze them to see what kind of fine scores are associated with each broad score.

	===Coming and Going: Avoiding Grader Fatigue===		===Coming and Going: Avoiding Grader Fatigue===

@@ Line 2: / Line 2: @@
 ===Special Update on Candidate Files (3 September 2006)===
-'''''Nota Bene''''' (The following description of the Mixed and Karaoke candidates has been updated, 3 September 2006). The length of candidates for the polyphonic task, drawn from the "Mixed" and "Karaoke" collections (Queries #7 through #17) is now more consistently in the 30 second range. We discovered that the timing information returned was not a reliable indicator of where the "best match" was found in each of the candidates. To correct for this, we took the "best match" starting information and then "padded" the candiate on both sides of the "best match" start point. Thus, the first 10 or so seconds of each candidate might, or might not, represent a strong "hit". The last 10 or so might also start "wandering away" from the "best match" region. Therfore, we recommend that you listen to the entire candidate before giving your grade and to be mindful the that "best match" section in each candidate is a bit slippery.
+'''''Nota Bene''''' (The following description of the Mixed and Karaoke candidates has been updated, 3 September 2006). The length of candidates for the polyphonic task, drawn from the "Mixed" and "Karaoke" collections (Queries #7 through #17) is now more consistently in the 30 second range. We discovered that the timing information returned was not a reliable indicator of where the "best match" was found in each of the candidates. To correct for this, we took the "best match" starting information and then "padded" the candiate on both sides of the "best match" start point. Thus, the first 10 or so seconds of each candidate might, or might not, represent a strong "hit". The last 10 or so might also start "wandering away" from the "best match" region. Therfore, we recommend that you listen to the entire candidate before giving your grade and to be mindful that the "best match" section in each candidate is a bit slippery.
 ===Task Description: Focusing on Melody===
@@ Line 58: / Line 58: @@
 ==Evaluation Pages: Queries and Candidate Lists==
-Once you have selected a player configuration, you'll see a query evaluation page (Fig. 4). On this page, you will see the query player and the list of candidate players. The queries and candidates drawn from the RISM collection (Queries #1 through #6) are fairly short - most candidates are about 4-5 seconds long. '''''Nota Bene''''' (The following description of the Mixed and Karaoke candidates has been updated, 3 September 2006). The length of candidates for the polyphonic task, drawn from the "Mixed" and "Karaoke" collections (Queries #7 through #17) is now more consistently in the 30 second range. We discovered that the timing information returned was not a reliable indicator of where the "best match" was found in each of the candidates. To correct for this, we took the "best match" starting information and then "padded" the candiate on both sides of the "best match" start point. Thus, the first 10 or so seconds of each candidate might, or might not, represent a strong "hit". The last 10 or so might also start "wandering away" from the "best match" region. Therfore, we recommend that you listen to the entire candidate before giving your grade and to be mindful the that "best match" section in each candidate is a bit slippery.
+Once you have selected a player configuration, you'll see a query evaluation page (Fig. 4). On this page, you will see the query player and the list of candidate players. The queries and candidates drawn from the RISM collection (Queries #1 through #6) are fairly short - most candidates are about 4-5 seconds long. '''''Nota Bene''''' (The following description of the Mixed and Karaoke candidates has been updated, 3 September 2006). The length of candidates for the polyphonic task, drawn from the "Mixed" and "Karaoke" collections (Queries #7 through #17) is now more consistently in the 30 second range. We discovered that the timing information returned was not a reliable indicator of where the "best match" was found in each of the candidates. To correct for this, we took the "best match" starting information and then "padded" the candiate on both sides of the "best match" start point. Thus, the first 10 or so seconds of each candidate might, or might not, represent a strong "hit". The last 10 or so might also start "wandering away" from the "best match" region. Therfore, we recommend that you listen to the entire candidate before giving your grade and to be mindful that the "best match" section in each candidate is a bit slippery.
 [[Image:E6ksms_evaluation_page_scaled.png]]

@@ Line 58: / Line 58: @@
 ==Evaluation Pages: Queries and Candidate Lists==
-Once you have selected a player configuration, you'll see a query evaluation page (Fig. 4). On this page, you will see the query player and the list of candidate players. The queries and candidates drawn from the RISM collection (Queries #1 through #6) are fairly short - most candidates are about 4-5 seconds long. '''''Nota Bene''''' (The following description of the Mixed and Karaoke candidates has been updated, 3 September 2006). The length of candidates for the polyphonic task, drawn from the "Mixed" and "Karaoke" collections (Queries #7 through #17) is now more consistently in the 30 second range. We discovered that the timing information returned was not a reliable indicator of where the "best match" was found in each of the candidates. To correct for this, we took the "best match" starting information and then "padded" the candiate on both sides of the "best match" start point. Thus, the first ten or so seconds of each candidate might, or might not, represent a strong "hit". Therfore, we recommend that you listen to the entire candidate before giving your grade and to be mindful the that "best match" section in each candidate is a bit slippery.
+Once you have selected a player configuration, you'll see a query evaluation page (Fig. 4). On this page, you will see the query player and the list of candidate players. The queries and candidates drawn from the RISM collection (Queries #1 through #6) are fairly short - most candidates are about 4-5 seconds long. '''''Nota Bene''''' (The following description of the Mixed and Karaoke candidates has been updated, 3 September 2006). The length of candidates for the polyphonic task, drawn from the "Mixed" and "Karaoke" collections (Queries #7 through #17) is now more consistently in the 30 second range. We discovered that the timing information returned was not a reliable indicator of where the "best match" was found in each of the candidates. To correct for this, we took the "best match" starting information and then "padded" the candiate on both sides of the "best match" start point. Thus, the first 10 or so seconds of each candidate might, or might not, represent a strong "hit". The last 10 or so might also start "wandering away" from the "best match" region. Therfore, we recommend that you listen to the entire candidate before giving your grade and to be mindful the that "best match" section in each candidate is a bit slippery.
 [[Image:E6ksms_evaluation_page_scaled.png]]

← Older revision		Revision as of 16:34, 4 September 2006
Line 1:		Line 1:
	==Special Comments about Grading the Symbolic Melodic Similarity Submissions ==		==Special Comments about Grading the Symbolic Melodic Similarity Submissions ==
		+
		+	===Special Update on Candidate Files (3 September 2006)===
		+	'''''Nota Bene''''' (The following description of the Mixed and Karaoke candidates has been updated, 3 September 2006). The length of candidates for the polyphonic task, drawn from the "Mixed" and "Karaoke" collections (Queries #7 through #17) is now more consistently in the 30 second range. We discovered that the timing information returned was not a reliable indicator of where the "best match" was found in each of the candidates. To correct for this, we took the "best match" starting information and then "padded" the candiate on both sides of the "best match" start point. Thus, the first 10 or so seconds of each candidate might, or might not, represent a strong "hit". The last 10 or so might also start "wandering away" from the "best match" region. Therfore, we recommend that you listen to the entire candidate before giving your grade and to be mindful the that "best match" section in each candidate is a bit slippery.

	===Task Description: Focusing on Melody===		===Task Description: Focusing on Melody===

@@ Line 32: / Line 32: @@
 # This system is NOT open to the general public. Due to the legalities imposed upon us by the University of Illinois and the US Federal Government, we are allowed to accept only those graders that have a stake in Music Information Retrieval and Music Digital Library research. Acceptable persons include MIREX participants, subscribers to the music-ir[AT]ircam.fr list, ISMIR attendees, computational musicologists, music technology researchers, etc. If you are in doubt about your acceptability, please contact Prof. Downie at [mailto:jdownie@uiuc.edu].
-# Do NOT create an account for yourself if you are merely curious to see what the Evalutron 6000 is set up to do. The account creation process scientifcally distributes Query and Candidate sets to each account holder. The act of creating an "curiousity account" seriously disrupts the adminstration of the results data we are collecting.
+# Do NOT create an account for yourself if you are merely curious to see what the Evalutron 6000 is set up to do. The account creation process scientifically distributes Query and Candidate sets to each account holder. The act of creating a "curiousity account" seriously disrupts the adminstration of the results data we are collecting.
 # IF YOU ARE REALLY CURIOUS: We have created a small "sandbox" version of the AudioSim Evalutron with fake data at: https://music-ir.org/eval6000/sandbox. You still must go through all the official procedures but your scores wil not affect anything as the data is "made up".
 # Please do NOT create multiple accounts for yourself for this particular evaluation task. If you are also grading the Audio Music Similarity and Retrieval (AudioSim) task, you will be creating another task-specific account for AudioSim in the AudioSim space. The system is designed to track individuals based upon unique sign-in IDs. The creation of multiple accounts for one person causes the system to improperly distribute the assignments to your fellow graders. If you have difficulties, such as forgetting your sign-in password, please contact us at [mailto:mrx-com09@lists.lis.uiuc.edu] or try the "Forgot Your Password?" link found at the top of the homepage (Fig 1).

@@ Line 39: / Line 39: @@
 First you must register a new account. Click on the "Register" link (listed under Step 2) on the page to create an account.
-The registration page is fairly straightforward (Fig. 2). Required fields are marked in blue with asterisks. You can create any username and password you wish. Passwords must be at least 6 characters long and are case-sensitive. Before completing the registration, you must read and agree to the terms of the Informed Consent document. The evaluation, because it is using human judgements of similarity, is considered a human-subjects research project and the Evaluatron is basically a survey instrument. To indicate your consent to participate in the evaluation, check the "I Agree" checkbox below the informed consent document.
+The registration page is fairly straightforward (Fig. 2). Required fields are marked in blue with asterisks. You can create any username and password you wish. Passwords must be at least 6 characters long and are case-sensitive. Before completing the registration, you must read and agree to the terms of the Informed Consent document. The evaluation, because it is using human judgements of similarity, is considered a human-subjects research project and the Evalutron is basically a survey instrument. To indicate your consent to participate in the evaluation, check the "I Agree" checkbox below the informed consent document.
 If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.