2010:Evalutron6000 Walkthrough - Revision history

Craig: /* Welcome to the Evalutron 6000 */

2010-07-17T15:13:23Z

‎Welcome to the Evalutron 6000

Xiao: /* Four Important Issues Before Proceeding */

2010-07-15T23:19:53Z

‎Four Important Issues Before Proceeding

Xiao: /* Evaluate */

2010-07-15T20:44:11Z

‎Evaluate

CameronJones: /* Evaluate */

2010-07-15T19:31:15Z

‎Evaluate

CameronJones: /* Agree to the Informed Consent */

2010-07-15T19:26:56Z

‎Agree to the Informed Consent

CameronJones: /* Register to the Evalutron 6000 */

2010-07-15T19:25:13Z

‎Register to the Evalutron 6000

CameronJones: /* Welcome to the Evalutron 6000 */

2010-07-15T19:24:48Z

‎Welcome to the Evalutron 6000

Xiao at 22:29, 14 July 2010

2010-07-14T22:29:55Z

Xiao: /* Evaluate */

2010-07-14T22:27:10Z

‎Evaluate

Xiao: /* Work on Another Query */

2010-07-14T22:25:54Z

‎Work on Another Query

@@ Line 11: / Line 11: @@
 ==Welcome to the Evalutron 6000==
-In order to use the Evalutron 6000 you will need to modern web browser (e.g., Firefox, Internet Explorer, Safari, Mozilla, etc) that supports JavaScript, Flash, and Cookies. Evalutron has been tested on Windows XP, Windows 7, MacOS X, and Ubuntu Linux. If you are using a different platform and having trouble, please try accessing Evalutron 6000 from another machine. If you are still having difficulty, contact us at [mailto:mirex@imirsel.org mirex@imirsel.org].
+In order to use the Evalutron 6000 you will need a modern web browser (e.g., Firefox, Internet Explorer, Safari, Mozilla, etc) that supports JavaScript, Flash, and Cookies. Evalutron has been tested on Windows XP, Windows 7, MacOS X, and Ubuntu Linux. If you are using a different platform and having trouble, please try accessing Evalutron 6000 from another machine. If you are still having difficulty, contact us at [mailto:mirex@imirsel.org mirex@imirsel.org].
 When first visiting the Evalutron 6000 homepage, you will see its home page (Fig. 1).

@@ Line 5: / Line 5: @@
 # This system is NOT open to the general public. Due to the legalities imposed upon us by the University of Illinois and the US Federal Government, we are allowed to accept only those graders that have a stake in Music Information Retrieval and Music Digital Library research. Acceptable persons include MIREX participants, subscribers to the music-ir[AT]ircam.fr list, ISMIR attendees, computational musicologists, music technology researchers, etc. If you are in doubt about your acceptability, please contact Prof. Downie at [mailto:jdownie@illinois.edu].
-# Do NOT click "Get Assignments" link if you are merely curious to see what the Evalutron 6000 is set up to do. The assignment distribution process scientifically distributes Query and Candidate sets to each account asking for assignments.
+# Do NOT click "Get Assignments" link if you are merely curious to see what the Evalutron 6000 is set up to do. The assignment distribution process scientifically distributes Query and Candidate sets to each account asking for assignments. To have a look at the Evalutron evaluation page, please click the "Try E6K Demo" link on the left-side of the page.
 # The Evalutron is set up for both Audio Music Similarity (AMS) task and Symbolic Melodic Similarity (SMS) task. Please do NOT click "Get Assignments" link under the task you do not intend to participate. Otherwise, you will get assignments for that task which will prevent other graders from getting the assignments you get.
 # Please do NOT create multiple accounts for yourself Evalutron. The system is designed to track individuals based upon unique sign-in IDs. The creation of multiple accounts for one person causes the system to improperly distribute the assignments to your fellow graders. If you have difficulties, such as forgetting your sign-in password, please contact us at [mailto:mirproject@lists.lis.illinois.edu] or try the "Forgot Password" link found at the left side of the homepage (Fig 1).

@@ Line 50: / Line 50: @@
 ==Evaluate==
-Clicking the "Evaluate Query" button under a query will lead you to the evaluation page of that query (Fig. 5). This page consists of instructions on the top and a list of query-candidate pairs. Please read the instructions carefully. The query for all candidates on a single page is the same, it is included with each candidate so that you can easily compare it to each candidate. For the Audio Music Similarity (AMS) task, each query and candidate is 30 second long. For Symbolic Melody Similarity (SMS) task, the length varies. Clicking on the player button besides each query or candidate will load the song into the player and begin playing. Clicking the player button again will pause it. We recommend you listen to the entire query at least once before evaluating any candidate files.
+Clicking the "Evaluate Query" button under a query will lead you to the evaluation page of that query (Fig. 5). This page consists of instructions on the top and a list of query-candidate pairs. Please read the instructions carefully. The query for all candidates on a single page is the same, it is included with each candidate so that you can easily compare it to each candidate. For the Audio Music Similarity (AMS) task, each query and candidate is 30 second long. For Symbolic Melody Similarity (SMS) task, the length is about 10 seconds. Clicking on the player button besides each query or candidate will load the song into the player and begin playing. Clicking the player button again will pause it. We recommend you listen to the entire query at least once before evaluating any candidate files.
 [[Image:2010_e6k_eval_page.png|border]]

@@ Line 50: / Line 50: @@
 ==Evaluate==
-Clicking the "Evaluate Query" button under a query will lead you to the evaluation page of that query (Fig. 5). This page consists of instructions on the top and a list of query-candidate pairs. Please read the instructions carefully. The query in each of the query-candidate pairs is the same, and it is aligned with each candidate so that you can replay it at any time when you evaluate the candidate. For Audio Music Similarity (AMS) task, each query and candidate is 30 second long. For Symbolic Melody Similarity (SMS) task, the length varies. Clicking on the player button besides each query or candidate will load the song into the player and begin playing. Clicking the player button again will pause it. We recommend you listen to the entire query at least once before evaluating any candidate files.
+Clicking the "Evaluate Query" button under a query will lead you to the evaluation page of that query (Fig. 5). This page consists of instructions on the top and a list of query-candidate pairs. Please read the instructions carefully. The query for all candidates on a single page is the same, it is included with each candidate so that you can easily compare it to each candidate. For the Audio Music Similarity (AMS) task, each query and candidate is 30 second long. For Symbolic Melody Similarity (SMS) task, the length varies. Clicking on the player button besides each query or candidate will load the song into the player and begin playing. Clicking the player button again will pause it. We recommend you listen to the entire query at least once before evaluating any candidate files.
 [[Image:2010_e6k_eval_page.png|border]]
@@ Line 58: / Line 58: @@
 Please note there are more candidates than may be immediately visible on the page. Please scroll to the bottom of the candidate list to make sure you've evaluated each song.
-Once you have a feeling for whether or not the candidate is similar to the query, click the "Not Similar", "Somewhat Similar" or "Very Similar" radio buttons to the right of the query-candidate pair. Each grader will also need to assign a fine-grained score for the similarity of the candidate to the query on a scale of 0-100, with 0 indicating completely different and 100 perfectly similar or identical). To input the fine score, you need to move the scaler, and once you let go of the scaler the system will automatically record the score.
+Once you have a feeling for whether or not the candidate is similar to the query, click the "Not Similar", "Somewhat Similar" or "Very Similar" radio buttons to the right of the query-candidate pair. Each grader will also need to assign a fine-grained score for the similarity of the candidate to the query on a scale of 0-100, with 0 indicating completely different and 100 perfectly similar or identical). To input the fine score, you need to move the slider, and once you let go of the slider the system will automatically record the score.
 [[Image:2010_e6k_qcq_detail.png|border]]
@@ Line 64: / Line 64: @@
 '''Figure 6. Close up image of a query-candidate pair and evaluation buttons.'''
-Only after you input BOTH the broad category and the fine score can a query-candidate pair be marked as green, indicating you have complete evaluating this query-candidate pair.
+Only after you input BOTH the broad category and the fine score can a query-candidate pair be marked as green, indicating you have completed evaluating this query-candidate pair.
 [[Image:2010_e6k_qcq_done_ detail.png|border]]

@@ Line 31: / Line 31: @@
 ==Agree to the Informed Consent ==
-Before starting evaluation, every evaluator must read and agree to the terms of the Informed Consent document. Otherwise, you will be redirect to the informed consent page when you try to get evaluation assignments. Clicking the "Informed Consent" link on the left-side menu will show the form (Fig. 3). The evaluation, because it is using human judgments of similarity, is considered a human-subjects research project and the Evaluatron is basically a survey instrument. To indicate your consent to participate in the evaluation, scroll down the page and check the "I Agree" checkbox below the informed consent document.
+Before starting evaluation, every evaluator must read and agree to the terms of the Informed Consent document. Otherwise, you will be redirect to the informed consent page when you try to get evaluation assignments. Clicking the "Informed Consent" link on the left-side menu will show the form (Fig. 3). The evaluation, because it is using human judgments of similarity, is considered a human-subjects research project and the Evalutron is a survey instrument. To indicate your consent to participate in the evaluation, scroll down the page and check the checkbox below the informed consent document.
 If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.

@@ Line 21: / Line 21: @@
 ==Register to the Evalutron 6000==
-If you have an account with the submission system, then you can use the same account for Evalutron 6000. Otherwise, you must register a new account. Click on the "Register" link on the left side of the page to create an account.
+If you have an account with the MIREX submission system, then you can use the same account for Evalutron 6000. Otherwise, you must register a new account. Click on the "Register" link on the left side of the page to create an account.
 The registration page is fairly straightforward (Fig. 2). All fields are required. You can create any username and password you wish. Username must be at least 5 characters long and passwords must be at least 8 characters long and are case-sensitive. Before completing the registration, you will receive an email with an activation link. Clicking that link will complete your registration.

← Older revision		Revision as of 22:29, 14 July 2010
Line 98:		Line 98:

	For each query-candidate pair, we need you to assign BOTH a Broad Category score AND a Fine Score (i.e., a numeric grade between 0 and 100, 0 is meant to represent complete different and 100 perfectly similar or identical.). You have the freedom to make whatever associations you desire between a particular Broad Category score and its related Fine Score. In fact, we expect to see variations across evaluators with regard to the relationships between Broad Categories and Fine Scores as this is a normal part of human subjectivity. However, we will be using the two different types of scores to do important inter-related post-Evalutron calculations so, please, do be thoughtful in selecting your Broad Categories and related Fine Scores. What we are really asking here is that you apply a level of "reasonableness" to both your scores and your associations. For example, if you score a candidate in the VERY SIMILAR category, a Fine Score of 21 would not be, by most standards, "reasonable". Same applies at the other extreme. For example, a Broad Category score of NOT SIMILAR should not be associated with a Fine Score of, say, 72 or 84, etc.		For each query-candidate pair, we need you to assign BOTH a Broad Category score AND a Fine Score (i.e., a numeric grade between 0 and 100, 0 is meant to represent complete different and 100 perfectly similar or identical.). You have the freedom to make whatever associations you desire between a particular Broad Category score and its related Fine Score. In fact, we expect to see variations across evaluators with regard to the relationships between Broad Categories and Fine Scores as this is a normal part of human subjectivity. However, we will be using the two different types of scores to do important inter-related post-Evalutron calculations so, please, do be thoughtful in selecting your Broad Categories and related Fine Scores. What we are really asking here is that you apply a level of "reasonableness" to both your scores and your associations. For example, if you score a candidate in the VERY SIMILAR category, a Fine Score of 21 would not be, by most standards, "reasonable". Same applies at the other extreme. For example, a Broad Category score of NOT SIMILAR should not be associated with a Fine Score of, say, 72 or 84, etc.
		+
		+
		+	==Coming and Going: Avoiding Grader Fatigue==
		+
		+	Listening to, and then comparing, hundreds of audio files is very tiring. We have built into the back-end of the Evalutron 6000 system a rather robust database system that records in near-real-time your grading scores. These scores are saved along with information about which queries and candidates you have yet to review. All this information is stored in association with your personal sign-in ID. This means you can break up your grading over several days at times convenient and productive to you. In fact, we recommend that you not try to tackle your "assignment" in one big chunk as fresh ears are happy ears and happy ears make for better evaluations.

@@ Line 82: / Line 82: @@
 You may also click the "My Assignments" link on the left-side menu to go to the My Assignments page.
-On the My Assignments page, you may click another "Evaluate Query" button to load evaluation page (like Fig. 5) for another query.
+On the My Assignments page, you may click another "Evaluate Query" button to load evaluation page (like Fig. 5) of that query.
 You will see a list of all of the queries you have evaluated (or are evaluating) at the My Assignments page. You can return to any query by clicking on the "Evaluate Query" underneath it (Fig. 4). You can re-evaluate any candidate for any query at any time, up to the closing of the evaluation system.