2011:Evalutron6000 Walkthrough - Revision history

Kahyun Choi at 04:25, 30 September 2011

2011-09-30T04:25:06Z

2011-09-30T04:24:05Z

‎Monitor Progress

2011-09-30T04:20:56Z

‎Work on Another Query

2011-09-30T04:18:43Z

‎Evaluate

2011-09-30T04:15:44Z

‎Evaluate

2011-09-30T04:10:42Z

‎Evaluate

2011-09-30T04:07:38Z

‎Get Your Assignments

2011-09-30T02:57:24Z

‎Agree to the Informed Consent

2011-09-30T02:49:16Z

‎Welcome to the Evalutron 6000

2011-09-30T02:48:06Z

‎Register to the Evalutron 6000

@@ Line 1: / Line 1: @@
-==UPDATE 2010==
+==UPDATE 2011==
-The 2010 Evalutron has a new look and implementation. It is easier to use than the original implementation. However, it is necessary to read through this document before you start using the system. Enjoy!
+The 2011 Evalutron has a new look and implementation. It is easier to use than the original implementation. However, it is necessary to read through this document before you start using the system. Enjoy!
 == Four Important Issues Before Proceeding ==

@@ Line 90: / Line 90: @@
 At any time a grader may monitor his/her progress on the My Assignment page (Fig. 9). Each query has a status bar where the completed portion will be marked as green and the unfinished part red. When all the bars become green, the assignments are all completed.
-[[Image:2010_e6k_progress.png|border]]
+[[Image:2011_e6k_progress.png|border]]
 '''Figure 9. Progress indicated on the My Assignment Page '''
 ==Grading Expectations and "Reasonableness"==

@@ Line 76: / Line 76: @@
 At the bottom of the evaluation page, there is a "View All Assignments" button (Fig. 8). At any time, clicking this button will direct you to your assignment page similar to the one shown in Fig. 4. When you have completed evaluating all of the candidates for this query, you may click this button to continue on another query.
-[[Image:2010_e6k_eval_page_bottom.png|border]]
+[[Image:2011_e6k_eval_page_bottom.png|border]]
 '''Figure 8. Close up image of "View All Assignments" button which loads the assignment page. '''

@@ Line 66: / Line 66: @@
 Only after you input BOTH the broad category and the fine score can a query-candidate pair be marked as green, indicating you have completed evaluating this query-candidate pair.
-[[Image:2010_e6k_qcq_done_ detail.png|border]]
+[[Image:2011_e6k_qcq_done_ detail.png|border]]
 '''Figure 7. Close up image of a completed query-candidate pair.'''

@@ Line 60: / Line 60: @@
 Once you have a feeling for whether or not the candidate is similar to the query, click the "Not Similar", "Somewhat Similar" or "Very Similar" radio buttons to the right of the query-candidate pair. Each grader will also need to assign a fine-grained score for the similarity of the candidate to the query on a scale of 0-100, with 0 indicating completely different and 100 perfectly similar or identical). To input the fine score, you need to move the slider, and once you let go of the slider the system will automatically record the score.
-[[Image:2010_e6k_qcq_detail.png|border]]
+[[Image:2011_e6k_qcq_detail.png|border]]
 '''Figure 6. Close up image of a query-candidate pair and evaluation buttons.'''

@@ Line 52: / Line 52: @@
 Clicking the "Evaluate Query" button under a query will lead you to the evaluation page of that query (Fig. 5). This page consists of instructions on the top and a list of query-candidate pairs. Please read the instructions carefully. The query for all candidates on a single page is the same, it is included with each candidate so that you can easily compare it to each candidate. For the Audio Music Similarity (AMS) task, each query and candidate is 30 second long. For Symbolic Melody Similarity (SMS) task, the length is about 10 seconds. Clicking on the player button besides each query or candidate will load the song into the player and begin playing. Clicking the player button again will pause it. We recommend you listen to the entire query at least once before evaluating any candidate files.
-[[Image:2010_e6k_eval_page.png|border]]
+[[Image:2011_e6k_eval_page.png|border]]
 '''Figure 5. Sample evaluation page.'''

@@ Line 45: / Line 45: @@
 Please note that once assignments are made, they cannot be changed or removed. Therefore, please NEVER click "Get Assignment" for the task you do not intend to participate.
-[[Image:2010_e6k_getAssignment.png|border]]
+[[Image:2011_e6k_getAssignment.png|border]]
 '''Figure 4. Evalutron 6000 get assignment page.'''

@@ Line 35: / Line 35: @@
 If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.
-[[Image:2011_e6k_consent.png|800px|border]]
+[[Image:2011_e6k_consent.png|border]]
 '''Figure 3. Evalutron 6000 informed consent page.'''

@@ Line 15: / Line 15: @@
 When first visiting the Evalutron 6000 homepage, you will see its home page (Fig. 1).
-[[Image:2011_e6k_home.png]]
+[[Image:2011_e6k_home.png|border]]
 '''Figure 1. Evalutron 6000 home page.'''

@@ Line 25: / Line 25: @@
 The registration page is fairly straightforward (Fig. 2). All fields are required. You can create any username and password you wish. Username must be at least 5 characters long and passwords must be at least 8 characters long and are case-sensitive. Before completing the registration, you will receive an email with an activation link. Clicking that link will complete your registration.
-[[Image:2011 e6k register.png|800px|border]]
+[[Image:2011 e6k register.png|border]]
 '''Figure 2. Evalutron 6000 registration page.'''