Difference between revisions of "2010:Evalutron6000 Walkthrough"
| Line 20: | Line 20: | ||
The registration page is fairly straightforward (Fig. 2). All fields are required. You can create any username and password you wish. Username must be at least 5 characters long and passwords must be at least 8 characters long and are case-sensitive. Before completing the registration, you will receive an email with an activation link. Clicking that link will complete your registration. | The registration page is fairly straightforward (Fig. 2). All fields are required. You can create any username and password you wish. Username must be at least 5 characters long and passwords must be at least 8 characters long and are case-sensitive. Before completing the registration, you will receive an email with an activation link. Clicking that link will complete your registration. | ||
| − | [[Image:2010_e6k_register.png]] | + | [[Image:2010_e6k_register.png|border]] |
'''Figure 2. Evalutron 6000 registration page.''' | '''Figure 2. Evalutron 6000 registration page.''' | ||
| Line 30: | Line 30: | ||
If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066. | If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066. | ||
| − | [[Image:2010_e6k_consent.png]] | + | [[Image:2010_e6k_consent.png|border]] |
'''Figure 3. Evalutron 6000 informed consent page.''' | '''Figure 3. Evalutron 6000 informed consent page.''' | ||
| Line 40: | Line 40: | ||
Please note once assignments are made, they cannot be changed and you are responsible to finish evaluations assigned to you. Therefore, please NEVER click "Get Assignment" for the task you do not intend to participate. | Please note once assignments are made, they cannot be changed and you are responsible to finish evaluations assigned to you. Therefore, please NEVER click "Get Assignment" for the task you do not intend to participate. | ||
| − | [[Image:2010_e6k_getAssignment.png]] | + | [[Image:2010_e6k_getAssignment.png|border]] |
'''Figure 4. Evalutron 6000 get assignment page.''' | '''Figure 4. Evalutron 6000 get assignment page.''' | ||
| Line 47: | Line 47: | ||
Clicking the "Evaluate Query" button will lead you to the evaluation page (Fig. 5). This page consists of instructions on the top and a list of query-candidate pairs. Please read the instructions carefully. The query in each of the query-candidate pairs is the same, and it is aligned with each candidate so that you can replay it at any time when you evaluate the candidate. For Audio Music Similarity (AMS) task, each query and candidate is 30 second long. For Symbolic Melody Similarity (SMS) task, the length varies. Clicking on the player button besides each query or candidate will load the song into the player and begin playing. Clicking the player button again will pause it. We recommend you listen to the entire query at least once before evaluating any candidate files. | Clicking the "Evaluate Query" button will lead you to the evaluation page (Fig. 5). This page consists of instructions on the top and a list of query-candidate pairs. Please read the instructions carefully. The query in each of the query-candidate pairs is the same, and it is aligned with each candidate so that you can replay it at any time when you evaluate the candidate. For Audio Music Similarity (AMS) task, each query and candidate is 30 second long. For Symbolic Melody Similarity (SMS) task, the length varies. Clicking on the player button besides each query or candidate will load the song into the player and begin playing. Clicking the player button again will pause it. We recommend you listen to the entire query at least once before evaluating any candidate files. | ||
| − | [[Image:2010_e6k_eval_page.png]] | + | [[Image:2010_e6k_eval_page.png|border]] |
'''Figure 5. Sample evaluation page.''' | '''Figure 5. Sample evaluation page.''' | ||
| Line 55: | Line 55: | ||
Once you have a feeling for whether or not the candidate is similar to the query, click the "Not Similar", "Somewhat Similar" or "Very Similar" radio buttons to the right of the query-candidate pair. Each grader will also need to assign a fine-grained score for the similarity of the candidate to the query on a scale of 0-100. To input the fine score, you need to move the scaler, and once you let go of the scaler the system will automatically record the score. | Once you have a feeling for whether or not the candidate is similar to the query, click the "Not Similar", "Somewhat Similar" or "Very Similar" radio buttons to the right of the query-candidate pair. Each grader will also need to assign a fine-grained score for the similarity of the candidate to the query on a scale of 0-100. To input the fine score, you need to move the scaler, and once you let go of the scaler the system will automatically record the score. | ||
| − | [[Image:2010_e6k_qcq_detail.png]] | + | [[Image:2010_e6k_qcq_detail.png|border]] |
'''Figure 6. Close up image of a query-candidate pair and evaluation buttons.''' | '''Figure 6. Close up image of a query-candidate pair and evaluation buttons.''' | ||
| Line 61: | Line 61: | ||
Only after you input BOTH the broad category and the fine score can a query-candidate pair be marked as green, indicating you have complete evaluating this query-candidate pair. | Only after you input BOTH the broad category and the fine score can a query-candidate pair be marked as green, indicating you have complete evaluating this query-candidate pair. | ||
| − | [[Image:2010_e6k_qcq_done_ detail.png]] | + | [[Image:2010_e6k_qcq_done_ detail.png|border]] |
'''Figure 7. Close up image of a completed query-candidate pair.''' | '''Figure 7. Close up image of a completed query-candidate pair.''' | ||
| Line 71: | Line 71: | ||
At the bottom of the evaluation page, there is a "View All Assignments" button (Fig. 8). At any time, clicking this button will direct you to your assignment page similar to the one shown in Fig. 4. When you have completed evaluating all of the candidates for this query, you may click this button to continue on another query. | At the bottom of the evaluation page, there is a "View All Assignments" button (Fig. 8). At any time, clicking this button will direct you to your assignment page similar to the one shown in Fig. 4. When you have completed evaluating all of the candidates for this query, you may click this button to continue on another query. | ||
| − | [[Image:2010_e6k_eval_page_bottom.png]] | + | [[Image:2010_e6k_eval_page_bottom.png|border]] |
'''Figure 8. Close up image of "View All Assignments" button which loads the assignment page. ''' | '''Figure 8. Close up image of "View All Assignments" button which loads the assignment page. ''' | ||
| Line 83: | Line 83: | ||
==Monitor Progress== | ==Monitor Progress== | ||
| − | [[Image:2006_eval6_queries_detail.png]] | + | [[Image:2006_eval6_queries_detail.png|border]] |
'''Figure 9. Close up image of Query List buttons which allow for revisiting of completed query/candidate sets. ''' | '''Figure 9. Close up image of Query List buttons which allow for revisiting of completed query/candidate sets. ''' | ||
Revision as of 15:47, 14 July 2010
Contents
UPDATE 14 July 2010
The 2010 Evalutron has a new look and implementation. It is easier to use than the original implementation. However, it is necessary to read through this document before you start using the system. Enjoy!
Welcome to the Evalutron 6000
In order to use the Evalutron 6000 you will need to be using a modern web browser (e.g., Firefox, Internet Explorer, Safari, Mozilla, etc) that supports JavaScript (ECMAScript) and Cookies. Evalutron has been tested on Windows XP, MacOS X, and RedHat Linux. If you are using a different platform and having trouble, please try accessing Evalutron 6000 from another machine. If you are still having difficulty, contact
mirproject@lists.lis.uiuc.edu.
When first visiting the Evalutron 6000 homepage, you will see a page similar to this (Fig. 1).
Figure 1. Evalutron 6000 home page.
Register to the Evalutron 6000
If you have an account with the submission system, then you can use the same account for Evalutron 6000. Otherwise, you must register a new account. Click on the "Register" link on the left side of the page to create an account.
The registration page is fairly straightforward (Fig. 2). All fields are required. You can create any username and password you wish. Username must be at least 5 characters long and passwords must be at least 8 characters long and are case-sensitive. Before completing the registration, you will receive an email with an activation link. Clicking that link will complete your registration.
Figure 2. Evalutron 6000 registration page.
Agree to the Informed Consent
Before starting evaluation, every evaluator must read and agree to the terms of the Informed Consent document. Otherwise, you will be redirect to the informed consent page when you try to get evaluation assignments. Clicking the "Informed Consent" link on the left-side menu will show the form (Fig. 3). The evaluation, because it is using human judgments of similarity, is considered a human-subjects research project and the Evaluatron is basically a survey instrument. To indicate your consent to participate in the evaluation, scroll down the page and check the "I Agree" checkbox below the informed consent document.
If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.
Figure 3. Evalutron 6000 informed consent page.
Get Your Assignments
To start the evaluation process, click the "My Assignments" link on the left-side menu. The assignment page is similar to Fig. 4. This page shows all tasks available in the Evalutron 6000, and initially there is no assignment given. Be careful to select the task you intend to participate, and click the "Get Assignment" button under this task. The system will assign you a number of queries to evaluate.
Please note once assignments are made, they cannot be changed and you are responsible to finish evaluations assigned to you. Therefore, please NEVER click "Get Assignment" for the task you do not intend to participate.
Figure 4. Evalutron 6000 get assignment page.
Evaluate
Clicking the "Evaluate Query" button will lead you to the evaluation page (Fig. 5). This page consists of instructions on the top and a list of query-candidate pairs. Please read the instructions carefully. The query in each of the query-candidate pairs is the same, and it is aligned with each candidate so that you can replay it at any time when you evaluate the candidate. For Audio Music Similarity (AMS) task, each query and candidate is 30 second long. For Symbolic Melody Similarity (SMS) task, the length varies. Clicking on the player button besides each query or candidate will load the song into the player and begin playing. Clicking the player button again will pause it. We recommend you listen to the entire query at least once before evaluating any candidate files.
Figure 5. Sample evaluation page.
Please note that the list of query-candidate pairs scrolls within the page, there are more candidates than may be immediately visible on the page. Please scroll to the bottom of the candidate list to make sure you've evaluated each song.
Once you have a feeling for whether or not the candidate is similar to the query, click the "Not Similar", "Somewhat Similar" or "Very Similar" radio buttons to the right of the query-candidate pair. Each grader will also need to assign a fine-grained score for the similarity of the candidate to the query on a scale of 0-100. To input the fine score, you need to move the scaler, and once you let go of the scaler the system will automatically record the score.
Figure 6. Close up image of a query-candidate pair and evaluation buttons.
Only after you input BOTH the broad category and the fine score can a query-candidate pair be marked as green, indicating you have complete evaluating this query-candidate pair.
Figure 7. Close up image of a completed query-candidate pair.
You can always change your evaluation for any candidate by toggling the radio buttons and adjust the Fine Score selection scale. Once an evaluation has been made, however, it cannot be retracted, only changed. (i.e., you cannot "unvote").
Work on Another Query
At the bottom of the evaluation page, there is a "View All Assignments" button (Fig. 8). At any time, clicking this button will direct you to your assignment page similar to the one shown in Fig. 4. When you have completed evaluating all of the candidates for this query, you may click this button to continue on another query.
Figure 8. Close up image of "View All Assignments" button which loads the assignment page.
You may also click the "My Assignments" link on the left-side menu to go to the My Assignments page.
On the My Assignments page, you may click another "Evaluate Query" button to load evaluation page (like Fig. 5) for another query.
You will see a list of all of the queries you have evaluated (or are evaluating) at the My Assignments page. You can return to any query by clicking on the "Evaluate Query" underneath it (Fig. 4). You can re-evaluate any candidate for any query at any time, up to the closing of the evaluation system.
Monitor Progress
Figure 9. Close up image of Query List buttons which allow for revisiting of completed query/candidate sets.
