2010:Evalutron6000 Walkthrough

UPDATE 14 July 2010

The 2010 Evalutron has a new look and implementation. It is easier to use than the original implementation. However, it is necessary to read through this document before you start using the system. Enjoy!

Welcome to the Evalutron 6000

In order to use the Evalutron 6000 you will need to be using a modern web browser (e.g., Firefox, Internet Explorer, Safari, Mozilla, etc) that supports JavaScript (ECMAScript) and Cookies. Evalutron has been tested on Windows XP, MacOS X, and RedHat Linux. If you are using a different platform and having trouble, please try accessing Evalutron 6000 from another machine. If you are still having difficulty, contact

mirproject@lists.lis.uiuc.edu.

When first visiting the Evalutron 6000 homepage, you will see a page similar to this (Fig. 1).

Figure 1. Evalutron 6000 home page.

If you have an account with the submission system, then you can use the same account for Evalutron 6000. Otherwise, you must register a new account. Click on the "Register" link on the left side of the page to create an account.

The registration page is fairly straightforward (Fig. 2). All fields are required. You can create any username and password you wish. Username must be at least 5 characters long and passwords must be at least 8 characters long and are case-sensitive. Before completing the registration, you will receive an email with an activation link. Clicking that link will complete your registration.

Figure 2. Evalutron 6000 registration page.

Before starting evaluation, every evaluator must read and agree to the terms of the Informed Consent document. Otherwise, you will be redirect to the informed consent page when you try to get evaluation assignments. Clicking the "Informed Consent" link on the left-side menu will show the form (Fig. 3). The evaluation, because it is using human judgments of similarity, is considered a human-subjects research project and the Evaluatron is basically a survey instrument. To indicate your consent to participate in the evaluation, scroll down the page and check the "I Agree" checkbox below the informed consent document.

If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.

Figure 3. Evalutron 6000 informed consent page.

To start the evaluation process, click the "My Assignments" link on the left-side menu. The assignment page is similar to Fig. 4. This page shows all tasks available in the Evalutron 6000, and initially there is no assignment given. Be careful to select the task you intend to participate, and click the "Get Assignment" button under this task. The system will assign you a number of queries to evaluate. Clicking the "Evaluate Query" button will lead you to the evaluation page (Fig. 5).

Please note once assignments are made, they cannot be changed and you are responsible to finish evaluations assigned to you. Therefore, please NEVER click "Get Assignment" for the task you do not intend to participate.

Figure 4. Evalutron 6000 get assignment page.

Once you have selected a player configuration, you'll see a query evaluation page (Fig. 4). This page consists of two main parts - the query player and candidate players at the top, and the candidate list below. Each song (query and candidates) has been split into three 30 second clips. The first clip is taken from the first 30 seconds of the audio file (0:00 - 0:30). The second clip is the middle 30 seconds of the song (middle-0:15 - middle+0:15). The third clip is the last 30 seconds of the song (end-0:30 - end). Clicking on these buttons will load the song into the player and begin playing. Depending on the player you are using and your browser's configuration, you should be able to seek through the 30 second clip and replay it using the player's play controls. We recommend you listen to the entire query at least once before evaluating any candidate files.

Figure 4. Sample evaluation page.

Please note that the list of candidates scrolls within the page, there are more candidates than may be immediately visible on the page. Please scroll to the bottom of the candidate list to make sure you've evaluated each song.

The procedure for listening to a candidate is the same as listening to a query -- click the 0:30-clip buttons to load the clip into the player and listen to it (Fig. 5). Once you have a feeling for whether or not the candidate is similar to the query, click the "Similar" or "Not Similar" radio buttons to the right of the candidate. Note: you can continue to replay the query during candidate playback.

Figure 5. Close up image of candidate manipulation buttons.

Depending on how you graded the candidate, you should see the candidate box react to your grade, indicating that the vote has been logged in the database (Fig. 6). If you indicated the song was "Similar" the box will turn green and state "SAVED".

Figure 6. Close up image of candidate manipulation buttons with "Similar" selection "SAVED".

If you indicated the song was "Not Similar" the box will turn red and state "SAVED" (Fig. 7).

Figure 7. Close up image of candidate manipulation buttons with "Not Similar" selection "SAVED".

You can always change your evaluation for any candidate by toggling the radio buttons. The system should reflect any changes with an appropriate color change. Once an evaluation has been made, however, it cannot be retracted, only reversed. (i.e., you cannot "unvote").

When you have completed evaluating all of the candidates for a single query, you can click on the "Next Random Query" button at the bottom of the page (Fig. 8). This button will load a new query and list of candidates for you to evaluate.

Figure 8. Close up image of "Next Random Query" button which generates next set of candidates for evaluation.

You will see a list of all of the queries you have evaluated (or are evaluating) at the bottom of the page. You can return to any query by clicking on the button for that query here (Fig. 9). You can re-evaluate any candidate for any query at any time, up to the closing of the evaluation system.

Figure 9. Close up image of Query List buttons which allow for revisiting of completed query/candidate sets.