2006:Evalutron6000 Walkthrough
In order to use Evalutron6000 you will need to be using a modern web browser (e.g., Firefox, Internet Explorer, Safari, Mozilla, etc) that supports JavaScript (ECMAScript) and Cookies. Evalutron has been tested on Windows XP, MacOS X, and RedHat Linux. Please you are using a different platform and having trouble, try accessing Evalutron from another machine. If you are still having difficulty, contact mrx-com09@lists.lis.uiuc.edu.
When first visiting the Evalutron6000 homepage, you will see a page similar to this.
First you must register a new account. Click on the "Register" link on the left side of the page to create an account.
The registration page is fairly straightforward. Required fields are marked in blue with an asterisks. You can create any username and password you wish. Passwords must be at least 6 characters long and are case-sensitive. Before completing the registration, you must read and agree to the terms of the Informed Consent document. The evaluation, because it is using human judgements of similarity, is considered a human-subjects research project and the Evaluatron is basically a survey instrument. If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.
After completing the Registration, the system will ask you to log-in with your newly created username and password. After logging-in, you can begin evaluation. To start the evaluation process, click on the "Start Evaluation" link on the homepage. You will be presented with several options for what media player you want to use. We have invested considerable effort coding various interfaces to be compatible with the maximum number of browsers, platforms, and players. Most users should be able to use the Flash MP3 Player option -- which we've found to work the best. The Windows Media Player and Quicktime options should also work in most modern browsers.
Once you have selected a player configuration, you'll see a query evaluation page. This page consists of two main parts - the query player and candidate players at the top, and the candidate list below. Each song (query and candidates) has been split into three 30 second clips. The first clip is taken from the first 30 seconds of the audio file (0:00 - 0:30). The second clip is the middle 30 seconds of the song (middle-0:15, middle+0:15). The third clip is the last 30 seconds of the song (end-0:30, end). Clicking on these buttons will load the song into the player and begin playing. Depending on the player you are using and your browser's configuration, you should be able to seek through the 30 second clip and replay it using the players play controls. We recommend you listen to the entire query at least once before evaluating any candidate files.
The procedure for listening to a candidate is the same as listening to a query -- click the 30-clip buttons to load the clip into the player and listen to it. Once you have a feeling for whether or not the candidate is similar to the query, click the "Similar" or "Not Similar" radio buttons to the right of the candidate. Note: you can continue to replay the query during candidate playback.
Depending on how you graded the candidate, you should see the candidate box react to your grade, indicating that the vote has been logged in the database. If you indicated the song was "Similar" the box will turn green and state "SAVED".
If you indicated the song was "Not Similar" the box will turn red and state "SAVED".
You can always change your evaluation for any candidate by toggling the radio buttons. The system should reflect any changes with an appropriate color change. Once an evaluation has been made, however, it cannot be retracted, only reversed. (i.e., you cannot "unvote").
Please note that the list of candidates scrolls within the page, there are more candidates that may be immediately visible on the page. Please scroll to the bottom of the candidate list to make sure you've evaluated each song.
When you have completed evaluating all of the candidates for a single query, you can click on the "Next Random Query" button at the bottom of the page. This button will load a new query and list of candidates for you to evaluate.
You will see a list of all of the queries you have evaluated (or are evaluating) at the bottom of the page. You can return to any query by clicking on the button for that query here. You can re-evaluate any candidate for any query at any time, up to the closing of the evaluation system.
