2006:Evalutron6000 Walkthrough For Audio Music Similarity and Retrieval

UPDATE 23 August 2006

As of 23 August there is good chance the some of the finer details of the Evalutron 6000 will change in response to community input. One probable change is that the evaluation choices will expand beyond the binary to a tenary scheme for some of the evaluation. Will modify the page once confirmed. Cheers, J. Stephen Downie

Welcome to the Evalutron 6000

In order to use the Evalutron 6000 you will need to be using a modern web browser (e.g., Firefox, Internet Explorer, Safari, Mozilla, etc) that supports JavaScript (ECMAScript) and Cookies. Evalutron has been tested on Windows XP, MacOS X, and RedHat Linux. If you are using a different platform and having trouble, please try accessing Evalutron 6000 from another machine. If you are still having difficulty, contact

mrx-com09@lists.lis.uiuc.edu.

When first visiting the Evalutron 6000 homepage, you will see a page similar to this (Fig. 1).

Figure 1. Evalutron 6000 start page.

First you must register a new account. Click on the "Register" link on the left side of the page to create an account.

The registration page is fairly straightforward (Fig. 2). Required fields are marked in blue with an asterisks. You can create any username and password you wish. Passwords must be at least 6 characters long and are case-sensitive. Before completing the registration, you must read and agree to the terms of the Informed Consent document. The evaluation, because it is using human judgements of similarity, is considered a human-subjects research project and the Evaluatron is basically a survey instrument. To indicate your consent to participate in the evaluation, check the "I Agree" checkbox below the informed consent document.

If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.

Figure 2. Evalutron 6000 start page.

After completing the Registration, the system will ask you to log-in with your newly created username and password. After logging-in, you can begin evaluation (Fig. 3). To start the evaluation process, click on the "Start Evaluation" link on the homepage. You will be presented with several options for what media player you want to use. We have invested considerable effort coding various interfaces to be compatible with the maximum number of browsers, platforms, and players. Most users should be able to use the Flash MP3 Player option -- which we've found to work the best. The Windows Media Player and Quicktime options should also work in most modern browsers.

Figure 3. Audio player selection page.

Once you have selected a player configuration, you'll see a query evaluation page (Fig. 4). This page consists of two main parts - the query player and candidate players at the top, and the candidate list below. Each song (query and candidates) has been split into three 30 second clips. The first clip is taken from the first 30 seconds of the audio file (0:00 - 0:30). The second clip is the middle 30 seconds of the song (middle-0:15 - middle+0:15). The third clip is the last 30 seconds of the song (end-0:30 - end). Clicking on these buttons will load the song into the player and begin playing. Depending on the player you are using and your browser's configuration, you should be able to seek through the 30 second clip and replay it using the player's play controls. We recommend you listen to the entire query at least once before evaluating any candidate files.

Figure 4. Sample evaluation page.

Please note that the list of candidates scrolls within the page, there are more candidates than may be immediately visible on the page. Please scroll to the bottom of the candidate list to make sure you've evaluated each song.

The procedure for listening to a candidate is the same as listening to a query -- click the 0:30-clip buttons to load the clip into the player and listen to it (Fig. 5). Once you have a feeling for whether or not the candidate is similar to the query, click the "Similar" or "Not Similar" radio buttons to the right of the candidate. Note: you can continue to replay the query during candidate playback.

Figure 5. Close up image of candidate manipulation buttons.

Depending on how you graded the candidate, you should see the candidate box react to your grade, indicating that the vote has been logged in the database (Fig. 6). If you indicated the song was "Similar" the box will turn green and state "SAVED".

Figure 6. Close up image of candidate manipulation buttons with "Similar" selection "SAVED".

If you indicated the song was "Not Similar" the box will turn red and state "SAVED" (Fig. 7).

Figure 7. Close up image of candidate manipulation buttons with "Not Similar" selection "SAVED".

You can always change your evaluation for any candidate by toggling the radio buttons. The system should reflect any changes with an appropriate color change. Once an evaluation has been made, however, it cannot be retracted, only reversed. (i.e., you cannot "unvote").

When you have completed evaluating all of the candidates for a single query, you can click on the "Next Random Query" button at the bottom of the page (Fig. 8). This button will load a new query and list of candidates for you to evaluate.

Figure 8. Close up image of "Next Random Query" button which generates next set of candidates for evaluation.

You will see a list of all of the queries you have evaluated (or are evaluating) at the bottom of the page. You can return to any query by clicking on the button for that query here (Fig. 9). You can re-evaluate any candidate for any query at any time, up to the closing of the evaluation system.

Figure 9. Close up image of Query List buttons which allow for revisiting of completed query/candidate sets.