Difference between revisions of "2006:Evalutron6000 Walkthrough"

From MIREX Wiki
m (Evalutron6000 walkthrough moved to Evalutron6000 Walkthrough)
m
Line 7: Line 7:
 
First you must register a new account. Click on the "Register" link on the left side of the page to create an account.
 
First you must register a new account. Click on the "Register" link on the left side of the page to create an account.
  
The registration page is fairly straightforward. Required fields are marked in blue with an asterisks. You can create any username and password you wish. Passwords must be at least 6 characters long and are case-sensitive. Before completing the registration, you must read and agree to the terms of the Informed Consent document. The evaluation, because it is using human judgements of similarity, is considered a human-subjects research project and the Evaluatron is basically a survey instrument. If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.
+
The registration page is fairly straightforward. Required fields are marked in blue with an asterisks. You can create any username and password you wish. Passwords must be at least 6 characters long and are case-sensitive. Before completing the registration, you must read and agree to the terms of the Informed Consent document. The evaluation, because it is using human judgements of similarity, is considered a human-subjects research project and the Evaluatron is basically a survey instrument. To indicate your consent to participate in the evaluation, check the "I Agree" checkbox below the informed consent document.
 +
 
 +
If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.
  
 
[[Image:Eval6_register.png]]
 
[[Image:Eval6_register.png]]

Revision as of 14:41, 22 August 2006

In order to use Evalutron6000 you will need to be using a modern web browser (e.g., Firefox, Internet Explorer, Safari, Mozilla, etc) that supports JavaScript (ECMAScript) and Cookies. Evalutron has been tested on Windows XP, MacOS X, and RedHat Linux. If you are using a different platform and having trouble, please try accessing Evalutron6K from another machine. If you are still having difficulty, contact mrx-com09@lists.lis.uiuc.edu.

When first visiting the Evalutron6K homepage, you will see a page similar to this.

File:Eval6 home.png

First you must register a new account. Click on the "Register" link on the left side of the page to create an account.

The registration page is fairly straightforward. Required fields are marked in blue with an asterisks. You can create any username and password you wish. Passwords must be at least 6 characters long and are case-sensitive. Before completing the registration, you must read and agree to the terms of the Informed Consent document. The evaluation, because it is using human judgements of similarity, is considered a human-subjects research project and the Evaluatron is basically a survey instrument. To indicate your consent to participate in the evaluation, check the "I Agree" checkbox below the informed consent document.

If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.

File:Eval6 register.png

After completing the Registration, the system will ask you to log-in with your newly created username and password. After logging-in, you can begin evaluation. To start the evaluation process, click on the "Start Evaluation" link on the homepage. You will be presented with several options for what media player you want to use. We have invested considerable effort coding various interfaces to be compatible with the maximum number of browsers, platforms, and players. Most users should be able to use the Flash MP3 Player option -- which we've found to work the best. The Windows Media Player and Quicktime options should also work in most modern browsers.

File:Eval6 start.png

Once you have selected a player configuration, you'll see a query evaluation page. This page consists of two main parts - the query player and candidate players at the top, and the candidate list below. Each song (query and candidates) has been split into three 30 second clips. The first clip is taken from the first 30 seconds of the audio file (0:00 - 0:30). The second clip is the middle 30 seconds of the song (middle-0:15, middle+0:15). The third clip is the last 30 seconds of the song (end-0:30, end). Clicking on these buttons will load the song into the player and begin playing. Depending on the player you are using and your browser's configuration, you should be able to seek through the 30 second clip and replay it using the players play controls. We recommend you listen to the entire query at least once before evaluating any candidate files.

File:Eval6 eval page.png

The procedure for listening to a candidate is the same as listening to a query -- click the 30-clip buttons to load the clip into the player and listen to it. Once you have a feeling for whether or not the candidate is similar to the query, click the "Similar" or "Not Similar" radio buttons to the right of the candidate. Note: you can continue to replay the query during candidate playback.

File:Eval6 cand detail.png

Depending on how you graded the candidate, you should see the candidate box react to your grade, indicating that the vote has been logged in the database. If you indicated the song was "Similar" the box will turn green and state "SAVED".

File:Eval6 sim saved.png

If you indicated the song was "Not Similar" the box will turn red and state "SAVED".

File:Eval6 notsim saved.png

You can always change your evaluation for any candidate by toggling the radio buttons. The system should reflect any changes with an appropriate color change. Once an evaluation has been made, however, it cannot be retracted, only reversed. (i.e., you cannot "unvote").

Please note that the list of candidates scrolls within the page, there are more candidates that may be immediately visible on the page. Please scroll to the bottom of the candidate list to make sure you've evaluated each song.

When you have completed evaluating all of the candidates for a single query, you can click on the "Next Random Query" button at the bottom of the page. This button will load a new query and list of candidates for you to evaluate.

File:Eval6 next button.png

You will see a list of all of the queries you have evaluated (or are evaluating) at the bottom of the page. You can return to any query by clicking on the button for that query here. You can re-evaluate any candidate for any query at any time, up to the closing of the evaluation system.

File:Eval6 queries detail.png