2007:Evalutron6000 Walkthrough For Audio Mood Classification

Special Comments about Validating the Audio Mood Classification Candidates

Task Description: Ignoring Lyrics

The goal of Audio Music Mood Classification is to evaluate how well various algorithms can classify music pieces according to the mood they express. You will find in the candidate files a variety of different genres, melodies, instrumentations and tempos. We need you to look at the whole effect of the music pieces. In case of vocal music, please ignore the lyrics as it is unreseasonable to expect algorithms to tell the mood by understanding the lyrics.

Validating Expectations and "Reasonableness"

For each candidate, we need you to assign a mood cluster label. The 5 mood clusters used in this contest are derived from the AMG (allmusicguide.com) mood labels which collectively define the clusters (see the task wiki 2007:Audio Music Mood Classification for more details):

Cluster_1: passionate, rousing, confident, boisterous, rowdy
Cluster_2: rollicking, cheerful, fun, sweet, amiable/good natured
Cluster_3: literate, poignant, wistful, bittersweet, autumnal, brooding
Cluster_4: humorous, silly, campy, quirky, whimsical, witty, wry
Cluster_5: aggressive, fiery,tense/anxious, intense, volatile, visceral

If you don't think a candidate fits in any mood cluster, you may choose "Other" for it. Please bear in mind that your reasonable judgments are very important -- only those candidates on which majority of human judges reach agreement will be used in the task.

Understanding the Mood Clusters

We urge you to listen to the exemplar clips in each mood cluster located in the Instruction page of the Evalutron before you start and apply a level of "reasonableness" to your judgments. And whenever you feel in need, please go back to this page and refresh your ears.

The exemplar clips have been manually selected by 6 human assessors from a set of songs in the USPOP collection. Their purpose is to help clarify the perceptual identities of the mood clusters. Bibliographic information of more exemplar songs can be find in 2007:exemplar songs page.

Each of the candidate clips to be evaluated is pre-labeled with one of mood clusters, according to its metadata provided by APM (apmmusic.com). Depending on the quality of such metadata and the method we used to select the candidates, the pre-labels may or may not correct. This is why we need your judgments to validate and/or correct the labels. However, the pre-labels do provide a sense of identification of the clusters, and thus the candidates are organized by their pre-labels in the Evalutron. While you should keep suspicion towards the pre-labels, we encourage you to navigate and listen to clips in different clusters from time to time, which can help remind your ears the nuance between clusters.

Evalutron 6000 design and use details follow below and should clarify for you the use of the system.

Coming and Going: Avoiding Grader Fatigue

We plan to assign each evaluator 50 candidates in each mood clusters -- 250 clips of 30 seconds in total. Listening to, and then labeling, hundreds of audio files is very tiring. We have built into the back-end of the Evalutron 6000 system a rather robust database system that records in near-real-time your category selections. These selections are saved along with information about which candidates you have yet to review. All this information is stored in association with your personal sign-in ID. This means you can break up your grading over several days at times convenient and productive to you. In fact, we recommend that you not try to tackle your "assignment" in one big chunk as fresh ears are happy ears and happy ears make for better evaluations.

Basic Technical Requirements

In order to use the Evalutron 6000 you will need a modern web browser (e.g., Firefox, Mozilla, Safari, Internet Explorer ) that supports JavaScript (ECMAScript) and Cookies. Evalutron has been tested on Windows XP, MacOS X, RedHat Linux, and Solaris. We have found that the combination of Firefox and Flash Player to be most stable across platforms. Despite our best efforts, the use of Internet Explorer continues to generate seemingly random errors and we ask that you avoid using Internet Explorer. A decent amount of bandwidth is also advisable to minimize download times (i.e., DSL, cable modem and better).

A screen resolution of 1024 X 768 the minimal advised. Higher resolutions appear to work well too. We have noted some odd behaviour if you adjust your resolution in mid-session, so it is probably best not to do this. Adjusting font size via you browser can sometimes make things look a bit tidier and does not seem to have adverse effects.

In general, if you are having trouble, please try accessing the Evalutron 6000 using another machine/platform/browser combination. If you are still having difficulty, please contact

[1].

Getting Started

When first visiting the Evalutron 6000 homepage, you will see a page similar to this (Fig. 1).

Figure 1. Evalutron 6000 start page.

Four Important Issues Before Proceeding

1. This system is NOT open to the general public. Due to the legalities imposed upon us by the University of Illinois and the US Federal Government, we are allowed to accept only those graders that have a stake in Music Information Retrieval and Music Digital Library research. Acceptable persons include MIREX participants, subscribers to the music-ir[AT]ircam.fr list, ISMIR attendees, computational musicologists, music technology researchers, etc. If you are in doubt about your acceptability, please contact Prof. Downie at [2].
2. Do NOT create an account for yourself if you are merely curious to see what the Evalutron 6000 is set up to do. The account creation process scientifically distributes Query and Candidate sets to each account holder. The act of creating a "curiousity account" seriously disrupts the adminstration of the results data we are collecting.
3. IF YOU ARE REALLY CURIOUS: We have created a small "sandbox" version of the AudioSim Evalutron with fake data at: https://music-ir.org/eval6000/sandbox. You still must go through all the official procedures but your scores wil not affect anything as the data is "made up".
4. Please do NOT create multiple accounts for yourself for this particular evaluation task. If you are also grading the Audio Music Similarity and Retrieval (AudioSim) task or Symbolic Melody Similarity (SMS) task, you will be creating another task-specific account for AudioSim or SMS in the AudioSim or SMS space. The system is designed to track individuals based upon unique sign-in IDs. The creation of multiple accounts for one person causes the system to improperly distribute the assignments to your fellow graders. If you have difficulties, such as forgetting your sign-in password, please contact us at [3] or try the "Forgot Your Password?" link found at the top of the homepage (Fig 1).

Registration

First you must register a new account. Click on the "Register" link (listed under Step 2) on the page to create an account.

The registration page is fairly straightforward (Fig. 2). Required fields are marked in blue with asterisks. You can create any username and password you wish. Passwords must be at least 6 characters long and are case-sensitive. Before completing the registration, you must read and agree to the terms of the Informed Consent document. The evaluation, because it is using human judgments of music mood, is considered a human-subjects research project and the Evalutron is basically a survey instrument. To indicate your consent to participate in the evaluation, check the "I Agree" checkbox below the informed consent document.

If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.

Figure 2. Evalutron 6000 registration page.

Listening to Exemplar Clips

After completing the Registration, the system will ask you to sign-in with your newly created username and password. After signing-in for the first time, you will be asked to listen to exemplar clips (Fig. 3). To start listening, click on the "HERE" on this page or "Exemplar Audio Clips" link on the homepage.

Figure 3. Initial sign in page.

The links lead you to the Instructions page (Fig. 4). You will be presented with 3 exemplar clips for each cluster. You must finish listening to all the exemplar clips before the system can assign you candidates for your evaluation.

Figure 4. Instructions page.

When you finish listening to all the exemplar clips, you can proceed to the actual evaluation by clicking the "PLEASE START EVALUATION with Flash MP3 Player" link. At this time, a set of candidate clips across all the 5 clusters will be assigned to you and for you to finish before the deadline of Aug. 15.

If you haven't listened to all the examples, the system will keep directing you back to the Instructions page and tell you how many examples you have yet to listen to (Fig. 5). This is to make sure you have listend to all the examples before starting evaluation because the exemplar songs are important to help reaching cross-validator consistency.

Figure 5. Example check page.

After you listened to the examples, you have a chance to decide not to take part in the evaluation. Although we regret losing a valuable contributor, we would appreciate you leave without proceeding to the next stage, because otherwise the system will take you into consideration when distributing candidates. Although you may stop and leave at any time during the evaluation, we would appreciate you try to finish all (or most of) your assignments so as to make sure that each valid human judge shares equal working load.

Evaluation Pages: Mood Clusters and Candidate Lists

Once you have got your assignments, you'll see a mood cluster evaluation page (Fig. 6). On this page, you will see the mood cluster label and the list of candidate players. The candidates are drawn from the APM collection and are 30 seconds long.

Figure 6. Sample evaluation page.

Please do not be puzzled if your first assignment is not Mood Cluster #1 or if your Candidate List has candidates in a seemingly random order: This is deliberate! The Evalutron 6000 is designed to build customized randomized lists and orderings for each grader to minimize fatigue and ordering effects. We encourage you to jump around different Mood Clusters in the system to do your evaluations, so as to reinforce the distinctions between Mood Clusters. The "Mood Cluster #" buttons visible in Figure 6 are designed to guide you through differnt Mood Cluster Lists.

Please note that the list of candidates is longer than the visible page (i.e., there are more candidates than may be immediately visible on the page). Please scroll to the bottom of the candidate list to make sure you have evaluated each song. You can always review the exemplar clips during your evaluation by clicking "listen to exemplars again" link at the left side of each candidate box (Fig. 6).

To listen to a candidate, click the "Play Candidate" buttons to load the clip into the player and listen to it (Fig. 6). You may need to adjust the volume of your computer as some clips have lower volume. Once you have a feeling for which mood cluster the candidate should belong to, select one mood cluster from the drop down menu on the right. Or if you feel the candidate does not fit in any of the 5 clusters, please select "Other" for it (Fig. 7).

Figure 7. Close up image of mood cluster selection menu.

Depending on which cluster you choose for the candidate, you should see the candidate box react to your classification, indicating that the vote has been logged in the database (Fig. 8). If you indicated the song was in a cluster or "Other", the box will turn color and state "SAVED". However, if you selected blank , the box will remain or turn back to uncolored and the "SAVED" statement will disappear, indicating the selection was invalid.

Figure 8. Close up image of mood cluster selection menu with "Cluster_2" selected and "SAVED" automatically.

You can always change your evaluation for any candidate by reselect from the drop down menu.

When you have completed evaluating all of the candidates for a mood cluster, you can click on the "Mood Cluster #" buttons at the top right of the evaluation page. The buttons will load another Mood Cluster and associated Candidate List for you to evaluate. Using the "My Assignment" tab, you can check the list of mood clusters and candidates that were assigned to you, and also see how far you are in completing your evaluation task (Fig. 9). You must choose one cluster or "Other" for each candidate before the system registers it as a "completed" candidate.

After the system allocates your assignment, you will see the "My Assignment" page each time when you sign in. Please click one of the "evaluate with Flash Player" links to resume your evaluation.

Figure 9. My Assignment page.

You can also see a list of all of the 5 Mood Clusters at the bottom of each Candidate List page. You can return to any mood cluster by clicking on the button for that mood cluster here (Fig. 10). You can re-evaluate any candidate for any mood cluster at any time, up to the closing of the evaluation system.

Figure 10. Close up image of Mood Cluster List buttons which allow for navigating the pages.