2007:Evalutron6000 Walkthrough For Audio Mood Classification - Revision history

Kahyun Choi: /* Understanding the Mood Clusters */

2011-12-20T04:07:47Z

‎Understanding the Mood Clusters

IMIRSELBot: Bot: Fixing image links

2010-05-14T03:23:12Z

Bot: Fixing image links

IMIRSELBot: Robot: Automated text replacement (-[[Image: +[[Image:2007_)

2010-05-14T00:28:25Z

Robot: Automated text replacement (-[[Image: +[[Image:2007_)

IMIRSELBot: Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2007:\1)

2010-05-13T18:19:40Z

Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2007:\1)

XiaoHu: /* Listening to Exemplar Clips */

2007-07-31T17:07:14Z

‎Listening to Exemplar Clips

XiaoHu: /* Evaluation Pages: Mood Clusters and Candidate Lists */

2007-07-31T03:22:20Z

‎Evaluation Pages: Mood Clusters and Candidate Lists

XiaoHu: /* Evaluation Pages: Mood Clusters and Candidate Lists */

2007-07-31T02:49:44Z

‎Evaluation Pages: Mood Clusters and Candidate Lists

CameronJones: /* Understanding the Mood Clusters */

2007-07-30T22:16:41Z

‎Understanding the Mood Clusters

CameronJones: /* Understanding the Mood Clusters */

2007-07-30T22:15:20Z

‎Understanding the Mood Clusters

JDownie: /* Validating Expectations and "Reasonableness" */

2007-07-30T22:13:53Z

‎Validating Expectations and "Reasonableness"

@@ Line 19: / Line 19: @@
 The exemplar clips have been manually selected by 6 human assessors from a set of songs in the USPOP collection. Their purpose is to help clarify the perceptual identities of the mood clusters.
-Bibliographic information of more exemplar songs can be find in [[exemplar songs]] page.
+Bibliographic information of more exemplar songs can be find in [[2007:exemplar songs]] page.
 Each of the candidate clips to be evaluated is pre-labeled with one of mood clusters, according to its metadata provided by APM (apmmusic.com). Depending on the quality of such metadata and the method we used to select the candidates, the pre-labels may or may not correct. This is why we need your judgments to validate and/or correct the labels. However, the pre-labels do provide a sense of identification of the clusters, and thus the candidates are organized by their pre-labels in the Evalutron. While you should keep suspicion towards the pre-labels, we encourage you to navigate and listen to clips in different clusters from time to time, which can help remind your ears the nuance between clusters.

@@ Line 40: / Line 40: @@
 When first visiting the Evalutron 6000 homepage, you will see a page similar to this (Fig. 1).
-[[Image:2007_E6kamc_home_page_scaled.png]]
+[[Image:2007_e6kamc_home_page_scaled.png]]
 '''Figure 1. Evalutron 6000 start page.'''
@@ Line 58: / Line 58: @@
 If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.
-[[Image:2007_E6kamc_registration_page_scaled.png]]
+[[Image:2007_e6kamc_registration_page_scaled.png]]
 '''Figure 2. Evalutron 6000 registration page.'''
@@ Line 65: / Line 65: @@
 After completing the Registration, the system will ask you to sign-in with your newly created username and password. After signing-in for the first time, you will be asked to listen to exemplar clips (Fig. 3). To start listening, click on the "HERE" on this page or "Exemplar Audio Clips" link on the homepage.
-[[Image:2007_E6kamc_initial_sign_in_scaled.png]]
+[[Image:2007_e6kamc_initial_sign_in_scaled.png]]
 '''Figure 3. Initial sign in page.'''
@@ Line 71: / Line 71: @@
 The links lead you to the Instructions page (Fig. 4). You will be presented with 3 exemplar clips for each cluster. <font color="red">You must finish listening to all the exemplar clips before the system can assign you candidates for your evaluation.</font>
-[[Image:2007_E6kamc_instruction_page_scaled.png]]
+[[Image:2007_e6kamc_instruction_page_scaled.png]]
 '''Figure 4. Instructions page.'''
@@ Line 79: / Line 79: @@
 If you haven't listened to all the examples, the system will keep directing you back to the Instructions page and tell you how many examples you have yet to listen to (Fig. 5). This is to make sure you have listend to all the examples before starting evaluation because the exemplar songs are important to help reaching cross-validator consistency.
-[[Image:2007_E6kamc_example_check_scaled.png]]
+[[Image:2007_e6kamc_example_check_scaled.png]]
 '''Figure 5. Example check page.'''
@@ Line 88: / Line 88: @@
 Once you have got your assignments, you'll see a mood cluster evaluation page (Fig. 6). On this page, you will see the mood cluster label and the list of candidate players. The candidates are drawn from the APM collection and are 30 seconds long.
-[[Image:2007_E6kamc_evaluation_page_scaled.png]]
+[[Image:2007_e6kamc_evaluation_page_scaled.png]]
 '''Figure 6. Sample evaluation page.'''
@@ Line 98: / Line 98: @@
 To listen to a candidate, click the "Play Candidate" buttons to load the clip into the player and listen to it (Fig. 6). You may need to adjust the volume of your computer as some clips have lower volume. Once you have a feeling for which mood cluster the candidate should belong to, select one mood cluster from the drop down menu on the right. Or if you feel the candidate does not fit in any of the 5 clusters, please select "Other" for it (Fig. 7).
-[[Image:2007_E6kamc_select_cluster.png]]
+[[Image:2007_e6kamc_select_cluster.png]]
 '''Figure 7. Close up image of mood cluster selection menu.'''
@@ Line 104: / Line 104: @@
 Depending on which cluster you choose for the candidate, you should see the candidate box react to your classification, indicating that the vote has been logged in the database (Fig. 8). If you indicated the song was in a cluster or "Other", the box will turn color and state "SAVED". However, if you selected <i> blank </i>, the box will remain or turn back to uncolored and the "SAVED" statement will disappear, indicating the selection was invalid.
-[[Image:2007_E6ksms_select_category_saved.png]]
+[[Image:2007_e6ksms_select_category_saved.png]]
 '''Figure 8. Close up image of mood cluster selection menu with "Cluster_2" selected and "SAVED" automatically.'''
@@ Line 114: / Line 114: @@
 After the system allocates your assignment, you will see the "My Assignment" page each time when you sign in. Please click one of the "evaluate with Flash Player" links to resume your evaluation.
-[[Image:2007_E6kamc_my_assignment_scaled.png]]
+[[Image:2007_e6kamc_my_assignment_scaled.png]]
 '''Figure 9. My Assignment page. '''
@@ Line 120: / Line 120: @@
 You can also see a list of all of the 5 Mood Clusters at the bottom of each Candidate List page. You can return to any mood cluster by clicking on the button for that mood cluster here (Fig. 10). You can re-evaluate any candidate for any mood cluster at any time, up to the closing of the evaluation system.
-[[Image:2007_E6kamc_navigation_detail.png]]
+[[Image:2007_e6kamc_navigation_detail.png]]
 '''Figure 10. Close up image of Mood Cluster List buttons which allow for navigating the pages. '''

@@ Line 40: / Line 40: @@
 When first visiting the Evalutron 6000 homepage, you will see a page similar to this (Fig. 1).
-[[Image:E6kamc_home_page_scaled.png]]
+[[Image:2007_E6kamc_home_page_scaled.png]]
 '''Figure 1. Evalutron 6000 start page.'''
@@ Line 58: / Line 58: @@
 If you have questions about your rights as a subject in this research project, you should contact the UIUC IRB office (http://www.irb.uiuc.edu) for more information. The research protocol for this project is IRB# 07066.
-[[Image:E6kamc_registration_page_scaled.png]]
+[[Image:2007_E6kamc_registration_page_scaled.png]]
 '''Figure 2. Evalutron 6000 registration page.'''
@@ Line 65: / Line 65: @@
 After completing the Registration, the system will ask you to sign-in with your newly created username and password. After signing-in for the first time, you will be asked to listen to exemplar clips (Fig. 3). To start listening, click on the "HERE" on this page or "Exemplar Audio Clips" link on the homepage.
-[[Image:E6kamc_initial_sign_in_scaled.png]]
+[[Image:2007_E6kamc_initial_sign_in_scaled.png]]
 '''Figure 3. Initial sign in page.'''
@@ Line 71: / Line 71: @@
 The links lead you to the Instructions page (Fig. 4). You will be presented with 3 exemplar clips for each cluster. <font color="red">You must finish listening to all the exemplar clips before the system can assign you candidates for your evaluation.</font>
-[[Image:E6kamc_instruction_page_scaled.png]]
+[[Image:2007_E6kamc_instruction_page_scaled.png]]
 '''Figure 4. Instructions page.'''
@@ Line 79: / Line 79: @@
 If you haven't listened to all the examples, the system will keep directing you back to the Instructions page and tell you how many examples you have yet to listen to (Fig. 5). This is to make sure you have listend to all the examples before starting evaluation because the exemplar songs are important to help reaching cross-validator consistency.
-[[Image:E6kamc_example_check_scaled.png]]
+[[Image:2007_E6kamc_example_check_scaled.png]]
 '''Figure 5. Example check page.'''
@@ Line 88: / Line 88: @@
 Once you have got your assignments, you'll see a mood cluster evaluation page (Fig. 6). On this page, you will see the mood cluster label and the list of candidate players. The candidates are drawn from the APM collection and are 30 seconds long.
-[[Image:E6kamc_evaluation_page_scaled.png]]
+[[Image:2007_E6kamc_evaluation_page_scaled.png]]
 '''Figure 6. Sample evaluation page.'''
@@ Line 98: / Line 98: @@
 To listen to a candidate, click the "Play Candidate" buttons to load the clip into the player and listen to it (Fig. 6). You may need to adjust the volume of your computer as some clips have lower volume. Once you have a feeling for which mood cluster the candidate should belong to, select one mood cluster from the drop down menu on the right. Or if you feel the candidate does not fit in any of the 5 clusters, please select "Other" for it (Fig. 7).
-[[Image:E6kamc_select_cluster.png]]
+[[Image:2007_E6kamc_select_cluster.png]]
 '''Figure 7. Close up image of mood cluster selection menu.'''
@@ Line 104: / Line 104: @@
 Depending on which cluster you choose for the candidate, you should see the candidate box react to your classification, indicating that the vote has been logged in the database (Fig. 8). If you indicated the song was in a cluster or "Other", the box will turn color and state "SAVED". However, if you selected <i> blank </i>, the box will remain or turn back to uncolored and the "SAVED" statement will disappear, indicating the selection was invalid.
-[[Image:E6ksms_select_category_saved.png]]
+[[Image:2007_E6ksms_select_category_saved.png]]
 '''Figure 8. Close up image of mood cluster selection menu with "Cluster_2" selected and "SAVED" automatically.'''
@@ Line 114: / Line 114: @@
 After the system allocates your assignment, you will see the "My Assignment" page each time when you sign in. Please click one of the "evaluate with Flash Player" links to resume your evaluation.
-[[Image:E6kamc_my_assignment_scaled.png]]
+[[Image:2007_E6kamc_my_assignment_scaled.png]]
 '''Figure 9. My Assignment page. '''
@@ Line 120: / Line 120: @@
 You can also see a list of all of the 5 Mood Clusters at the bottom of each Candidate List page. You can return to any mood cluster by clicking on the button for that mood cluster here (Fig. 10). You can re-evaluate any candidate for any mood cluster at any time, up to the closing of the evaluation system.
-[[Image:E6kamc_navigation_detail.png]]
+[[Image:2007_E6kamc_navigation_detail.png]]
 '''Figure 10. Close up image of Mood Cluster List buttons which allow for navigating the pages. '''

@@ Line 5: / Line 5: @@
 ===Validating Expectations and "Reasonableness"===
-For each candidate, we need you to assign a mood cluster label. The 5 mood clusters used in this contest are derived from the AMG (allmusicguide.com) mood labels which collectively define the clusters (see the task wiki [[Audio Music Mood Classification]] for more details):
+For each candidate, we need you to assign a mood cluster label. The 5 mood clusters used in this contest are derived from the AMG (allmusicguide.com) mood labels which collectively define the clusters (see the task wiki [[2007:Audio Music Mood Classification]] for more details):
 Cluster_1: passionate, rousing, confident, boisterous, rowdy <br>

@@ Line 75: / Line 75: @@
 '''Figure 4. Instructions page.'''
-When you finish listening to all the exemplar clips, you can proceed to the actual evaluation by clicking the "PLEASE START EVALUATION with Flash MP3 Player" link. At this time, a set of candidate clips across all the 5 clusters will be assigned to you and for you to finish before the deadline of Aug. 20.
+When you finish listening to all the exemplar clips, you can proceed to the actual evaluation by clicking the "PLEASE START EVALUATION with Flash MP3 Player" link. At this time, a set of candidate clips across all the 5 clusters will be assigned to you and for you to finish before the deadline of Aug. 15.
 If you haven't listened to all the examples, the system will keep directing you back to the Instructions page and tell you how many examples you have yet to listen to (Fig. 5). This is to make sure you have listend to all the examples before starting evaluation because the exemplar songs are important to help reaching cross-validator consistency.

@@ Line 94: / Line 94: @@
 Please do not be puzzled if your first assignment is not Mood Cluster #1 or if your Candidate List has candidates in a seemingly random order: This is deliberate! The Evalutron 6000 is designed to build customized randomized lists and orderings for each grader to minimize fatigue and ordering effects. We encourage you to jump around different Mood Clusters in the system to do your evaluations, so as to reinforce the distinctions between Mood Clusters. The "Mood Cluster #" buttons visible in Figure 6 are designed to guide you through differnt Mood Cluster Lists.
-Please note that the list of candidates is longer than the visible page (i.e., there are more candidates than may be immediately visible on the page). Please scroll to the bottom of the candidate list to make sure you have evaluated each song.
+Please note that the list of candidates is longer than the visible page (i.e., there are more candidates than may be immediately visible on the page). Please scroll to the bottom of the candidate list to make sure you have evaluated each song. You can always review the exemplar clips during your evaluation by clicking "listen to exemplars again" link at the left side of each candidate box (Fig. 6).
 To listen to a candidate, click the "Play Candidate" buttons to load the clip into the player and listen to it (Fig. 6). You may need to adjust the volume of your computer as some clips have lower volume. Once you have a feeling for which mood cluster the candidate should belong to, select one mood cluster from the drop down menu on the right. Or if you feel the candidate does not fit in any of the 5 clusters, please select "Other" for it (Fig. 7).

@@ Line 16: / Line 16: @@
 ===Understanding the Mood Clusters===
-We urge you to listen to the <font color="red">exemplar clips </font> in each mood cluster located in the Instruction page of the Evalutron before you start and apply a level of "reasonableness" to your judgments. And whenever you feel in need, please go back to this page and refresh your ears.
+We urge you to listen to the '''exemplar clips''' in each mood cluster located in the Instruction page of the Evalutron before you start and apply a level of "reasonableness" to your judgments. And whenever you feel in need, please go back to this page and refresh your ears.
-The exemplar clips are manually selected by 6 human assessors from a set of songs in the USPOP collection. Their purpose is to help clarify the perceptual identities of the mood clusters.
+The exemplar clips have been manually selected by 6 human assessors from a set of songs in the USPOP collection. Their purpose is to help clarify the perceptual identities of the mood clusters.
 Bibliographic information of more exemplar songs can be find in [[exemplar songs]] page.
-Each of the candidate clip to be evaluated is pre-labeled with one of mood clusters, according to its metadata provided by APM (apmmusic.com). Depending on the quality of such metadata and the method we used to select the candidates, the pre-labels may or may not correct. This is why we need your judgments to validate and/or correct the labels. However, the pre-labels do provide a sense of identification of the clusters, and thus the candidates are organized by their pre-labels in the Evalutron. While you should keep suspicion towards the pre-labels, we encourage you to navigate and listen to clips in different clusters from time to time, which can help remind your ears the nuance between clusters.
+Each of the candidate clips to be evaluated is pre-labeled with one of mood clusters, according to its metadata provided by APM (apmmusic.com). Depending on the quality of such metadata and the method we used to select the candidates, the pre-labels may or may not correct. This is why we need your judgments to validate and/or correct the labels. However, the pre-labels do provide a sense of identification of the clusters, and thus the candidates are organized by their pre-labels in the Evalutron. While you should keep suspicion towards the pre-labels, we encourage you to navigate and listen to clips in different clusters from time to time, which can help remind your ears the nuance between clusters.
 Evalutron 6000 design and use details follow below and should clarify for you the use of the system.