Grand Challenge 2016: User Experience (GC16UX)
Contents
Purpose
Holistic, user-centered evaluation of the user experience in interacting with complete, user-facing music information retrieval (MIR) systems.
Goals
- To inspire the development of complete MIR systems.
- To promote the notion of user experience as a first-class research objective in the MIR community.
Dataset
A set of music 10,000 music audio tracks is provided for the GC16UX. It will be a subset of tracks drawn from the Jamendo collection's CC-BY licensed works.
The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.
The 10,000 tracks in GC16UX are sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems. It represents a randomly chosen subset the content available at Jamendo that is published under the terms of the Creative Commons Attribution-Non-Commercial-ShareAlike (by-nc-sa), where user-supplied data has tagged a track with 1 or more genre categories. Importantly, we have created a new dataset, which has better distribution of genres and artists in response to feedback from last year's evaluation. For more details about usage of this dataset, see the LICENSE.txt file contained in the downloaded files.
The dataset contains the MP3 tracks and the metadata the Jamendo site publishes on the respective items (represented in JSON format), retrieved using the site's API (6th July 2015). The dataset is available both zipped up and as a tar-ball (you only need one of these); however, at 60+ Gb it is a non-trival size of file to download over the web, and so we suggest you install a Download Manager extension to your browser if you do not already have one and make use of that. In a test using the DownThemAll! extension to Firefox, downloading the dataset between University of Illinois at Urbana-Champaign and Waikato University in New Zealand took a little under 2 hours.
You need to register to download the main dataset. If you have already submitted something to MIREX, your previous registration should still work. Please note that you need to login as a submitter. Once logged in as a submitter, you can find a download link on the left.
Metadata Extracted from JSON Files
The JSON files retrieved from Jamendo site contain various metadata:
- album_id
- album_image
- album_name
- artist_id
- artist_idstr
- artist_name
- audio
- audiodownload
- duration
- id
- license_ccurl
- musicinfo_lang
- musicinfo_speed
- musicinfo_acousticelectric
- musicinfo_vocalinstrumental
- musicinfo_gender
- musicinfo_tags_vartags
- musicinfo_tags_genres
- musicinfo_tags_instruments
- name
- position
- releasedate
- shareurl
- shorturl
JSON Metadata presents statistics and plots of selected fields.
Participating Systems
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC16UX team.
To ensure a consistent experience, evaluators will see participating systems in fixed size window: 1024x768. Please test your system for this screen size.
See the #Evaluation Webforms below for a better understanding of our E6K-inpsired evaluation system design.
Potential Participants
Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
| (Cool) Team Name | Name(s) | Email(s) |
|---|---|---|
Evaluation
As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.
Criteria
Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.
Given the GC16UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evaluators' cognitive load.
Evaluators will rate systems on the following criteria:
- Overall satisfaction: How would you rate your overall satisfaction with the system?
Very unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Very satisfactory
- Aesthetics: How would you rate the visual attractiveness of the system?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent
- Ease of use: How easy was it to figure out how to use the system?
Very difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Very easy
- Clarity: How well does the system communicate what is going on?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent
- Affordances: How well does the system allow you to perform what you want to do?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent
- Performance: Does the system work efficiently and without bugs/glitches?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent
- Open Text Feedback: An open-ended question is provided for evaluators to give feedback if they wish to do so.
Evaluators
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The #Evaluation Webforms developed by the GC16UX team will ensure all participating systems will get equal number of evaluators.
Task for evaluators
To motivate the evaluators, a defined yet open task is given to the evaluators:
You need to put together a playlist for a particular event (e.g., dinner party at your house, workout session). Try to use the assigned system to make playlists for at least a couple of different events.
The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evaluators' purposes ("music for their own situation"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects. This allows great flexibility and virtually unlimited possibility in system design.
Another important consideration in designing the task is the music collection available for this GC16UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.
Evaluation results
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.
Evaluation Webforms
Graders can take as many assignments as they wish in the My Assignments page. They are allowed to go back to the evaluation page anytime by clicking the thumbnail of the submission.
To facilitate the evaluators and minimize their burden, the GC16UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.
Organization
Important Dates
This year GC16UX adopts the two-phase model with two evaluations. The first phase will end by the ISMIR conference and we will disclose preliminary results at the conference like what we did last year. Then, phase II will start. Participating developers can continue improving their systems based on the feedback from the first phase and another round of evaluation will be conducted in February. We believe that this model serves the developers well since it is in accordance with the iterative nature of user-centered design. In this way, the developers will also have enough time to develop their complete MIR systems.
- July 15th: announce the GC
- Sep. 21st: the first deadline for system submission
- Feb. 21st: the second deadline for system submission
Updates for the first round
- Jan 4th: extended deadline for system submission
- Feb 3rd: start the evaluation
- Feb 24th: close the evaluation system
- MIREX and GC session in ISMIR2016: announce the results and discussion
What to Submit
A URL to the participanting system.
Contacts
The GC16UX team consists of:
- J. Stephen Downie, University of Illinois (MIREX director)
- Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
- Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
- Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
- David Bainbridge, Waikato University, New Zealand
- Kahyun Choi, University of Illinois
- Peter Organisciak, University of Illinois
- Yun Hao, University of Illinois
Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [1] or anyone in the team.
