2014:GC14UX

Purposes

Holistic evaluation of user experience in interacting with user-serving MIR systems

Goals

1) to inspire the development of complete MIR systems 2) to promote the notion of user experience as a first-class research objective in the MIR community

Dataset

A set of music audio of 10,000 tracks is provided for the GC14UX. It will be a subset of tracks drawn from Jamendo collection's CC-BY licensed works (http://www.jamendo.com/en/welcome).

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the CC-BY license (http://creativecommons.org/licenses/by/3.0/). The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.

Participating Systems

Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

Evaluation

Task

To ensure that the GC14UX does not become a system-centered evaluation in disguise, the process will remain as agnostic as possible concerning the technological means by which participating systems create and deliver their experiences to the users. The only requirement for the systems is they should support music discovery (in whichever ways). To design some criteria that’s not tied to the music content;

Task: You are creating a YouTube video and you need to find some open-source music to use.

Q. What kind of video you are trying to find the music for? (Need to discuss more about this.)

- Overall satisfaction (“On a scale of 0 to 100, rate your overall satisfaction with this system”) and bands with different colors (0-20: very unsatisfied; 20-40: somewhat unsatisfied; 40-60: neutral; 60-80: somewhat satisfied; 80-100: very satisfied).

- Aesthetics (look of the interface) - Interaction (smoothness, no stalls or bugs) - Awareness (what’s going on, where the user is)

Ten Heuristics here: http://www.nngroup.com/articles/ten-usability-heuristics/

       How to convert the 10 heuristics 

Any literature support for the three items? Textbox

• . Evaluation criterion:

Satisfaction: How much do you like using the interface? Memorability: Errors: How many errors did you make? How easy is it to recover from them? Learnability: How easy was it to figure out the interface? Does its layout match what you expect (e.g. buttons, search box locations, navigation)? Was anything confusing? How good is the interface at communicating functionality (help and documentation)? ---

This simplicity is because: 1) the GC14UX is all about how users perceive their experiences of the systems. We intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. 2) more data capturing opportunities will distract from the real user experience.

An open-ended question is provided but is optional for users to give feedback if they wish to do so.

Evaluation mechanism

The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

To prompt a scenario/context in which a user interacts with participating systems, a half-defined sentence will be provided to a user in the evaluation webforms *before* a set of participating systems are shown to the user:

“I am looking for music for ______________”

The user is free to fill in any need s/he would like to find music for, such as “drinking a cup of coffee”, or “waiting for my girlfriend”. In this way, the goal of the user is captured and the goal is as authentic to the user as possible.

Evaluators

They will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. It is planned that each submitted system will be evaluated by 50 to 60 evaluators. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

Evaluation results

Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

Certificates

The GC14UX will present certificates to all participating systems and the system(s) with the highest mean score.

Wireframes