MIREX Wiki - User contributions [en]

2015:GC15UX

2015-09-16T00:55:58Z

Xiao: /* Important Dates */

{{DISPLAYTITLE:Grand Challenge 2015: User Experience (GC15UX)}}
=Purpose=
Holistic, user-centered evaluation of the user experience in interacting with complete, user-facing music information retrieval (MIR) systems.

=Goals=
# To inspire the development of complete MIR systems.
# To promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC15UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

The 10,000 tracks in GC15UX are sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems. It represents a randomly chosen subset the content available at Jamendo that is published under the terms of the Creative Commons Attribution-Non-Commercial-ShareAlike (by-nc-sa), where user-supplied data has tagged a track with 1 or more genre categories. Importantly, we have created a new dataset, which has better distribution of genres and artists in response to feedback from last year's evaluation. For more details about usage of this dataset, see the LICENSE.txt file contained in the downloaded files.

The dataset contains the MP3 tracks and the metadata the Jamendo site publishes on the respective items (represented in JSON format), retrieved using the site's API (6th July 2015). The dataset is available both zipped up and as a tar-ball (you only need one of these); however, at 60+ Gb it is a non-trival size of file to download over the web, and so we suggest you install a Download Manager extension to your browser if you do not already have one and make use of that. In a test using the DownThemAll! extension to Firefox, downloading the dataset between University of Illinois at Urbana-Champaign and Waikato University in New Zealand took a little under 2 hours.

You need to [https://www.music-ir.org/mirex/gc15ux/ register] to download the main dataset. If you have already submitted something to MIREX, your previous registration should still work. Please note that you need to login as a submitter. Once logged in as a submitter, you can find a download link on the left.

==Metadata Extracted from JSON Files==
The JSON files retrieved from Jamendo site contain various metadata:

#album_id
#album_image
#album_name
#artist_id
#artist_idstr
#artist_name
#audio
#audiodownload
#duration
#id
#license_ccurl
#musicinfo_lang
#musicinfo_speed
#musicinfo_acousticelectric
#musicinfo_vocalinstrumental
#musicinfo_gender
#musicinfo_tags_vartags
#musicinfo_tags_genres
#musicinfo_tags_instruments
#name
#position
#releasedate
#shareurl
#shorturl

[[2015:GC15UX:JSON Metadata]] presents statistics and plots of selected fields.

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC15UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

See the [[#Evaluation Webforms]] below for a better understanding of our E6K-inpsired evaluation system design.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
|
|
|
|-
|-
|
|
|
|-
|-
|
|
|
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC15UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evaluators' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': How would you rate your overall satisfaction with the system?
Very unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Very satisfactory

* '''Aesthetics''': How would you rate the visual attractiveness of the system?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Ease of use''': How easy was it to figure out how to use the system?
Very difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Very easy

* '''Clarity''': How well does the system communicate what is going on?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Affordances''': How well does the system allow you to perform what you want to do?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Performance''': Does the system work efficiently and without bugs/glitches?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Open Text Feedback''': An open-ended question is provided for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC15UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

'''''You need to put together a playlist for a particular event (e.g., dinner party at your house, workout session). Try to use the assigned system to make playlists for at least a couple of different events.'''''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evaluators' purposes ("music for their own situation"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects. This allows great flexibility and virtually unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC15UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
Graders can take as many assignments as they wish in the My Assignments page. They are allowed to go back to the evaluation page anytime by clicking the thumbnail of the submission.

[[File:GCUX_wireframe_my_assignments.png|800px]]
 
To facilitate the evaluators and minimize their burden, the GC15UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Organization=

==Important Dates==
This year GC15UX adopts the two-phase model with two evaluations. The first phase will end by the ISMIR conference and we will disclose preliminary results at the conference like what we did last year. Then, phase II will start. Participating developers can continue improving their systems based on the feedback from the first phase and another round of evaluation will be conducted in February. We believe that this model serves the developers well since it is in accordance with the iterative nature of user-centered design. In this way, the developers will also have enough time to develop their complete MIR systems.

*July 15th: announce the GC
*Sep. 21st: the first deadline for system submission
*Feb. 21st: the second deadline for system submission

Updates for the first round (updated Sep. 15, 2015)
* Oct 1st: extended deadline for system submission
* Oct 5th: start the evaluation
* Oct 19th: close the evaluation system
* MIREX and GC session in ISMIR2015: announce the results and discussion

==What to Submit==

A URL to the participanting system.

==Contacts==
The GC15UX team consists of:
:J. Stephen Downie, University of Illinois (MIREX director)
:Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
:Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
:Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
:David Bainbridge, Waikato University, New Zealand
:Kahyun Choi, University of Illinois
:Peter Organisciak, University of Illinois
:Yun Hao, University of Illinois

Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [mailto:jdownie@illinois.edu] or anyone in the team.

2014:GC14UX

2014-10-11T08:03:06Z

Xiao: /* Potential Participants */

{{DISPLAYTITLE:Grand Challenge 2014: User Experience}}
=Purpose=
Holistic, user-centered evaluation of the user experience in interacting with complete, user-facing music information retrieval (MIR) systems.

=Goals=
# To inspire the development of complete MIR systems.
# To promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14,742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX are sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems. It represents a randomly chosen subset the content available at Jamendo that is published under the terms of the Creative Commons Attribution-Non-Commercial-ShareAlike (by-nc-sa), where user-supplied data has tagged a track with 1 or more genre categories. For more details about usage of this dataset, see the LICENSE.txt file contained in the downloaded files.

The dataset contains the MP3 tracks and the metadata the Jamendo site publishes on the respective items (represented in JSON format), retrieved using the site's API (6th Aug 2014). The dataset is available both zipped up and as a tar-ball(you only need one of these); however, at 60+ Gb it is a non-trival size of file to download over the web, and so we suggest you install a Download Manager extension to your browser if you do not already have one and make use of that. In a test using the DownThemAll! extension to Firefox, downloading the dataset between University of Illinois at Urbana-Champaign and Waikato University in New Zealand took a little under 2 hours.

You need to register to download the main dataset.

[https://www.music-ir.org/mirex/gc14ux/ https://www.music-ir.org/mirex/gc14ux/] 

==Metadata Extracted from JSON Files==
The JSON files retrieved from Jamendo site contain various metadata:

#album_id
#album_image
#album_name
#artist_id
#artist_idstr
#artist_name
#audio
#audiodownload
#duration
#id
#license_ccurl
#musicinfo_lang
#musicinfo_speed
#musicinfo_acousticelectric
#musicinfo_vocalinstrumental
#musicinfo_gender
#musicinfo_tags_vartags
#musicinfo_tags_genres
#musicinfo_tags_instruments
#name
#position
#releasedate
#shareurl
#shorturl

[[2014:GC14UX:JSON Metadata]] presents statistics and plots of selected fields.

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

See the [[#Evaluation Webforms]] below for a better understanding of our E6K-inpsired evaluation system design.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|-
| Moody
| Kai Xing,Pei-I Chen,Terry Lei,Jen-Yu Liu,Billy Vong,Yi-Hsuan Yang,Xiao Hu
| xkaics@gmail.com,gooa1121@gmail.com,skyhelpme@gmail.com,ciauaishere@gmail.com,billyvg@gmail.com,affige@gmail.com,xiaoxhu@hku.hk
|-
|-
| Tonic
| D. Bountouridis, J.V. Balen, M. Rodriguez, S. Manoli, A. Aljanaki, F. Wiering, R.C. Veltkamp
|d.bountouridis@uu.nl, J.M.H.VanBalen@uu.nl. ME.RodriguezLopez@uu.nl, stavma0891@gmail.com, a.aljanaki@uu.nl, F.Wiering@uu.nl, R.C.Veltkamp@uu.nl
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evaluators' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Very unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Very satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Very difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Very easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Affordances''': How well does the system allow you to perform what you want to do?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Open Text Feedback''': An open-ended question is provided for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

'''''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.'''''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evaluators' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects. This allows great flexibility and virtually unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
Graders can take as many assignments as they wish in the My Assignments page. They are allowed to go back to the evaluation page anytime by clicking the thumbnail of the submission.

[[File:GCUX_wireframe_my_assignments.png|800px]]
 
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Organization=

==Important Dates==

*July 1: announce the GC
*Sep. 28st: deadline for system submission
*Oct. 5th: start the evaluation
*Oct. 21st: close the evaluation system
*Oct. 29th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

==What to Submit==

A URL to the participanting system.

==Contacts==
The GC14UX team consists of:
:J. Stephen Downie, University of Illinois (MIREX director)
:Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
:Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
:Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
:David Bainbridge, Waikato University, New Zealand
:Kahyun Choi, University of Illinois
:Peter Organisciak, University of Illinois

Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [mailto:jdownie@illinois.edu] or anyone in the team.

2014:GC14UX

2014-09-12T07:19:17Z

Xiao: /* Potential Participants */

{{DISPLAYTITLE:Grand Challenge 2014: User Experience}}
=Purpose=
Holistic, user-centered evaluation of the user experience in interacting with complete, user-facing music information retrieval (MIR) systems.

=Goals=
# To inspire the development of complete MIR systems.
# To promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14,742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX are sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems. It represents a randomly chosen subset the content available at Jamendo that is published under the terms of the Creative Commons Attribution-Non-Commercial-ShareAlike (by-nc-sa), where user-supplied data has tagged a track with 1 or more genre categories. For more details about usage of this dataset, see the LICENSE.txt file contained in the downloaded files.

The dataset contains the MP3 tracks and the metadata the Jamendo site publishes on the respective items (represented in JSON format), retrieved using the site's API (6th Aug 2014). The dataset is available both zipped up and as a tar-ball(you only need one of these); however, at 60+ Gb it is a non-trival size of file to download over the web, and so we suggest you install a Download Manager extension to your browser if you do not already have one and make use of that. In a test using the DownThemAll! extension to Firefox, downloading the dataset between University of Illinois at Urbana-Champaign and Waikato University in New Zealand took a little under 2 hours.

You need to register to download the main dataset.

[https://www.music-ir.org/mirex/gc14ux/ https://www.music-ir.org/mirex/gc14ux/] 

==Metadata Extracted from JSON Files==
The JSON files retrieved from Jamendo site contain various metadata:

#album_id
#album_image
#album_name
#artist_id
#artist_idstr
#artist_name
#audio
#audiodownload
#duration
#id
#license_ccurl
#musicinfo_lang
#musicinfo_speed
#musicinfo_acousticelectric
#musicinfo_vocalinstrumental
#musicinfo_gender
#musicinfo_tags_vartags
#musicinfo_tags_genres
#musicinfo_tags_instruments
#name
#position
#releasedate
#shareurl
#shorturl

[[2014:GC14UX:JSON Metadata]] presents statistics and plots of selected fields.

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

See the [[#Evaluation Webforms]] below for a better understanding of our E6K-inpsired evaluation system design.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|-
| Moody
| Kai Xing,Pei-I Chen,Terry Lei,Jen-Yu Liu,Yi-Hsuan Yang,Xiao Hu
| xkaics@gmail.com,gooa1121@gmail.com,skyhelpme@gmail.com,ciauaishere@gmail.com,affige@gmail.com,xiaoxhu@hku.hk
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evaluators' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Very unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Very satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Very difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Very easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Affordances''': How well does the system allow you to perform what you want to do?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Open Text Feedback''': An open-ended question is provided for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

'''''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.'''''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evaluators' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects. This allows great flexibility and virtually unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
Graders can take as many assignments as they wish in the My Assignments page. They are allowed to go back to the evaluation page anytime by clicking the thumbnail of the submission.

[[File:GCUX_wireframe_my_assignments.png|800px]]
 
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Organization=

==Important Dates==

*July 1: announce the GC
*Sep. 28st: deadline for system submission
*Oct. 5th: start the evaluation
*Oct. 21st: close the evaluation system
*Oct. 29th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

==What to Submit==

A URL to the participanting system.

==Contacts==
The GC14UX team consists of:
:J. Stephen Downie, University of Illinois (MIREX director)
:Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
:Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
:Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
:David Bainbridge, Waikato University, New Zealand
:Kahyun Choi, University of Illinois
:Peter Organisciak, University of Illinois

Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [mailto:jdownie@illinois.edu] or anyone in the team.

2014:GC14UX:JSON Metadata

2014-08-30T11:07:30Z

Xiao: /* Genre */

The complete extracted JSON tags can be downloaded here: [[File:json_metadata.txt]]

=Genre=
Each song in the dataset has two to four genre tags. The following histgram shows how many songs have at least 1, 2, 3, or 4 genres.
[[Image:ghistgram.png]]

The most popular genre tags are shown in the following diagram:
[[Image:gdist.png]]

The complete list of genre tags and their counts can be found in this file: [[File:json_g_dist.txt]]

=Free Tags=
Each song has zero to six free tags. The following histgram shows how many songs have at least 1,..,or 6 tags.
[[Image:thistgram.png]]

The most popular free tags are shown in the following diagram:
[[Image:tdist.png]]

The complete list of genre tags and their counts can be found in this file: [[File:json_t_dist.txt]]

=Instrument=
Each song has zero to three instrument tags. The following histgram shows how many songs have at least 1, 2 or 3 instrument tags.
[[Image:instrhistgram.png]]

The most popular instrument tags are shown in the following diagram:
[[Image:idist.png]]

The complete list of genre tags and their counts can be found in this file: [[File:json_i_dist.txt]]

=Vocal vs Instrumental=
Part of the songs have labels indicating whether they are vocal or instrumental. The following diagram shows the distribution:
[[Image:vocinst_histgram.png]]

=Speed=
Part of the songs have labels indicating their speeds. The following diagram shows the distribution:
[[Image:speed_histgram.png]]

=Language=
Some songs have labels indicating their languages. The following diagram shows the distribution:
[[Image:lang_histgram.png]]

=Duration=
All songs have their duration indicated in seconds. The following diagram shows the duration distribution of most songs:
[[Image:duration.png]]

The complete list of duration and their counts can be found in this file: [[json_d_dist.txt]]

2014:GC14UX:JSON Metadata

2014-08-30T10:37:24Z

Xiao: /* Genre */

=Genre=
Each song in the dataset has two to four genre tags. The histgram shows how many songs have at least 1, 2, 3, or 4 genres.
[[Image:gdist.png]]

2014:GC14UX:JSON Metadata

2014-08-30T10:30:26Z

Xiao: Created page with "=Genre= Each song in the dataset has two to four genre tags. The histgram shows how many songs have at least 1, 2, 3, or 4 genres. File:gdist.png"

=Genre=
Each song in the dataset has two to four genre tags. The histgram shows how many songs have at least 1, 2, 3, or 4 genres.
[[File:gdist.png]]

2014:GC14UX

2014-08-30T10:25:51Z

Xiao: /* Metadata Extracted from JSON Files */

{{DISPLAYTITLE:Grand Challenge 2014: User Experience}}
=Purpose=
Holistic, user-centered evaluation of the user experience in interacting with complete, user-facing music information retrieval (MIR) systems.

=Goals=
# To inspire the development of complete MIR systems.
# To promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14,742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX are sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems. It represents a randomly chosen subset the content available at Jamendo that is published under the terms of the Creative Commons Attribution-Non-Commercial-ShareAlike (by-nc-sa), where user-supplied data has tagged a track with 1 or more genre categories. For more details about usage of this dataset, see the LICENSE.txt file contained in the downloaded files.

The dataset contains the MP3 tracks and the metadata the Jamendo site publishes on the respective items (represented in JSON format), retrieved using the site's API (6th Aug 2014). The dataset is available both zipped up and as a tar-ball(you only need one of these); however, at 60+ Gb it is a non-trival size of file to download over the web, and so we suggest you install a Download Manager extension to your browser if you do not already have one and make use of that. In a test using the DownThemAll! extension to Firefox, downloading the dataset between University of Illinois at Urbana-Champaign and Waikato University in New Zealand took a little under 2 hours.

[https://www.music-ir.org/mirex/gc14ux/dataset/gc14ux-jamendo-by-nc-sa-subset.zip gc14ux-jamendo-by-nc-sa-subset.zip (60.1 GB)] 
[https://www.music-ir.org/mirex/gc14ux/dataset/gc14ux-jamendo-by-nc-sa-subset.tar.gz gc14ux-jamendo-by-nc-sa-subset.tar.gz (60.1 GB)] 


==Metadata Extracted from JSON Files==
The JSON files retrieved from Jamendo site contain various metadata:

[album_id album_image album_name artist_id artist_idstr artist_name audio audiodownload duration id license_ccurl musicinfo_lang musicinfo_speed musicinfo_acousticelectric musicinfo_vocalinstrumental musicinfo_gender musicinfo_tags_vartags musicinfo_tags_genres musicinfo_tags_instruments name position releasedate shareurl shorturl]

[[2014:GC14UX:JSON Metadata]] presents statistics and plots of selected fields.

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

See the [[#Evaluation Webforms]] below for a better understanding of our E6K-inpsired evaluation system design.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evaluators' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Very unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Very satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Very difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Very easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Affordances''': How well does the system allow you to perform what you want to do?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Open Text Feedback''': An open-ended question is provided for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

'''''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.'''''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evaluators' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects. This allows great flexibility and virtually unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
Graders can take as many assignments as they wish in the My Assignments page. They are allowed to go back to the evaluation page anytime by clicking the thumbnail of the submission.

[[File:GCUX_wireframe_my_assignments.png|800px]]
 
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Organization=

==Important Dates==

*July 1: announce the GC
*Sep. 28st: deadline for system submission
*Oct. 5th: start the evaluation
*Oct. 27th: close the evaluation system
*Oct. 29th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

==What to Submit==

A URL to the participanting system.

==Contacts==
The GC14UX team consists of:
:J. Stephen Downie, University of Illinois (MIREX director)
:Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
:Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
:Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
:David Bainbridge, Waikato University, New Zealand
:Kahyun Choi, University of Illinois
:Peter Organisciak, University of Illinois

Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [mailto:jdownie@illinois.edu] or anyone in the team.

2014:GC14UX

2014-08-30T10:25:13Z

Xiao: /* Metadata Extracted from JSON Files */

{{DISPLAYTITLE:Grand Challenge 2014: User Experience}}
=Purpose=
Holistic, user-centered evaluation of the user experience in interacting with complete, user-facing music information retrieval (MIR) systems.

=Goals=
# To inspire the development of complete MIR systems.
# To promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14,742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX are sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems. It represents a randomly chosen subset the content available at Jamendo that is published under the terms of the Creative Commons Attribution-Non-Commercial-ShareAlike (by-nc-sa), where user-supplied data has tagged a track with 1 or more genre categories. For more details about usage of this dataset, see the LICENSE.txt file contained in the downloaded files.

The dataset contains the MP3 tracks and the metadata the Jamendo site publishes on the respective items (represented in JSON format), retrieved using the site's API (6th Aug 2014). The dataset is available both zipped up and as a tar-ball(you only need one of these); however, at 60+ Gb it is a non-trival size of file to download over the web, and so we suggest you install a Download Manager extension to your browser if you do not already have one and make use of that. In a test using the DownThemAll! extension to Firefox, downloading the dataset between University of Illinois at Urbana-Champaign and Waikato University in New Zealand took a little under 2 hours.

[https://www.music-ir.org/mirex/gc14ux/dataset/gc14ux-jamendo-by-nc-sa-subset.zip gc14ux-jamendo-by-nc-sa-subset.zip (60.1 GB)] 
[https://www.music-ir.org/mirex/gc14ux/dataset/gc14ux-jamendo-by-nc-sa-subset.tar.gz gc14ux-jamendo-by-nc-sa-subset.tar.gz (60.1 GB)] 


==Metadata Extracted from JSON Files==
The JSON files retrieved from Jamendo site contain various metadata:

[album_id album_image album_name artist_id artist_idstr artist_name audio audiodownload duration id license_ccurl musicinfo_lang musicinfo_speed musicinfo_acousticelectric musicinfo_vocalinstrumental musicinfo_gender musicinfo_tags_vartags musicinfo_tags_genres musicinfo_tags_instruments name position releasedate shareurl shorturl]

[[2014:GC14UX:JSON Metadata]] presents statistics and plots of selected fields.

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

See the [[#Evaluation Webforms]] below for a better understanding of our E6K-inpsired evaluation system design.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evaluators' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Very unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Very satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Very difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Very easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Affordances''': How well does the system allow you to perform what you want to do?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Open Text Feedback''': An open-ended question is provided for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

'''''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.'''''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evaluators' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects. This allows great flexibility and virtually unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
Graders can take as many assignments as they wish in the My Assignments page. They are allowed to go back to the evaluation page anytime by clicking the thumbnail of the submission.

[[File:GCUX_wireframe_my_assignments.png|800px]]
 
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Organization=

==Important Dates==

*July 1: announce the GC
*Sep. 28st: deadline for system submission
*Oct. 5th: start the evaluation
*Oct. 27th: close the evaluation system
*Oct. 29th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

==What to Submit==

A URL to the participanting system.

==Contacts==
The GC14UX team consists of:
:J. Stephen Downie, University of Illinois (MIREX director)
:Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
:Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
:Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
:David Bainbridge, Waikato University, New Zealand
:Kahyun Choi, University of Illinois
:Peter Organisciak, University of Illinois

Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [mailto:jdownie@illinois.edu] or anyone in the team.

2014:GC14UX

2014-08-30T10:24:32Z

Xiao: /* Dataset */

{{DISPLAYTITLE:Grand Challenge 2014: User Experience}}
=Purpose=
Holistic, user-centered evaluation of the user experience in interacting with complete, user-facing music information retrieval (MIR) systems.

=Goals=
# To inspire the development of complete MIR systems.
# To promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14,742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX are sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems. It represents a randomly chosen subset the content available at Jamendo that is published under the terms of the Creative Commons Attribution-Non-Commercial-ShareAlike (by-nc-sa), where user-supplied data has tagged a track with 1 or more genre categories. For more details about usage of this dataset, see the LICENSE.txt file contained in the downloaded files.

The dataset contains the MP3 tracks and the metadata the Jamendo site publishes on the respective items (represented in JSON format), retrieved using the site's API (6th Aug 2014). The dataset is available both zipped up and as a tar-ball(you only need one of these); however, at 60+ Gb it is a non-trival size of file to download over the web, and so we suggest you install a Download Manager extension to your browser if you do not already have one and make use of that. In a test using the DownThemAll! extension to Firefox, downloading the dataset between University of Illinois at Urbana-Champaign and Waikato University in New Zealand took a little under 2 hours.

[https://www.music-ir.org/mirex/gc14ux/dataset/gc14ux-jamendo-by-nc-sa-subset.zip gc14ux-jamendo-by-nc-sa-subset.zip (60.1 GB)] 
[https://www.music-ir.org/mirex/gc14ux/dataset/gc14ux-jamendo-by-nc-sa-subset.tar.gz gc14ux-jamendo-by-nc-sa-subset.tar.gz (60.1 GB)] 


==Metadata Extracted from JSON Files==
The JSON files retrieved from Jamendo site contain various metadata: album_id album_image album_name artist_id artist_idstr artist_name audio audiodownload duration id license_ccurl musicinfo_lang musicinfo_speed musicinfo_acousticelectric musicinfo_vocalinstrumental musicinfo_gender musicinfo_tags_vartags musicinfo_tags_genres musicinfo_tags_instruments name position releasedate shareurl shorturl
[[2014:GC14UX:JSON Metadata]] presents statistics and plots of selected fields.

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

See the [[#Evaluation Webforms]] below for a better understanding of our E6K-inpsired evaluation system design.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evaluators' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Very unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Very satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Very difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Very easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Affordances''': How well does the system allow you to perform what you want to do?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Open Text Feedback''': An open-ended question is provided for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

'''''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.'''''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evaluators' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects. This allows great flexibility and virtually unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
Graders can take as many assignments as they wish in the My Assignments page. They are allowed to go back to the evaluation page anytime by clicking the thumbnail of the submission.

[[File:GCUX_wireframe_my_assignments.png|800px]]
 
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Organization=

==Important Dates==

*July 1: announce the GC
*Sep. 28st: deadline for system submission
*Oct. 5th: start the evaluation
*Oct. 27th: close the evaluation system
*Oct. 29th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

==What to Submit==

A URL to the participanting system.

==Contacts==
The GC14UX team consists of:
:J. Stephen Downie, University of Illinois (MIREX director)
:Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
:Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
:Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
:David Bainbridge, Waikato University, New Zealand
:Kahyun Choi, University of Illinois
:Peter Organisciak, University of Illinois

Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [mailto:jdownie@illinois.edu] or anyone in the team.

2014:GC14UX

2014-07-01T13:44:24Z

Xiao: /* What to Submit */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Very unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Very satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Very difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Very easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects. This allows great flexibility and virtually unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Organization=

==Important Dates==

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

==What to Submit==

A URL to the participanting system.

==Contacts==
The GC14UX team consists of:
:J. Stephen Downie, University of Illinois (MIREX director)
:Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
:Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
:Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
:David Bainbridge, Waikato University, New Zealand
:Kahyun Choi, University of Illinois
:Peter Organisciak, University of Illinois

Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [mailto:jdownie@illinois.edu] or anyone in the team.

2014:GC14UX

2014-07-01T13:43:44Z

Xiao: /* Organization */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Very unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Very satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Very difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Very easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects. This allows great flexibility and virtually unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Organization=

==Important Dates==

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

==What to Submit==

==Contacts==
The GC14UX team consists of:
:J. Stephen Downie, University of Illinois (MIREX director)
:Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
:Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
:Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
:David Bainbridge, Waikato University, New Zealand
:Kahyun Choi, University of Illinois
:Peter Organisciak, University of Illinois

Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [mailto:jdownie@illinois.edu] or anyone in the team.

2014:GC14UX

2014-07-01T13:43:06Z

Xiao: /* Task for evaluators */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Very unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Very satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Very difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Very easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects. This allows great flexibility and virtually unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Organization=

==Important Dates==

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

==Contacts==
The GC14UX team consists of:
:J. Stephen Downie, University of Illinois (MIREX director)
:Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
:Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
:Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
:David Bainbridge, Waikato University, New Zealand
:Kahyun Choi, University of Illinois
:Peter Organisciak, University of Illinois

Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [mailto:jdownie@illinois.edu] or anyone in the team.

2014:GC14UX

2014-07-01T13:12:06Z

Xiao: /* Criteria */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Very unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Very satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Very difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Very easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects of music. This allows great flexibility and unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Organization=

==Important Dates==

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

==Contacts==
The GC14UX team consists of:
:J. Stephen Downie, University of Illinois (MIREX director)
:Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
:Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
:Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
:David Bainbridge, Waikato University, New Zealand
:Kahyun Choi, University of Illinois
:Peter Organisciak, University of Illinois

Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [mailto:jdownie@illinois.edu] or anyone in the team.

2014:GC14UX

2014-07-01T13:05:52Z

Xiao:

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects of music. This allows great flexibility and unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Organization=

==Important Dates==

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

==Contacts==
The GC14UX team consists of:
:J. Stephen Downie, University of Illinois (MIREX director)
:Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
:Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
:Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
:David Bainbridge, Waikato University, New Zealand
:Kahyun Choi, University of Illinois
:Peter Organisciak, University of Illinois

Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [mailto:jdownie@illinois.edu] or anyone in the team.

2014:GC14UX

2014-07-01T13:04:30Z

Xiao: /* The GC14UX Team */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects of music. This allows great flexibility and unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

=The GC14UX Team=
The GC14UX team consists of:
:J. Stephen Downie, University of Illinois (MIREX director)
:Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
:Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
:Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
:David Bainbridge, Waikato University, New Zealand
:Kahyun Choi, University of Illinois
:Peter Organisciak, University of Illinois

Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [mailto:jdownie@illinois.edu] or anyone in the team.

2014:GC14UX

2014-07-01T13:01:57Z

Xiao: /* The GC14UX Team */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects of music. This allows great flexibility and unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

=The GC14UX Team=
:J. Stephen Downie, University of Illinois (MIREX director)
:Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
:Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
:Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
:David Bainbridge, Waikato University, New Zealand
:Kahyun Choi, University of Illinois
:Peter Organisciak, University of Illinois

Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [mailto:jdownie@illinois.edu].

2014:GC14UX

2014-07-01T12:48:34Z

Xiao:

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects of music. This allows great flexibility and unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

=The GC14UX Team=
Should you have any questions or inquires, please contact the GC14UX Team.

2014:GC14UX

2014-07-01T12:45:29Z

Xiao: /* Evaluation Webforms */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects of music. This allows great flexibility and unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

[[File:GCUX wireframe evaluation.png|800px]]

=Wireframes=

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T12:44:05Z

Xiao: /* Wireframes */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects of music. This allows great flexibility and unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe.

=Wireframes=

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T12:43:26Z

Xiao: /* Evaluation Webforms */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects of music. This allows great flexibility and unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
To facilitate the evaluators and minimize their burden, the GC14UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T12:38:33Z

Xiao: /* Task for evaluators */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects of music. This allows great flexibility and unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T12:37:12Z

Xiao:

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects of music. This allows great flexibility and unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T12:36:45Z

Xiao: /* Participating Systems */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects of music. This allows great flexibility and unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T12:36:30Z

Xiao: /* Participating Systems */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects of music. This allows great flexibility and unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T12:35:48Z

Xiao: /* Task for evaluators */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.
=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

To motivate the evaluators, a defined yet open task is given to the evaluators:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some (copyright-free) songs to use as background music.''

The task is to ensure that evaluators have a goal when they interact with the systems. The goal is flexible and authentic to the evalutors' lives ("a recent, memorable occasion"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects of music. This allows great flexibility and unlimited possibility in system design.

Another important consideration in designing the task is the music collection available for this GC14UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T12:17:04Z

Xiao: /* Evaluators */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.
=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The [[#Evaluation Webforms]] developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

Evaluators are given the following task:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some songs to use as background music.''

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T12:12:49Z

Xiao: /* Evaluation mechanism */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.
=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

Evaluators are given the following task:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some songs to use as background music.''

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation Webforms==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T12:11:53Z

Xiao: /* Evaluation */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.
=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

Evaluators are given the following task:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some songs to use as background music.''

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T12:10:33Z

Xiao: /* Criteria */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.
=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria:

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

Evaluators are given the following task:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some songs to use as background music.''

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T12:10:10Z

Xiao: /* Evaluation */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.
=Evaluation=

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

==Criteria==

''Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.''

Given the GC14UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evalutors' cognitive load.

Evaluators will rate systems on the following criteria

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How beautiful is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for evaluators to give feedback if they wish to do so.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Task for evaluators==

Evaluators are given the following task:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some songs to use as background music.''

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T11:59:25Z

Xiao: /* Potential Participants */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.
=Evaluation=
==Task==

Evaluators are given the following task:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some songs to use as background music.''

==Criteria==

''Note that the evaluation criteria or its descriptions may change in the months leading up to the submission deadline, as we test it and work to improve it.''

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How good is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for users to give feedback if they wish to do so.

This simplicity is because: 1) the GC14UX is all about how users perceive their experiences of the systems. We intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. 2) more data capturing opportunities will distract from the real user experience.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T11:58:22Z

Xiao: /* Goals */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.

=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.
=Evaluation=
==Task==

Evaluators are given the following task:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some songs to use as background music.''

==Criteria==

''Note that the evaluation criteria or its descriptions may change in the months leading up to the submission deadline, as we test it and work to improve it.''

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How good is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for users to give feedback if they wish to do so.

This simplicity is because: 1) the GC14UX is all about how users perceive their experiences of the systems. We intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. 2) more data capturing opportunities will distract from the real user experience.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T01:38:34Z

Xiao: /* Task */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014

=Purposes=
Holistic evaluation of user experience in interacting with user-serving MIR systems

=Goals=
1) to inspire the development of complete MIR systems
2) to promote the notion of user experience as a first-class research objective in the MIR community

=Dataset=
A set of music audio of 10,000 tracks is provided for the GC14UX. It will be a subset of tracks drawn from Jamendo collection's CC-BY licensed works (http://www.jamendo.com/en/welcome).

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the CC-BY license (http://creativecommons.org/licenses/by/3.0/). The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=
==Task==
To ensure that the GC14UX does not become a system-centered evaluation in disguise, the process will remain as agnostic as possible concerning the technological means by which participating systems create and deliver their experiences to the users. The only requirement for the systems is they should support music discovery (in whichever ways).

Evaluators will be recruited from the MIR community. After all the participating systems are ready, evaluators will be given the following task:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some songs to use as background music.''

Criteria:

* '''Overall satisfaction''': Rate your overall satisfaction with this system. (0-20: very unsatisfied; 20-40: somewhat unsatisfied; 40-60: neutral; 60-80: somewhat satisfied; 80-100: very satisfied).

* '''Learnability''': How easy was it to figure out how to use the system?

* '''Interaction''': Does the system “warn” you when you’re about to make a mistake? Are you able to backtrack when you want to? Are you able to recover from unexpected operations? Are you able to efficiently complete your task?

* '''Presentation''': How well does the system communicate what’s going on? How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system? Is the design aesthetically pleasing?

An open-ended question is provided but is optional for users to give feedback if they wish to do so.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

To prompt a scenario/context in which a user interacts with participating systems, a half-defined sentence will be provided to a user in the evaluation webforms *before* a set of participating systems are shown to the user:

“I am looking for music for ______________”

The user is free to fill in any need s/he would like to find music for, such as “drinking a cup of coffee”, or “waiting for my girlfriend”. In this way, the goal of the user is captured and the goal is as authentic to the user as possible.

==Evaluators==
They will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. It is planned that each submitted system will be evaluated by 50 to 60 evaluators. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Certificates==
The GC14UX will present certificates to all participating systems and the system(s) with the highest mean score.

=Wireframes=
[[File:GCUX_wireframe_(broad).png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T01:36:50Z

Xiao: /* Potential Participants */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014

=Purposes=
Holistic evaluation of user experience in interacting with user-serving MIR systems

=Goals=
1) to inspire the development of complete MIR systems
2) to promote the notion of user experience as a first-class research objective in the MIR community

=Dataset=
A set of music audio of 10,000 tracks is provided for the GC14UX. It will be a subset of tracks drawn from Jamendo collection's CC-BY licensed works (http://www.jamendo.com/en/welcome).

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the CC-BY license (http://creativecommons.org/licenses/by/3.0/). The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Evaluation=
==Task==
To ensure that the GC14UX does not become a system-centered evaluation in disguise, the process will remain as agnostic as possible concerning the technological means by which participating systems create and deliver their experiences to the users. The only requirement for the systems is they should support music discovery (in whichever ways).
To design some criteria that’s not tied to the music content;

Task: You are creating a YouTube video and you need to find some open-source music to use.

Q. What kind of video you are trying to find the music for? (Need to discuss more about this.)

- Overall satisfaction (“On a scale of 0 to 100, rate your overall satisfaction with this system”)
and bands with different colors (0-20: very unsatisfied; 20-40: somewhat unsatisfied; 40-60: neutral; 60-80: somewhat satisfied; 80-100: very satisfied).

- Aesthetics (look of the interface)
- Interaction (smoothness, no stalls or bugs)
- Awareness (what’s going on, where the user is)

Ten Heuristics here: http://www.nngroup.com/articles/ten-usability-heuristics/
How to convert the 10 heuristics

Any literature support for the three items?
Textbox

*. Evaluation criterion:

Satisfaction: How much do you like using the interface?
Memorability: 
Errors: How many errors did you make? How easy is it to recover from them?
Learnability: How easy was it to figure out the interface? Does its layout match what you expect (e.g. buttons, search box locations, navigation)? Was anything confusing? How good is the interface at communicating functionality (help and documentation)?
---

This simplicity is because: 1) the GC14UX is all about how users perceive their experiences of the systems. We intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. 2) more data capturing opportunities will distract from the real user experience.

An open-ended question is provided but is optional for users to give feedback if they wish to do so.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

To prompt a scenario/context in which a user interacts with participating systems, a half-defined sentence will be provided to a user in the evaluation webforms *before* a set of participating systems are shown to the user:

“I am looking for music for ______________”

The user is free to fill in any need s/he would like to find music for, such as “drinking a cup of coffee”, or “waiting for my girlfriend”. In this way, the goal of the user is captured and the goal is as authentic to the user as possible.

==Evaluators==
They will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. It is planned that each submitted system will be evaluated by 50 to 60 evaluators. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Certificates==
The GC14UX will present certificates to all participating systems and the system(s) with the highest mean score.

=Wireframes=
[[File:GCUX_wireframe_(broad).png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T01:33:35Z

Xiao: /* Participating Systems */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014

=Purposes=
Holistic evaluation of user experience in interacting with user-serving MIR systems

=Goals=
1) to inspire the development of complete MIR systems
2) to promote the notion of user experience as a first-class research objective in the MIR community

=Dataset=
A set of music audio of 10,000 tracks is provided for the GC14UX. It will be a subset of tracks drawn from Jamendo collection's CC-BY licensed works (http://www.jamendo.com/en/welcome).

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the CC-BY license (http://creativecommons.org/licenses/by/3.0/). The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==
||Name(s) || Email(s)||

=Evaluation=
==Task==
To ensure that the GC14UX does not become a system-centered evaluation in disguise, the process will remain as agnostic as possible concerning the technological means by which participating systems create and deliver their experiences to the users. The only requirement for the systems is they should support music discovery (in whichever ways).
To design some criteria that’s not tied to the music content;

Task: You are creating a YouTube video and you need to find some open-source music to use.

Q. What kind of video you are trying to find the music for? (Need to discuss more about this.)

- Overall satisfaction (“On a scale of 0 to 100, rate your overall satisfaction with this system”)
and bands with different colors (0-20: very unsatisfied; 20-40: somewhat unsatisfied; 40-60: neutral; 60-80: somewhat satisfied; 80-100: very satisfied).

- Aesthetics (look of the interface)
- Interaction (smoothness, no stalls or bugs)
- Awareness (what’s going on, where the user is)

Ten Heuristics here: http://www.nngroup.com/articles/ten-usability-heuristics/
How to convert the 10 heuristics

Any literature support for the three items?
Textbox

*. Evaluation criterion:

Satisfaction: How much do you like using the interface?
Memorability: 
Errors: How many errors did you make? How easy is it to recover from them?
Learnability: How easy was it to figure out the interface? Does its layout match what you expect (e.g. buttons, search box locations, navigation)? Was anything confusing? How good is the interface at communicating functionality (help and documentation)?
---

This simplicity is because: 1) the GC14UX is all about how users perceive their experiences of the systems. We intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. 2) more data capturing opportunities will distract from the real user experience.

An open-ended question is provided but is optional for users to give feedback if they wish to do so.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

To prompt a scenario/context in which a user interacts with participating systems, a half-defined sentence will be provided to a user in the evaluation webforms *before* a set of participating systems are shown to the user:

“I am looking for music for ______________”

The user is free to fill in any need s/he would like to find music for, such as “drinking a cup of coffee”, or “waiting for my girlfriend”. In this way, the goal of the user is captured and the goal is as authentic to the user as possible.

==Evaluators==
They will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. It is planned that each submitted system will be evaluated by 50 to 60 evaluators. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Certificates==
The GC14UX will present certificates to all participating systems and the system(s) with the highest mean score.

=Wireframes=
[[File:GCUX_wireframe_(broad).png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T01:32:33Z

Xiao: /* Important Dates */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014

=Purposes=
Holistic evaluation of user experience in interacting with user-serving MIR systems

=Goals=
1) to inspire the development of complete MIR systems
2) to promote the notion of user experience as a first-class research objective in the MIR community

=Dataset=
A set of music audio of 10,000 tracks is provided for the GC14UX. It will be a subset of tracks drawn from Jamendo collection's CC-BY licensed works (http://www.jamendo.com/en/welcome).

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the CC-BY license (http://creativecommons.org/licenses/by/3.0/). The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

=Evaluation=
==Task==
To ensure that the GC14UX does not become a system-centered evaluation in disguise, the process will remain as agnostic as possible concerning the technological means by which participating systems create and deliver their experiences to the users. The only requirement for the systems is they should support music discovery (in whichever ways).
To design some criteria that’s not tied to the music content;

Task: You are creating a YouTube video and you need to find some open-source music to use.

Q. What kind of video you are trying to find the music for? (Need to discuss more about this.)

- Overall satisfaction (“On a scale of 0 to 100, rate your overall satisfaction with this system”)
and bands with different colors (0-20: very unsatisfied; 20-40: somewhat unsatisfied; 40-60: neutral; 60-80: somewhat satisfied; 80-100: very satisfied).

- Aesthetics (look of the interface)
- Interaction (smoothness, no stalls or bugs)
- Awareness (what’s going on, where the user is)

Ten Heuristics here: http://www.nngroup.com/articles/ten-usability-heuristics/
How to convert the 10 heuristics

Any literature support for the three items?
Textbox

*. Evaluation criterion:

Satisfaction: How much do you like using the interface?
Memorability: 
Errors: How many errors did you make? How easy is it to recover from them?
Learnability: How easy was it to figure out the interface? Does its layout match what you expect (e.g. buttons, search box locations, navigation)? Was anything confusing? How good is the interface at communicating functionality (help and documentation)?
---

This simplicity is because: 1) the GC14UX is all about how users perceive their experiences of the systems. We intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. 2) more data capturing opportunities will distract from the real user experience.

An open-ended question is provided but is optional for users to give feedback if they wish to do so.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

To prompt a scenario/context in which a user interacts with participating systems, a half-defined sentence will be provided to a user in the evaluation webforms *before* a set of participating systems are shown to the user:

“I am looking for music for ______________”

The user is free to fill in any need s/he would like to find music for, such as “drinking a cup of coffee”, or “waiting for my girlfriend”. In this way, the goal of the user is captured and the goal is as authentic to the user as possible.

==Evaluators==
They will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. It is planned that each submitted system will be evaluated by 50 to 60 evaluators. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Certificates==
The GC14UX will present certificates to all participating systems and the system(s) with the highest mean score.

=Wireframes=
[[File:GCUX_wireframe_(broad).png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T01:32:08Z

Xiao: /* Important Dates */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014

=Purposes=
Holistic evaluation of user experience in interacting with user-serving MIR systems

=Goals=
1) to inspire the development of complete MIR systems
2) to promote the notion of user experience as a first-class research objective in the MIR community

=Dataset=
A set of music audio of 10,000 tracks is provided for the GC14UX. It will be a subset of tracks drawn from Jamendo collection's CC-BY licensed works (http://www.jamendo.com/en/welcome).

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the CC-BY license (http://creativecommons.org/licenses/by/3.0/). The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

=Evaluation=
==Task==
To ensure that the GC14UX does not become a system-centered evaluation in disguise, the process will remain as agnostic as possible concerning the technological means by which participating systems create and deliver their experiences to the users. The only requirement for the systems is they should support music discovery (in whichever ways).
To design some criteria that’s not tied to the music content;

Task: You are creating a YouTube video and you need to find some open-source music to use.

Q. What kind of video you are trying to find the music for? (Need to discuss more about this.)

- Overall satisfaction (“On a scale of 0 to 100, rate your overall satisfaction with this system”)
and bands with different colors (0-20: very unsatisfied; 20-40: somewhat unsatisfied; 40-60: neutral; 60-80: somewhat satisfied; 80-100: very satisfied).

- Aesthetics (look of the interface)
- Interaction (smoothness, no stalls or bugs)
- Awareness (what’s going on, where the user is)

Ten Heuristics here: http://www.nngroup.com/articles/ten-usability-heuristics/
How to convert the 10 heuristics

Any literature support for the three items?
Textbox

*. Evaluation criterion:

Satisfaction: How much do you like using the interface?
Memorability: 
Errors: How many errors did you make? How easy is it to recover from them?
Learnability: How easy was it to figure out the interface? Does its layout match what you expect (e.g. buttons, search box locations, navigation)? Was anything confusing? How good is the interface at communicating functionality (help and documentation)?
---

This simplicity is because: 1) the GC14UX is all about how users perceive their experiences of the systems. We intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. 2) more data capturing opportunities will distract from the real user experience.

An open-ended question is provided but is optional for users to give feedback if they wish to do so.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

To prompt a scenario/context in which a user interacts with participating systems, a half-defined sentence will be provided to a user in the evaluation webforms *before* a set of participating systems are shown to the user:

“I am looking for music for ______________”

The user is free to fill in any need s/he would like to find music for, such as “drinking a cup of coffee”, or “waiting for my girlfriend”. In this way, the goal of the user is captured and the goal is as authentic to the user as possible.

==Evaluators==
They will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. It is planned that each submitted system will be evaluated by 50 to 60 evaluators. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Certificates==
The GC14UX will present certificates to all participating systems and the system(s) with the highest mean score.

=Wireframes=
[[File:GCUX_wireframe_(broad).png|900px]]

=Important Dates=

July 1: announce the GC
Sep. 21st: deadline for system submission
Sep. 28th: start the evaluation
Oct. 20th: close the evaluation system
Oct. 27th: announce the results
Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T01:30:03Z

Xiao:

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014

=Purposes=
Holistic evaluation of user experience in interacting with user-serving MIR systems

=Goals=
1) to inspire the development of complete MIR systems
2) to promote the notion of user experience as a first-class research objective in the MIR community

=Dataset=
A set of music audio of 10,000 tracks is provided for the GC14UX. It will be a subset of tracks drawn from Jamendo collection's CC-BY licensed works (http://www.jamendo.com/en/welcome).

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the CC-BY license (http://creativecommons.org/licenses/by/3.0/). The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

=Evaluation=
==Task==
To ensure that the GC14UX does not become a system-centered evaluation in disguise, the process will remain as agnostic as possible concerning the technological means by which participating systems create and deliver their experiences to the users. The only requirement for the systems is they should support music discovery (in whichever ways).
To design some criteria that’s not tied to the music content;

Task: You are creating a YouTube video and you need to find some open-source music to use.

Q. What kind of video you are trying to find the music for? (Need to discuss more about this.)

- Overall satisfaction (“On a scale of 0 to 100, rate your overall satisfaction with this system”)
and bands with different colors (0-20: very unsatisfied; 20-40: somewhat unsatisfied; 40-60: neutral; 60-80: somewhat satisfied; 80-100: very satisfied).

- Aesthetics (look of the interface)
- Interaction (smoothness, no stalls or bugs)
- Awareness (what’s going on, where the user is)

Ten Heuristics here: http://www.nngroup.com/articles/ten-usability-heuristics/
How to convert the 10 heuristics

Any literature support for the three items?
Textbox

*. Evaluation criterion:

Satisfaction: How much do you like using the interface?
Memorability: 
Errors: How many errors did you make? How easy is it to recover from them?
Learnability: How easy was it to figure out the interface? Does its layout match what you expect (e.g. buttons, search box locations, navigation)? Was anything confusing? How good is the interface at communicating functionality (help and documentation)?
---

This simplicity is because: 1) the GC14UX is all about how users perceive their experiences of the systems. We intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. 2) more data capturing opportunities will distract from the real user experience.

An open-ended question is provided but is optional for users to give feedback if they wish to do so.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

To prompt a scenario/context in which a user interacts with participating systems, a half-defined sentence will be provided to a user in the evaluation webforms *before* a set of participating systems are shown to the user:

“I am looking for music for ______________”

The user is free to fill in any need s/he would like to find music for, such as “drinking a cup of coffee”, or “waiting for my girlfriend”. In this way, the goal of the user is captured and the goal is as authentic to the user as possible.

==Evaluators==
They will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. It is planned that each submitted system will be evaluated by 50 to 60 evaluators. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

==Certificates==
The GC14UX will present certificates to all participating systems and the system(s) with the highest mean score.

=Wireframes=
[[File:GCUX_wireframe_(broad).png|900px]]

=Important Dates=

2013:Audio K-POP Mood Classification

2013-07-31T08:38:03Z

Xiao: /* Data */

= K-POP Mood Classification =

== Description ==

This year, IMIRSEL and KETI collaboratively developed a mood classification dataset with K-POP music. There are 1894 K-POP songs in five mood categories, annotated by a number of American annotators and Korean annotators. The goals of this task are 1)to see if mood classification models developed on Western (or other cultural) music can be applied to K-POP music; 2)to see if classification models can be equally effective on predicting mood labels by American annotator and predicting genre labels by Korean annotator.

== Data ==

There are 1438 songs, and each song is labeled with one of the five mood categories (the same five mood categories as in the Audio Mood Classification sub-task in [[2013:Audio Classification (Train/Test) Tasks]] :

* Cluster_1: passionate, rousing, confident,boisterous, rowdy
* Cluster_2: rollicking, cheerful, fun, sweet, amiable/good natured
* Cluster_3: literate, poignant, wistful, bittersweet, autumnal, brooding
* Cluster_4: humorous, silly, campy, quirky, whimsical, witty, wry
* Cluster_5: aggressive, fiery,tense/anxious, intense, volatile,visceral

How the mood categories were identified can be found in Xiao Hu and J. Stephen Downie (2007). Exploring mood metadata: Relationships with genre, artist and usage metadata, ''Proceedings of the 8th International Conference on Music Information Retrieval (ISMIR)'' [http://ismir2007.ismir.net/proceedings/ISMIR2007_p067_hu.pdf] .

The songs are NOT evenly distributed across the five moods. The folds in the cross validation will be split in a stratified manner (i.e., songs in one genre is evenly distributed across folds).

The audio is in 22.05kHz mono WAV format, each clip is 30 seconds long.

There will be two independent tasks: one uses genre annotations by American annotators, the other uses those by Korean annotators.

=== Annotations ===
Each song is annotated by three American annotators and three Korean annotators. The songs used in this task have majority vote (2 of 3 annotators agreed) or unanimous agreement (all 3 annotators agreed) on one of the five mood categories.

== Evaluation ==
The evaluation method is the same as that in the mood classification sub-tasks in Audio Classification (Train/Test) Task.
Participating algorithms will be evaluated with 3-fold cross validation. Artist filtering will be used in the test and training splits, i.e. training and test sets will contain different artists.

The raw classification (identification) accuracy, standard deviation and a confusion matrix for each algorithm will be computed.

Classification accuracies will be tested for statistically significant differences using Friedman's Anova with Tukey-Kramer honestly significant difference (HSD) tests for multiple comparisons. This test will be used to rank the algorithms and to group them into sets of equivalent performance.

In addition computation times for feature extraction and training/classification will be measured.

== Submission Format ==
The following specifications are the same as those in [[2013:Audio Classification (Train/Test) Tasks]]

=== File I/O Format ===
The audio files to be used in these tasks will be specified in a simple ASCII list file. The formats for the list files are specified below:

==== Feature extraction list file ====
The list file passed for feature extraction will be a simple ASCII list file. This file will contain one path per line with no header line.
I.e.
<example path and filename>

E.g.
/path/to/track1.wav
/path/to/track2.wav
...

==== Training list file ====
The list file passed for model training will be a simple ASCII list file. This file will contain one path per line, followed by a tab character and the class (artist, genre or mood) label, again with no header line.

I.e.
<example path and filename>\t<class label>

E.g.
/path/to/track1.wav rock
/path/to/track2.wav blues
...

==== Test (classification) list file ====
The list file passed for testing classification will be a simple ASCII list file identical in format to the Feature extraction list file. This file will contain one path per line with no header line.

I.e.
<example path and filename>

E.g.
/path/to/track1.wav
/path/to/track2.wav
...

==== Classification output file ====
Participating algorithms should produce a simple ASCII list file identical in format to the Training list file. This file will contain one path per line, followed by a tab character and the artist label, again with no header line.

I.e.
<example path and filename>\t<class label>

E.g.
/path/to/track1.wav classical
/path/to/track2.wav blues
...

=== Submission calling formats ===
Algorithms should divide their feature extraction and training/classification into separate runs. This will facilitate a single feature extraction step for the task, while training and classification can be run for each cross-validation fold.

Hence, participants should provide two executables or command line parameters for a single executable to run the two separate processes.

Executables will have to accept the paths to the aforementioned list files as command line parameters.

Scratch folders will be provided for all submissions for the storage of feature files and any model files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique file names will be assigned to each audio track.

==== Example submission calling formats ====

extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt
TrainAndClassify.sh /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt
Train.sh /path/to/scratch/folder /path/to/trainListFile.txt
Classify.sh /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt

myAlgo.sh -extract /path/to/scratch/folder /path/to/featureExtractionListFile.txt
myAlgo.sh -train /path/to/scratch/folder /path/to/trainListFile.txt
myAlgo.sh -classify /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt

Multi-processor compute nodes will be used to run this task, however, we ask that submissions use no more than 4 cores (as we will be running a lot of submissions and will need to run some in parallel). Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 1, 2 or 4 thread/core configurations.

extractFeatures.sh -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt
TrainAndClassify.sh -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

myAlgo.sh -extract -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt
myAlgo.sh -TrainAndClassify -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

=== Packaging submissions ===

* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).
* Be sure to follow the [[2006:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2013 Submission Instructions]]

All submissions should include a README file including the following the information:

* Command line calling format for all executables including examples
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Approximately how much scratch disk space will the submission need to store any feature/cache files?
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.
* Any special notice regarding to running your algorithm

Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.

=== Time and hardware limits ===
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.

A hard limit of 24 hours will be imposed on feature extraction times.

A hard limit of 48 hours will be imposed on the 3 training/classification cycles, leading to a total runtime limit of 72 hours for each submission.

=== Potential Participants ===
name / email

2013:Task Captains

2013-07-31T08:11:02Z

Xiao:

In response to discussions at ISMIR 2012, we are prepared to improve the distribution of tasks for the upcoming MIREX 2013. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead one or more tasks, please add your name in the "Captains" column.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

{| class="wikitable" style="margin-left: 20px"
!ID !! Task !! Captain(s)
|-
|abt
|[[2013:Audio Beat Tracking]]
|Fu-Hai Frank Wu, (Andreas Ehmann?)
|-
|ace
|[[2013:Audio Chord Estimation]]
|John Ashley Burgoyne, W. Bas de Haas, Johan Pauwels
|-
|act
|[[2013:Audio Classification (Train/Test) Tasks]]
|IMIRSEL
|-
|acs
|[[2013:Audio Cover Song Identification]]
|
|-
|akd
|[[2013:Audio Key Detection]]
|Johan Pauwels
|-
|ame
|[[2013:Audio Melody Extraction]]
|KETI
|-
|ams
|[[2013:Audio Music Similarity and Retrieval]]
|IMIRSEL
|-
|aod
|[[2013:Audio Onset Detection]]
|Sebastian Böck, (Andreas Ehmann?)
|-
|ate
|[[2013:Audio Tempo Estimation]]
|Aggelos Gkiokas, Anders Elowsson
|-
|atg
|[[2013:Audio Tag Classification]]
|(Xiao Hu?)
|-
|mf0
|[[2013:Multiple Fundamental Frequency Estimation & Tracking]]
|Mert Bay
|-
|qbsh
|[[2013:Query by Singing/Humming]]
|KETI
|-
|qbt
|[[2013:Query by Tapping]]
|KETI
|-
|scofo
|[[2013:Real-time Audio to Score Alignment (a.k.a Score Following)]]
|
|-
|sms
|[[2013:Symbolic Melodic Similarity]]
|IMIRSEL
|-
|struct
|[[2013:Structural Segmentation]]
|
|-
|drts
|[[2013:Discovery of Repeated Themes & Sections]]
|Tom Collins
|-
|kgc
|[[2013:Audio K-POP Genre Classification]]
|Kahyun Choi, Peter Organisciak
|-
|kmc
|[[2013:Audio K-POP Mood Classification]]
|Kahyun Choi, Peter Organisciak
|}

2013:Task Captains

2013-07-31T08:10:29Z

Xiao:

In response to discussions at ISMIR 2012, we are prepared to improve the distribution of tasks for the upcoming MIREX 2013. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead one or more tasks, please add your name in the "Captains" column.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

{| class="wikitable" style="margin-left: 20px"
!ID !! Task !! Captain(s)
|-
|abt
|[[2013:Audio Beat Tracking]]
|Fu-Hai Frank Wu, (Andreas Ehmann?)
|-
|ace
|[[2013:Audio Chord Estimation]]
|John Ashley Burgoyne, W. Bas de Haas, Johan Pauwels
|-
|act
|[[2013:Audio Classification (Train/Test) Tasks]]
|IMIRSEL
|-
|acs
|[[2013:Audio Cover Song Identification]]
|
|-
|akd
|[[2013:Audio Key Detection]]
|Johan Pauwels
|-
|ame
|[[2013:Audio Melody Extraction]]
|KETI
|-
|ams
|[[2013:Audio Music Similarity and Retrieval]]
|IMIRSEL
|-
|aod
|[[2013:Audio Onset Detection]]
|Sebastian Böck, (Andreas Ehmann?)
|-
|ate
|[[2013:Audio Tempo Estimation]]
|Aggelos Gkiokas, Anders Elowsson
|-
|atg
|[[2013:Audio Tag Classification]]
|(Xiao Hu?)
|-
|mf0
|[[2013:Multiple Fundamental Frequency Estimation & Tracking]]
|Mert Bay
|-
|qbsh
|[[2013:Query by Singing/Humming]]
|KETI
|-
|qbt
|[[2013:Query by Tapping]]
|KETI
|-
|scofo
|[[2013:Real-time Audio to Score Alignment (a.k.a Score Following)]]
|
|-
|sms
|[[2013:Symbolic Melodic Similarity]]
|IMIRSEL
|-
|struct
|[[2013:Structural Segmentation]]
|
|-
|drts
|[[2013:Discovery of Repeated Themes & Sections]]
|Tom Collins
|-
|kgc
|[[2013:Audio K-POP Genre Classification]]
|Peter Organisciak
|-
|kmc
|[[2013:Audio K-POP Mood Classification]]
|Peter Organisciak
|}

2013:Task Captains

2013-07-31T08:06:41Z

Xiao:

In response to discussions at ISMIR 2012, we are prepared to improve the distribution of tasks for the upcoming MIREX 2013. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead one or more tasks, please add your name in the "Captains" column.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

{| class="wikitable" style="margin-left: 20px"
!ID !! Task !! Captain(s)
|-
|abt
|[[2013:Audio Beat Tracking]]
|Fu-Hai Frank Wu, (Andreas Ehmann?)
|-
|ace
|[[2013:Audio Chord Estimation]]
|John Ashley Burgoyne, W. Bas de Haas, Johan Pauwels
|-
|act
|[[2013:Audio Classification (Train/Test) Tasks]]
|IMIRSEL
|-
|acs
|[[2013:Audio Cover Song Identification]]
|
|-
|akd
|[[2013:Audio Key Detection]]
|Johan Pauwels
|-
|ame
|[[2013:Audio Melody Extraction]]
|KETI
|-
|ams
|[[2013:Audio Music Similarity and Retrieval]]
|IMIRSEL
|-
|aod
|[[2013:Audio Onset Detection]]
|Sebastian Böck, (Andreas Ehmann?)
|-
|ate
|[[2013:Audio Tempo Estimation]]
|Aggelos Gkiokas, Anders Elowsson
|-
|atg
|[[2013:Audio Tag Classification]]
|(Xiao Hu?)
|-
|mf0
|[[2013:Multiple Fundamental Frequency Estimation & Tracking]]
|Mert Bay
|-
|qbsh
|[[2013:Query by Singing/Humming]]
|KETI
|-
|qbt
|[[2013:Query by Tapping]]
|KETI
|-
|scofo
|[[2013:Real-time Audio to Score Alignment (a.k.a Score Following)]]
|
|-
|sms
|[[2013:Symbolic Melodic Similarity]]
|IMIRSEL
|-
|struct
|[[2013:Structural Segmentation]]
|
|-
|drts
|[[2013:Discovery of Repeated Themes & Sections]]
|Tom Collins
|-
|kgc
|[[2013:Audio K-POP Genre Classification]]
|
|-
|kmc
|[[2013:Audio K-POP Mood Classification]]
|
|}

2013:Audio K-POP Mood Classification

2013-07-31T08:03:17Z

Xiao: Created page with "= K-POP Mood Classification = == Description == This year, IMIRSEL and KETI collaboratively developed a mood classification dataset with K-POP music. There are 1894 K-POP songs..."

= K-POP Mood Classification =

== Description ==

This year, IMIRSEL and KETI collaboratively developed a mood classification dataset with K-POP music. There are 1894 K-POP songs in five mood categories, annotated by a number of American annotators and Korean annotators. The goals of this task are 1)to see if mood classification models developed on Western (or other cultural) music can be applied to K-POP music; 2)to see if classification models can be equally effective on predicting mood labels by American annotator and predicting genre labels by Korean annotator.

== Data ==

There are 1894 songs, and each song is labeled with one of the five mood categories (the same five mood categories as in the Audio Mood Classification sub-task in [[2013:Audio Classification (Train/Test) Tasks]] :

* Cluster_1: passionate, rousing, confident,boisterous, rowdy
* Cluster_2: rollicking, cheerful, fun, sweet, amiable/good natured
* Cluster_3: literate, poignant, wistful, bittersweet, autumnal, brooding
* Cluster_4: humorous, silly, campy, quirky, whimsical, witty, wry
* Cluster_5: aggressive, fiery,tense/anxious, intense, volatile,visceral

5 mood categories each of which contains 120 clips:
How the mood categories were identified can be found in Xiao Hu and J. Stephen Downie (2007). Exploring mood metadata: Relationships with genre, artist and usage metadata, ''Proceedings of the 8th International Conference on Music Information Retrieval (ISMIR)'' [http://ismir2007.ismir.net/proceedings/ISMIR2007_p067_hu.pdf] .

The songs are NOT evenly distributed across the five moods. The folds in the cross validation will be split in a stratified manner (i.e., songs in one genre is evenly distributed across folds).

The audio is in 22.05kHz mono WAV format, each clip is 30 seconds long.

There will be two independent tasks: one uses genre annotations by American annotators, the other uses those by Korean annotators.

== Evaluation ==
The evaluation method is the same as that in the mood classification sub-tasks in Audio Classification (Train/Test) Task.
Participating algorithms will be evaluated with 3-fold cross validation. Artist filtering will be used in the test and training splits, i.e. training and test sets will contain different artists.

The raw classification (identification) accuracy, standard deviation and a confusion matrix for each algorithm will be computed.

Classification accuracies will be tested for statistically significant differences using Friedman's Anova with Tukey-Kramer honestly significant difference (HSD) tests for multiple comparisons. This test will be used to rank the algorithms and to group them into sets of equivalent performance.

In addition computation times for feature extraction and training/classification will be measured.

== Submission Format ==
The following specifications are the same as those in [[2013:Audio Classification (Train/Test) Tasks]]

=== File I/O Format ===
The audio files to be used in these tasks will be specified in a simple ASCII list file. The formats for the list files are specified below:

==== Feature extraction list file ====
The list file passed for feature extraction will be a simple ASCII list file. This file will contain one path per line with no header line.
I.e.
<example path and filename>

E.g.
/path/to/track1.wav
/path/to/track2.wav
...

==== Training list file ====
The list file passed for model training will be a simple ASCII list file. This file will contain one path per line, followed by a tab character and the class (artist, genre or mood) label, again with no header line.

I.e.
<example path and filename>\t<class label>

E.g.
/path/to/track1.wav rock
/path/to/track2.wav blues
...

==== Test (classification) list file ====
The list file passed for testing classification will be a simple ASCII list file identical in format to the Feature extraction list file. This file will contain one path per line with no header line.

I.e.
<example path and filename>

E.g.
/path/to/track1.wav
/path/to/track2.wav
...

==== Classification output file ====
Participating algorithms should produce a simple ASCII list file identical in format to the Training list file. This file will contain one path per line, followed by a tab character and the artist label, again with no header line.

I.e.
<example path and filename>\t<class label>

E.g.
/path/to/track1.wav classical
/path/to/track2.wav blues
...

=== Submission calling formats ===
Algorithms should divide their feature extraction and training/classification into separate runs. This will facilitate a single feature extraction step for the task, while training and classification can be run for each cross-validation fold.

Hence, participants should provide two executables or command line parameters for a single executable to run the two separate processes.

Executables will have to accept the paths to the aforementioned list files as command line parameters.

Scratch folders will be provided for all submissions for the storage of feature files and any model files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique file names will be assigned to each audio track.

==== Example submission calling formats ====

extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt
TrainAndClassify.sh /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt
Train.sh /path/to/scratch/folder /path/to/trainListFile.txt
Classify.sh /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt

myAlgo.sh -extract /path/to/scratch/folder /path/to/featureExtractionListFile.txt
myAlgo.sh -train /path/to/scratch/folder /path/to/trainListFile.txt
myAlgo.sh -classify /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt

Multi-processor compute nodes will be used to run this task, however, we ask that submissions use no more than 4 cores (as we will be running a lot of submissions and will need to run some in parallel). Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 1, 2 or 4 thread/core configurations.

extractFeatures.sh -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt
TrainAndClassify.sh -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

myAlgo.sh -extract -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt
myAlgo.sh -TrainAndClassify -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

=== Packaging submissions ===

* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).
* Be sure to follow the [[2006:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2013 Submission Instructions]]

All submissions should include a README file including the following the information:

* Command line calling format for all executables including examples
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Approximately how much scratch disk space will the submission need to store any feature/cache files?
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.
* Any special notice regarding to running your algorithm

Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.

=== Time and hardware limits ===
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.

A hard limit of 24 hours will be imposed on feature extraction times.

A hard limit of 48 hours will be imposed on the 3 training/classification cycles, leading to a total runtime limit of 72 hours for each submission.

=== Potential Participants ===
name / email

2013:Audio K-POP Genre Classification

2013-07-31T07:52:58Z

Xiao: /* Data */

= K-POP Genre Classification =

== Description ==

This year, IMIRSEL and KETI collaboratively developed a genre classification dataset with K-POP music. There are 1894 K-POP songs in eight genres, annotated by one American annotator and one Korean annotator. The goals of this task are 1)to see if genre classification models developed on Western (or other cultural) music can be applied to K-POP music; 2)to see if classification models can be equally effective on predicting genre labels by American annotator and predicting genre labels by Korean annotator.

== Data ==

There are 1894 songs, and each song is labeled with one of the eight genre labels:
* Ballad
* Dance
* Folk
* Hip-hop
* R&B
* Rock
* Trot
* Other
How the genres were identified can be found in Jin Ha Lee, Kahyun Choi, Xiao Hu and J. Stephen Downie (2013) K-POP GENRES: A CROSS-CULTURAL EXPLORATION, Proceedings of the 14th International Conference on Music Information Retrieval (ISMIR).

The songs are NOT evenly distributed across the eight genres. The folds in the cross validation will be split in a stratified manner (i.e., songs in one genre is evenly distributed across folds).

The audio is in 22.05kHz mono WAV format, each clip is 30 seconds long.

There will be two independent tasks: one uses genre annotations by the American annotator, the other uses those by the Korean annotator.

== Evaluation ==
The evaluation method is the same as that in the genre classification sub-tasks in Audio Classification (Train/Test) Task.
Participating algorithms will be evaluated with 3-fold cross validation. Artist filtering will be used the test and training splits, i.e. training and test sets will contain different artists.

The raw classification (identification) accuracy, standard deviation and a confusion matrix for each algorithm will be computed.

Classification accuracies will be tested for statistically significant differences using Friedman's Anova with Tukey-Kramer honestly significant difference (HSD) tests for multiple comparisons. This test will be used to rank the algorithms and to group them into sets of equivalent performance.

In addition computation times for feature extraction and training/classification will be measured.

== Submission Format ==
The following specifications are the same as those in [[2013:Audio Classification (Train/Test) Tasks]]

=== File I/O Format ===
The audio files to be used in these tasks will be specified in a simple ASCII list file. The formats for the list files are specified below:

==== Feature extraction list file ====
The list file passed for feature extraction will be a simple ASCII list file. This file will contain one path per line with no header line.
I.e.
<example path and filename>

E.g.
/path/to/track1.wav
/path/to/track2.wav
...

==== Training list file ====
The list file passed for model training will be a simple ASCII list file. This file will contain one path per line, followed by a tab character and the class (artist, genre or mood) label, again with no header line.

I.e.
<example path and filename>\t<class label>

E.g.
/path/to/track1.wav rock
/path/to/track2.wav blues
...

==== Test (classification) list file ====
The list file passed for testing classification will be a simple ASCII list file identical in format to the Feature extraction list file. This file will contain one path per line with no header line.

I.e.
<example path and filename>

E.g.
/path/to/track1.wav
/path/to/track2.wav
...

==== Classification output file ====
Participating algorithms should produce a simple ASCII list file identical in format to the Training list file. This file will contain one path per line, followed by a tab character and the artist label, again with no header line.

I.e.
<example path and filename>\t<class label>

E.g.
/path/to/track1.wav classical
/path/to/track2.wav blues
...

=== Submission calling formats ===
Algorithms should divide their feature extraction and training/classification into separate runs. This will facilitate a single feature extraction step for the task, while training and classification can be run for each cross-validation fold.

Hence, participants should provide two executables or command line parameters for a single executable to run the two separate processes.

Executables will have to accept the paths to the aforementioned list files as command line parameters.

Scratch folders will be provided for all submissions for the storage of feature files and any model files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique file names will be assigned to each audio track.

==== Example submission calling formats ====

extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt
TrainAndClassify.sh /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt
Train.sh /path/to/scratch/folder /path/to/trainListFile.txt
Classify.sh /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt

myAlgo.sh -extract /path/to/scratch/folder /path/to/featureExtractionListFile.txt
myAlgo.sh -train /path/to/scratch/folder /path/to/trainListFile.txt
myAlgo.sh -classify /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt

Multi-processor compute nodes will be used to run this task, however, we ask that submissions use no more than 4 cores (as we will be running a lot of submissions and will need to run some in parallel). Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 1, 2 or 4 thread/core configurations.

extractFeatures.sh -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt
TrainAndClassify.sh -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

myAlgo.sh -extract -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt
myAlgo.sh -TrainAndClassify -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

=== Packaging submissions ===

* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).
* Be sure to follow the [[2006:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2013 Submission Instructions]]

All submissions should include a README file including the following the information:

* Command line calling format for all executables including examples
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Approximately how much scratch disk space will the submission need to store any feature/cache files?
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.
* Any special notice regarding to running your algorithm

Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.

=== Time and hardware limits ===
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.

A hard limit of 24 hours will be imposed on feature extraction times.

A hard limit of 48 hours will be imposed on the 3 training/classification cycles, leading to a total runtime limit of 72 hours for each submission.

=== Potential Participants ===
name / email

2013:Audio K-POP Genre Classification

2013-07-31T07:52:32Z

Xiao: /* Data */

= K-POP Genre Classification =

== Description ==

This year, IMIRSEL and KETI collaboratively developed a genre classification dataset with K-POP music. There are 1894 K-POP songs in eight genres, annotated by one American annotator and one Korean annotator. The goals of this task are 1)to see if genre classification models developed on Western (or other cultural) music can be applied to K-POP music; 2)to see if classification models can be equally effective on predicting genre labels by American annotator and predicting genre labels by Korean annotator.

== Data ==

There are 1894 songs, and each song is labeled with one of the eight genre labels:
** Ballad
** Dance
** Folk
** Hip-hop
** R&B
** Rock
** Trot
** Other
How the genres were identified can be found in Jin Ha Lee, Kahyun Choi, Xiao Hu and J. Stephen Downie (2013) K-POP GENRES: A CROSS-CULTURAL EXPLORATION, Proceedings of the 14th International Conference on Music Information Retrieval (ISMIR).

The songs are NOT evenly distributed across the eight genres. The folds in the cross validation will be split in a stratified manner (i.e., songs in one genre is evenly distributed across folds).

The audio is in 22.05kHz mono WAV format，　each clip is 30 seconds long.

There will be two independent tasks: one uses genre annotations by the American annotator, the other uses those by the Korean annotator.

== Evaluation ==
The evaluation method is the same as that in the genre classification sub-tasks in Audio Classification (Train/Test) Task.
Participating algorithms will be evaluated with 3-fold cross validation. Artist filtering will be used the test and training splits, i.e. training and test sets will contain different artists.

The raw classification (identification) accuracy, standard deviation and a confusion matrix for each algorithm will be computed.

Classification accuracies will be tested for statistically significant differences using Friedman's Anova with Tukey-Kramer honestly significant difference (HSD) tests for multiple comparisons. This test will be used to rank the algorithms and to group them into sets of equivalent performance.

In addition computation times for feature extraction and training/classification will be measured.

== Submission Format ==
The following specifications are the same as those in [[2013:Audio Classification (Train/Test) Tasks]]

=== File I/O Format ===
The audio files to be used in these tasks will be specified in a simple ASCII list file. The formats for the list files are specified below:

==== Feature extraction list file ====
The list file passed for feature extraction will be a simple ASCII list file. This file will contain one path per line with no header line.
I.e.
<example path and filename>

E.g.
/path/to/track1.wav
/path/to/track2.wav
...

==== Training list file ====
The list file passed for model training will be a simple ASCII list file. This file will contain one path per line, followed by a tab character and the class (artist, genre or mood) label, again with no header line.

I.e.
<example path and filename>\t<class label>

E.g.
/path/to/track1.wav rock
/path/to/track2.wav blues
...

==== Test (classification) list file ====
The list file passed for testing classification will be a simple ASCII list file identical in format to the Feature extraction list file. This file will contain one path per line with no header line.

I.e.
<example path and filename>

E.g.
/path/to/track1.wav
/path/to/track2.wav
...

==== Classification output file ====
Participating algorithms should produce a simple ASCII list file identical in format to the Training list file. This file will contain one path per line, followed by a tab character and the artist label, again with no header line.

I.e.
<example path and filename>\t<class label>

E.g.
/path/to/track1.wav classical
/path/to/track2.wav blues
...

=== Submission calling formats ===
Algorithms should divide their feature extraction and training/classification into separate runs. This will facilitate a single feature extraction step for the task, while training and classification can be run for each cross-validation fold.

Hence, participants should provide two executables or command line parameters for a single executable to run the two separate processes.

Executables will have to accept the paths to the aforementioned list files as command line parameters.

Scratch folders will be provided for all submissions for the storage of feature files and any model files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique file names will be assigned to each audio track.

==== Example submission calling formats ====

extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt
TrainAndClassify.sh /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt
Train.sh /path/to/scratch/folder /path/to/trainListFile.txt
Classify.sh /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt

myAlgo.sh -extract /path/to/scratch/folder /path/to/featureExtractionListFile.txt
myAlgo.sh -train /path/to/scratch/folder /path/to/trainListFile.txt
myAlgo.sh -classify /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt

Multi-processor compute nodes will be used to run this task, however, we ask that submissions use no more than 4 cores (as we will be running a lot of submissions and will need to run some in parallel). Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 1, 2 or 4 thread/core configurations.

extractFeatures.sh -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt
TrainAndClassify.sh -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

myAlgo.sh -extract -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt
myAlgo.sh -TrainAndClassify -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

=== Packaging submissions ===

* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).
* Be sure to follow the [[2006:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2013 Submission Instructions]]

All submissions should include a README file including the following the information:

* Command line calling format for all executables including examples
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Approximately how much scratch disk space will the submission need to store any feature/cache files?
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.
* Any special notice regarding to running your algorithm

Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.

=== Time and hardware limits ===
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.

A hard limit of 24 hours will be imposed on feature extraction times.

A hard limit of 48 hours will be imposed on the 3 training/classification cycles, leading to a total runtime limit of 72 hours for each submission.

=== Potential Participants ===
name / email

2013:Audio K-POP Genre Classification

2013-07-31T07:51:22Z

Xiao:

= K-POP Genre Classification =

== Description ==

This year, IMIRSEL and KETI collaboratively developed a genre classification dataset with K-POP music. There are 1894 K-POP songs in eight genres, annotated by one American annotator and one Korean annotator. The goals of this task are 1)to see if genre classification models developed on Western (or other cultural) music can be applied to K-POP music; 2)to see if classification models can be equally effective on predicting genre labels by American annotator and predicting genre labels by Korean annotator.

== Data ==

There are 1894 songs, and each song is labeled with one of the eight genre labels:

** Ballad
** Dance
** Folk
** Hip-hop
** R&B
** Rock
** Trot
** Other
How the genres were identified can be found in Jin Ha Lee, Kahyun Choi, Xiao Hu and J. Stephen Downie (2013) K-POP GENRES: A CROSS-CULTURAL EXPLORATION, Proceedings of the 14th International Conference on Music Information Retrieval (ISMIR).

The songs are NOT evenly distributed across the eight genres. The folds in the cross validation will be split in a stratified manner (i.e., songs in one genre is evenly distributed across folds).

The audio is in 22.05kHz mono WAV format，　each clip is 30 seconds long.

There will be two independent tasks: one uses genre annotations by the American annotator, the other uses those by the Korean annotator.

== Evaluation ==
The evaluation method is the same as that in the genre classification sub-tasks in Audio Classification (Train/Test) Task.
Participating algorithms will be evaluated with 3-fold cross validation. Artist filtering will be used the test and training splits, i.e. training and test sets will contain different artists.

The raw classification (identification) accuracy, standard deviation and a confusion matrix for each algorithm will be computed.

Classification accuracies will be tested for statistically significant differences using Friedman's Anova with Tukey-Kramer honestly significant difference (HSD) tests for multiple comparisons. This test will be used to rank the algorithms and to group them into sets of equivalent performance.

In addition computation times for feature extraction and training/classification will be measured.

== Submission Format ==
The following specifications are the same as those in [[2013:Audio Classification (Train/Test) Tasks]]

=== File I/O Format ===
The audio files to be used in these tasks will be specified in a simple ASCII list file. The formats for the list files are specified below:

==== Feature extraction list file ====
The list file passed for feature extraction will be a simple ASCII list file. This file will contain one path per line with no header line.
I.e.
<example path and filename>

E.g.
/path/to/track1.wav
/path/to/track2.wav
...

==== Training list file ====
The list file passed for model training will be a simple ASCII list file. This file will contain one path per line, followed by a tab character and the class (artist, genre or mood) label, again with no header line.

I.e.
<example path and filename>\t<class label>

E.g.
/path/to/track1.wav rock
/path/to/track2.wav blues
...

==== Test (classification) list file ====
The list file passed for testing classification will be a simple ASCII list file identical in format to the Feature extraction list file. This file will contain one path per line with no header line.

I.e.
<example path and filename>

E.g.
/path/to/track1.wav
/path/to/track2.wav
...

==== Classification output file ====
Participating algorithms should produce a simple ASCII list file identical in format to the Training list file. This file will contain one path per line, followed by a tab character and the artist label, again with no header line.

I.e.
<example path and filename>\t<class label>

E.g.
/path/to/track1.wav classical
/path/to/track2.wav blues
...

=== Submission calling formats ===
Algorithms should divide their feature extraction and training/classification into separate runs. This will facilitate a single feature extraction step for the task, while training and classification can be run for each cross-validation fold.

Hence, participants should provide two executables or command line parameters for a single executable to run the two separate processes.

Executables will have to accept the paths to the aforementioned list files as command line parameters.

Scratch folders will be provided for all submissions for the storage of feature files and any model files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique file names will be assigned to each audio track.

==== Example submission calling formats ====

extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt
TrainAndClassify.sh /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt
Train.sh /path/to/scratch/folder /path/to/trainListFile.txt
Classify.sh /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt

myAlgo.sh -extract /path/to/scratch/folder /path/to/featureExtractionListFile.txt
myAlgo.sh -train /path/to/scratch/folder /path/to/trainListFile.txt
myAlgo.sh -classify /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt

Multi-processor compute nodes will be used to run this task, however, we ask that submissions use no more than 4 cores (as we will be running a lot of submissions and will need to run some in parallel). Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 1, 2 or 4 thread/core configurations.

extractFeatures.sh -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt
TrainAndClassify.sh -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

myAlgo.sh -extract -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt
myAlgo.sh -TrainAndClassify -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

=== Packaging submissions ===

* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).
* Be sure to follow the [[2006:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2013 Submission Instructions]]

All submissions should include a README file including the following the information:

* Command line calling format for all executables including examples
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Approximately how much scratch disk space will the submission need to store any feature/cache files?
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.
* Any special notice regarding to running your algorithm

Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.

=== Time and hardware limits ===
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.

A hard limit of 24 hours will be imposed on feature extraction times.

A hard limit of 48 hours will be imposed on the 3 training/classification cycles, leading to a total runtime limit of 72 hours for each submission.

=== Potential Participants ===
name / email

2013:Audio K-POP Genre Classification

2013-07-31T07:42:53Z

Xiao: Created page with "This year, IMIRSEL and KETI collaboratively developed a genre classification dataset with K-POP music. There are 1894 K-POP songs in eight genres, annotated by one American annot..."

This year, IMIRSEL and KETI collaboratively developed a genre classification dataset with K-POP music. There are 1894 K-POP songs in eight genres, annotated by one American annotator and one Korean annotator. The goals of this task are 1)to see if genre classification models developed on Western (or other cultural) music can be applied to K-POP music; 2)to see if classification models can be equally effective on predicting genre labels by American annotator and predicting genre labels by Korean annotator.

The genres are:
** Ballad
** Dance
** Folk
** Hip-hop
** R&B
** Rock
** Trot
** Other
How the genres were identified can be found in Jin Ha Lee, Kahyun Choi, Xiao Hu and J. Stephen Downie (2013) K-POP GENRES: A CROSS-CULTURAL EXPLORATION, Proceedings of the 14th International Conference on Music Information Retrieval (ISMIR).

The audio is in 22.05kHz mono WAV format，　each clip is 30 seconds long.

There will be two independent tasks: one uses genre annotations by the American annotator, the other uses those by the Korean annotator.

== Evaluation ==
The evaluation method is the same as that in the genre classification sub-tasks in Audio Classification (Train/Test) Task.
Participating algorithms will be evaluated with 3-fold cross validation. Artist filtering will be used the test and training splits, i.e. training and test sets will contain different artists.

The raw classification (identification) accuracy, standard deviation and a confusion matrix for each algorithm will be computed.

Classification accuracies will be tested for statistically significant differences using Friedman's Anova with Tukey-Kramer honestly significant difference (HSD) tests for multiple comparisons. This test will be used to rank the algorithms and to group them into sets of equivalent performance.

In addition computation times for feature extraction and training/classification will be measured.

== Submission Format ==
The following specifications are the same as those in [[2013:Audio Classification (Train/Test) Tasks]]

=== File I/O Format ===
The audio files to be used in these tasks will be specified in a simple ASCII list file. The formats for the list files are specified below:

==== Feature extraction list file ====
The list file passed for feature extraction will be a simple ASCII list file. This file will contain one path per line with no header line.
I.e.
<example path and filename>

E.g.
/path/to/track1.wav
/path/to/track2.wav
...

==== Training list file ====
The list file passed for model training will be a simple ASCII list file. This file will contain one path per line, followed by a tab character and the class (artist, genre or mood) label, again with no header line.

I.e.
<example path and filename>\t<class label>

E.g.
/path/to/track1.wav rock
/path/to/track2.wav blues
...

==== Test (classification) list file ====
The list file passed for testing classification will be a simple ASCII list file identical in format to the Feature extraction list file. This file will contain one path per line with no header line.

I.e.
<example path and filename>

E.g.
/path/to/track1.wav
/path/to/track2.wav
...

==== Classification output file ====
Participating algorithms should produce a simple ASCII list file identical in format to the Training list file. This file will contain one path per line, followed by a tab character and the artist label, again with no header line.

I.e.
<example path and filename>\t<class label>

E.g.
/path/to/track1.wav classical
/path/to/track2.wav blues
...

=== Submission calling formats ===
Algorithms should divide their feature extraction and training/classification into separate runs. This will facilitate a single feature extraction step for the task, while training and classification can be run for each cross-validation fold.

Hence, participants should provide two executables or command line parameters for a single executable to run the two separate processes.

Executables will have to accept the paths to the aforementioned list files as command line parameters.

Scratch folders will be provided for all submissions for the storage of feature files and any model files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique file names will be assigned to each audio track.

==== Example submission calling formats ====

extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt
TrainAndClassify.sh /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt
Train.sh /path/to/scratch/folder /path/to/trainListFile.txt
Classify.sh /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt

myAlgo.sh -extract /path/to/scratch/folder /path/to/featureExtractionListFile.txt
myAlgo.sh -train /path/to/scratch/folder /path/to/trainListFile.txt
myAlgo.sh -classify /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt

Multi-processor compute nodes will be used to run this task, however, we ask that submissions use no more than 4 cores (as we will be running a lot of submissions and will need to run some in parallel). Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 1, 2 or 4 thread/core configurations.

extractFeatures.sh -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt
TrainAndClassify.sh -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

myAlgo.sh -extract -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt
myAlgo.sh -TrainAndClassify -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt

=== Packaging submissions ===

* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).
* Be sure to follow the [[2006:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2013 Submission Instructions]]

All submissions should include a README file including the following the information:

* Command line calling format for all executables including examples
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Approximately how much scratch disk space will the submission need to store any feature/cache files?
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.
* Any special notice regarding to running your algorithm

Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.

=== Time and hardware limits ===
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.

A hard limit of 24 hours will be imposed on feature extraction times.

A hard limit of 48 hours will be imposed on the 3 training/classification cycles, leading to a total runtime limit of 72 hours for each submission.

=== Potential Participants ===
name / email

2013:Main Page

2013-07-31T04:29:49Z

Xiao: /* MIREX 2013 Possible Evaluation Tasks */

==Welcome to MIREX 2013==

This is the main page for the ninth running of the Music Information Retrieval Evaluation eXchange (MIREX 2013). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2013.

The MIREX 2013 community will hold its annual meeting as part of [http://ismir2013.ismir.net/ The 14th International Conference on Music Information Retrieval], ISMIR 2013, which will be held in Curitiba, PR, Brazil, the 4-8 November, 2013. The MIREX plenary and poster sessions will be held during the conference.

J. Stephen Downie 
Director, IMIRSEL 

==New Task Leadership Model==

In response to discussions at ISMIR 2012, we are prepared to improve the distribution of tasks for the upcoming MIREX 2013. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead a task, please add your name to the "Captains" column on the new [[2013:Task Captains]] page. Please direct any communication to the [https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest] mailing list.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

We really need leaders to help us this year!

==MIREX 2013 Deadline Dates==

TBA: 2013 dates have not been established.

Nota Bene: In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.

Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.

PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.

==MIREX 2013 Submission Instructions==
* Be sure to read through the rest of this page
* Be sure to read though the task pages for which you are submitting
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2013 Submission Instructions]] including both the tutorial video and the text

==MIREX 2013 Possible Evaluation Tasks==

* [[2013:Audio Classification (Train/Test) Tasks]], incorporating:
** Audio US Pop Genre Classification
** Audio Latin Genre Classification
** Audio Music Mood Classification
** Audio Classical Composer Identification
* [[2013:Audio Cover Song Identification]]
* [[2013:Audio Tag Classification]]
* [[2013:Audio Music Similarity and Retrieval]]
* [[2013:Symbolic Melodic Similarity]]
* [[2013:Audio Onset Detection]]
* [[2013:Audio Key Detection]]
* [[2013:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2013:Query by Singing/Humming]]
* [[2013:Audio Melody Extraction]]
* [[2013:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2013:Audio Chord Estimation]]
* [[2013:Query by Tapping]]
* [[2013:Audio Beat Tracking]]
* [[2013:Structural Segmentation]]
* [[2013:Audio Tempo Estimation]]
* [[2013:Discovery of Repeated Themes & Sections]]
* [[2013:Audio K-POP Genre Classification]]
* [[2013:Audio K-POP Mood Classification]]

===Note to New Participants===
Please take the time to read the following review articles that explain the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

Downie, J. Stephen, Andreas F. Ehmann, Mert Bay and M. Cameron Jones. (2010). 
The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. 
''Advances in Music Information Retrieval'' Vol. 274, pp. 93-115 
Available at: [http://bit.ly/KpM5u5 http://bit.ly/KpM5u5]

===Runtime Limits===

We reserve the right to stop any process that exceeds runtime limits for each task. We will do our best to notify you in enough time to allow revisions, but this may not be possible in some cases. Please respect the published runtime limits.

===Note to All Participants===

Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2013 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)
# present a poster at the MIREX 2013 poster session at ISMIR 2013

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL currently supports some of the software/architecture dependencies for your submission a [https://docs.google.com/spreadsheet/viewform?formkey=dFpmNF9PUGdvd1o1OHVhMkZ4cXZvdkE6MA#gid=0 dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you.

Due to the high volume of submissions expected at MIREX 2013, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2013==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2013 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2013 tasks. Subscription information at:
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central].

If you are participating in MIREX 2013, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2013 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2013 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
If you find that you cannot edit a MIREX wiki page, you will need to create a new account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2012 Wikis==
Content from MIREX 2005 - 2012 are available at:
'''[[2012:Main_Page|MIREX 2012]]'''
'''[[2011:Main_Page|MIREX 2011]]'''
'''[[2010:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

2013:Main Page

2013-07-31T04:28:26Z

Xiao: /* MIREX 2013 Possible Evaluation Tasks */

==Welcome to MIREX 2013==

This is the main page for the ninth running of the Music Information Retrieval Evaluation eXchange (MIREX 2013). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2013.

The MIREX 2013 community will hold its annual meeting as part of [http://ismir2013.ismir.net/ The 14th International Conference on Music Information Retrieval], ISMIR 2013, which will be held in Curitiba, PR, Brazil, the 4-8 November, 2013. The MIREX plenary and poster sessions will be held during the conference.

J. Stephen Downie 
Director, IMIRSEL 

==New Task Leadership Model==

In response to discussions at ISMIR 2012, we are prepared to improve the distribution of tasks for the upcoming MIREX 2013. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead a task, please add your name to the "Captains" column on the new [[2013:Task Captains]] page. Please direct any communication to the [https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest] mailing list.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

We really need leaders to help us this year!

==MIREX 2013 Deadline Dates==

TBA: 2013 dates have not been established.

Nota Bene: In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.

Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.

PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.

==MIREX 2013 Submission Instructions==
* Be sure to read through the rest of this page
* Be sure to read though the task pages for which you are submitting
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2013 Submission Instructions]] including both the tutorial video and the text

==MIREX 2013 Possible Evaluation Tasks==

* [[2013:Audio Classification (Train/Test) Tasks]], incorporating:
** Audio US Pop Genre Classification
** Audio Latin Genre Classification
** Audio Music Mood Classification
** Audio Classical Composer Identification
** Audio K-POP Genre Classification
** Audio K-POP Mood Classification
* [[2013:Audio Cover Song Identification]]
* [[2013:Audio Tag Classification]]
* [[2013:Audio Music Similarity and Retrieval]]
* [[2013:Symbolic Melodic Similarity]]
* [[2013:Audio Onset Detection]]
* [[2013:Audio Key Detection]]
* [[2013:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2013:Query by Singing/Humming]]
* [[2013:Audio Melody Extraction]]
* [[2013:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2013:Audio Chord Estimation]]
* [[2013:Query by Tapping]]
* [[2013:Audio Beat Tracking]]
* [[2013:Structural Segmentation]]
* [[2013:Audio Tempo Estimation]]
* [[2013:Discovery of Repeated Themes & Sections]]

===Note to New Participants===
Please take the time to read the following review articles that explain the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

Downie, J. Stephen, Andreas F. Ehmann, Mert Bay and M. Cameron Jones. (2010). 
The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. 
''Advances in Music Information Retrieval'' Vol. 274, pp. 93-115 
Available at: [http://bit.ly/KpM5u5 http://bit.ly/KpM5u5]

===Runtime Limits===

We reserve the right to stop any process that exceeds runtime limits for each task. We will do our best to notify you in enough time to allow revisions, but this may not be possible in some cases. Please respect the published runtime limits.

===Note to All Participants===

Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2013 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)
# present a poster at the MIREX 2013 poster session at ISMIR 2013

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL currently supports some of the software/architecture dependencies for your submission a [https://docs.google.com/spreadsheet/viewform?formkey=dFpmNF9PUGdvd1o1OHVhMkZ4cXZvdkE6MA#gid=0 dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you.

Due to the high volume of submissions expected at MIREX 2013, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2013==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2013 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2013 tasks. Subscription information at:
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central].

If you are participating in MIREX 2013, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2013 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2013 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
If you find that you cannot edit a MIREX wiki page, you will need to create a new account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2012 Wikis==
Content from MIREX 2005 - 2012 are available at:
'''[[2012:Main_Page|MIREX 2012]]'''
'''[[2011:Main_Page|MIREX 2011]]'''
'''[[2010:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

2010:Symbolic Melodic Similarity Results

2010-08-02T19:40:05Z

Xiao: /* Friedman Test with Multiple Comparisons Results (p=0.05) */

==Introduction==
These are the results for the 2010 running of the Symbolic Melodic Similarity task set. For background information about this task set please refer to the [[2010:Symbolic Melodic Similarity]] page.

Each system was given a query and returned the 10 most melodically similar songs from those taken from the Essen Collection (5274 pieces in the MIDI format; see [http://www.esac-data.org/ ESAC Data Homepage] for more information). For each query, we made four classes of error-mutations, thus the set comprises the following query classes:

* 0. No errors
* 1. One note deleted
* 2. One note inserted
* 3. One interval enlarged
* 4. One interval compressed

For each query (and its 4 mutations), the returned results (candidates) from all systems were then grouped together (query set) for evaluation by the human graders. The graders were provide with only heard perfect version against which to evaluate the candidates and did not know whether the candidates came from a perfect or mutated query. Each query/candidate set was evaluated by 1 individual grader. Using the Evalutron 6000 system, the graders gave each query/candidate pair two types of scores. Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 100).

====Evalutron 6000 Summary Data====
'''Number of evaluators''' = 6 
'''Number of evaluations per query/candidate pair''' = 1 
'''Number of queries per grader''' = 1 
'''Total number of candidates returned''' = 3900 
'''Total number of unique query/candidate pairs graded''' = 895 
'''Average number of query/candidate pairs evaluated per grader: 149 
'''Number of queries''' = 6 (perfect) with each perfect query error-mutated 4 different ways = 30 

===General Legend===
== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: #efefef;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="540" | Contributors
|-
! HFRA1
| SMS simbals || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/HFRA1.pdf PDF] || [http://simbals.labri.fr Pierre Hanna], [http://simbals.labri.fr Pascal Ferraro], [http://simbals.labri.fr Matthias Robine], [http://simbals.labri.fr Julien Allali]
|-
! JU1
| SMS-Domain || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/JU1.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns], [http://www.kr.inf.uc3m.es Jorge Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]
|-
! JU2
| SMS-PitchDeriv || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/JU2.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns], [http://www.kr.inf.uc3m.es Jorge Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]
|-
! JU3
| SMS-ParamDeriv || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/JU3.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns], [http://www.kr.inf.uc3m.es Jorge Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]
|-
! JU4
| SMS-Shape || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/JU4.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns], [http://www.kr.inf.uc3m.es Jorge Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]
|-
! LL1
| CbrahmsS2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/LL1.pdf PDF] || [http://cs.helsinki.fi Mika Laitinen], [http://cs.helsinki.fi Kjell Lemström]
|-
! LL2
| CbrahmsW2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/LL2.pdf PDF] || [http://cs.helsinki.fi Mika Laitinen], [http://cs.helsinki.fi Kjell Lemström]
|-
! RI1
| UAC || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RI1.pdf PDF] || [http://grfia.dlsi.ua.es/cm/ David Rizo], [http://grfia.dlsi.ua.es/cm José Manuel Iñesta]
|-
! RI2
| UAT || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RI2.pdf PDF] || [http://grfia.dlsi.ua.es/cm/ David Rizo], [http://grfia.dlsi.ua.es/cm José Manuel Iñesta]
|-
! RI3
| UAT3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RI3.pdf PDF] || [http://grfia.dlsi.ua.es/cm/ David Rizo], [http://grfia.dlsi.ua.es/cm José Manuel Iñesta]
|-
! RI4
| UAPR || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/RI4.pdf PDF] || [http://grfia.dlsi.ua.es/cm/ David Rizo], [http://grfia.dlsi.ua.es/cm José Manuel Iñesta]
|-
! SU1
| NGR5 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/SU1.pdf PDF] || [http://ade.ishs.net Iman Suyoto], [http://www.cs.rmit.edu.au/~alu Alexandra Uitdenbogerd]
|-
! SU2
| PIOI || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2010/SU2.pdf PDF] || [http://ade.ishs.net Iman Suyoto], [http://www.cs.rmit.edu.au/~alu Alexandra Uitdenbogerd]
|}

====Broad Categories====
'''NS''' = Not Similar 
'''SS''' = Somewhat Similar 
'''VS''' = Very Similar 

====Table Headings ====
'''ADR''' = Average Dynamic Recall 
'''NRGB''' = Normalize Recall at Group Boundaries 
'''AP''' = Average Precision (non-interpolated) 
'''PND''' = Precision at N Documents 

===Calculating Summary Measures===
'''Fine'''(1) = Sum of fine-grained human similarity decisions (0-100). 
'''PSum'''(1) = Sum of human broad similarity decisions: NS=0, SS=1, VS=2. 
'''WCsum'''(1) = 'World Cup' scoring: NS=0, SS=1, VS=3 (rewards Very Similar). 
'''SDsum'''(1) = 'Stephen Downie' scoring: NS=0, SS=1, VS=4 (strongly rewards Very Similar). 
'''Greater0'''(1) = NS=0, SS=1, VS=1 (binary relevance judgment). 
'''Greater1'''(1) = NS=0, SS=0, VS=1 (binary relevance judgment using only Very Similar). 

(1)Normalized to the range 0 to 1.

==Summary Results==
===Run Times===
<csv>2010/sms/sms_runtimes.csv</csv>
===Overall Scores (Includes Perfect and Error Candidates)===
<csv>2010/sms/Overall.csv</csv>

===Scores by Query Error Types===

<csv>2010/sms/No_Errors.csv</csv>

<csv>2010/sms/Deleted.csv</csv>

<csv>2010/sms/Inserted.csv</csv>

<csv>2010/sms/Enlarged.csv</csv>

<csv>2010/sms/Compressed.csv</csv>

===Friedman Test with Multiple Comparisons Results (p=0.05)===
The Friedman test was run in MATLAB against the Fine summary data over the 30 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv>2010/sms/sms_fine_scores_friedman.csv</csv>

[[Image:2010 sms fine scores friedmans.png]]

==Raw Scores==
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2010:Symbolic Melodic Similarity Raw Data]] page.

[[Category: Results]]