MIREX Wiki - User contributions [en]

2025:Symbolic Music Generation Results

2025-09-11T12:51:06Z

Zizzi wang: /* Results */

= Submissions =

{| class="wikitable"
|- style="font-weight:bold;"
! Team
! Extended Abstract
! Methods
|-
| RWKV (Zhou-Zheng et al.)
| [https://www.music-ir.org/mirex/wiki/MIREX_HOME]
| RWKV
|-
| PixelGen
| [https://www.music-ir.org/mirex/wiki/MIREX_HOME]
| Hierarchical Transformer
|-
| MuseCoco (BL-1)
| [https://arxiv.org/abs/2306.00110]
| Transformer
|-
| Anticipatory Music Transformer (BL-2)
| [https://arxiv.org/abs/2306.08620]
| Transformer
|-
|}

=Results=

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; vertical-align:center;"
! rowspan="2" | Team
! colspan="4" | Subjective Evaluation
|- style="font-weight:bold; vertical-align:center;"
| Coherecy ↑
| Structure ↑
| Creativity ↑
| Musicality ↑
|-
| RWKV (Zhou-Zheng et al.)
| 3.57 ± 0.10a
| 3.58 ± 0.10a
| 3.26 ± 0.10a
| '''3.5 ± 0.10a'''
|-
| PixelGen
| 2.39 ± 0.10c
| 2.37 ± 0.09c
| 2.85 ± 0.09b
| 2.48 ± 0.09c
|-
| MuseCoco (BL-1)
| 3.11 ± 0.10b
| 3.07 ± 0.09b
| 3.08 ± 0.09ab
| 2.95 ± 0.09b
|-
| Anticipatory Music Transformer (BL-2)
| '''3.70 ± 0.10c'''
| '''3.69 ± 0.09b'''
| '''3.30 ± 0.10b'''
| 3.45 ± 0.10b
|}

'''Note''': Results are reported in the form of mean ± sems (sem refers to standard error of mean), where s is a letter. Different letters within a column indicate significant differences (p-value p < 0.05) based on a Wilcoxon signed rank test.

'''Subjective Evaluation Details''': One piece cherry-picked from 8 samples of each test piece, resulting in 6 pages of questions. We collect responses from 22 participants (18 complete submissions and 4 partial submissions). For complete submissions, the average completion time is 16min 59s.

2025:Symbolic Music Generation Results

2025-09-11T03:27:09Z

Zizzi wang: /* Results */

= Submissions =

{| class="wikitable"
|- style="font-weight:bold;"
! Team
! Extended Abstract
! Methods
|-
| RWKV (Zhou-Zheng et al.)
| [https://www.music-ir.org/mirex/wiki/MIREX_HOME]
| RWKV
|-
| PixelGen
| [https://www.music-ir.org/mirex/wiki/MIREX_HOME]
| Hierarchical Transformer
|-
| MuseCoco (BL-1)
| [https://arxiv.org/abs/2306.00110]
| Transformer
|-
| Anticipatory Music Transformer (BL-2)
| [https://arxiv.org/abs/2306.08620]
| Transformer
|-
|}

=Results=

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; vertical-align:center;"
! rowspan="2" | Team
! colspan="4" | Subjective Evaluation
|- style="font-weight:bold; vertical-align:center;"
| Coherecy ↑
| Structure ↑
| Creativity ↑
| Musicality ↑
|-
| RWKV (Zhou-Zheng et al.)
| 3.57 ± 0.10a
| 3.58 ± 0.10a
| 3.26 ± 0.10a
| '''3.5 ± 0.10a'''
|-
| PixelGen
| 2.39 ± 0.10c
| 2.37 ± 0.09c
| 2.85 ± 0.09b
| 2.48 ± 0.09c
|-
| MuseCoco (BL-1)
| 3.11 ± 0.10b
| 3.07 ± 0.09b
| 3.08 ± 0.09ab
| 2.95 ± 0.09b
|-
| Anticipatory Music Transformer (BL-2)
| '''3.70 ± 0.10c'''
| '''3.69 ± 0.09b'''
| '''3.30 ± 0.10b'''
| 3.45 ± 0.10b
|}

'''Note''': Results are reported in the form of mean ± sems (sem refers to standard error of mean), where s is a letter. Different letters within a column indicate significant differences (p-value p < 0.05) based on a Wilcoxon signed rank test.

'''Subjective Evaluation Details''': One piece cherry-picked from 16 samples of each test piece, resulting in 6 pages of questions. We collect responses from 22 participants (18 complete submissions and 4 partial submissions). For complete submissions, the average completion time is 16min 59s.

2025:Symbolic Music Generation Results

2025-09-11T03:24:47Z

Zizzi wang: Created page with " = Submissions = {| class="wikitable" |- style="font-weight:bold;" ! Team ! Extended Abstract ! Methods |- | RWKV (Zhou-Zheng et al.) | [https://www.music-ir.org/mirex/wiki/..."

= Submissions =

{| class="wikitable"
|- style="font-weight:bold;"
! Team
! Extended Abstract
! Methods
|-
| RWKV (Zhou-Zheng et al.)
| [https://www.music-ir.org/mirex/wiki/MIREX_HOME]
| RWKV
|-
| PixelGen
| [https://www.music-ir.org/mirex/wiki/MIREX_HOME]
| Hierarchical Transformer
|-
| MuseCoco (BL-1)
| [https://arxiv.org/abs/2306.00110]
| Transformer
|-
| Anticipatory Music Transformer (BL-2)
| [https://arxiv.org/abs/2306.08620]
| Transformer
|-
|}

=Results=

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; vertical-align:center;"
! rowspan="2" | Team
! colspan="4" | Subjective Evaluation
|- style="font-weight:bold; vertical-align:center;"
| Coherecy ↑
| Structure ↑
| Creativity ↑
| Musicality ↑
|-
| RWKV (Zhou-Zheng et al.)
| 3.57 ± 0.10a
| 3.58 ± 0.10a
| 3.26 ± 0.10a
| '''3.5 ± 0.10a'''
|-
| PixelGen
| 2.39 ± 0.10c
| 2.37 ± 0.09c
| 2.85 ± 0.09b
| 2.48 ± 0.09c
|-
| MuseCoco (BL-1)
| 3.11 ± 0.10b
| 3.07 ± 0.09b
| 3.08 ± 0.09ab
| 2.95 ± 0.09b
|-
| Anticipatory Music Transformer (BL-2)
| '''3.70 ± 0.10c'''
| '''3.69 ± 0.09b'''
| '''3.30 ± 0.10b'''
| 3.45 ± 0.10b
|}

'''Note''': Results are reported in the form of mean ± sems (sem refers to standard error of mean), where s is a letter. Different letters within a column indicate significant differences (p-value p < 0.05) based on a Wilcoxon signed rank test.

'''Objective Evaluation Details''': Each model generates 16 samples for each of 6 test pieces. Negative Log Likelihood (NLL) is computed by inputing the molody and accompaniment into the MuseCoco 1B model.

'''Subjective Evaluation Details''': One piece cherry-picked from 16 samples of each test piece, resulting in 6 pages of questions. We collect responses from 22 participants (18 complete submissions and 4 partial submissions). For complete submissions, the average completion time is 16min 59s.

2025:Symbolic Music Generation

2025-08-18T01:21:33Z

Zizzi wang: /* Submission */

=Description=
Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

Please refer to [https://github.com/ZZWaang/mirex2025-musecoco this repository] to access the baseline method and know more about the submission format.

=Data Format=
Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==
* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As described in the Evaluation and Competition Format, there are four types of submissions. Below is a list of them:

{| class="wikitable"
|-
! Task !! Submission Method !! Deadline
|-
| 2 Prompts for the test set || Email JSON files to organizers. || Aug 15, 2025
|-
| Algorithm || Email code/github link/docker to organizers. Check Algorithm Submission below. || Aug 21, 2025
|-
| Cherry-picked IDs || Email IDs to organizers. || Aug 28, 2025
|-
| Evaluation metric (optional) || Email organizers. || Aug 21, 2025
|}

==Algorithm Submission==
Participants must include a <code>generation.sh</code> script in their submission. The task captain will use the script to generate output files using the following format:

<pre>
./generation.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: path to the input .json file.
* Output Folder: path to the folder where the generated output files will be saved.
* n_sample: number of samples to generate.
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

=Baseline=

We provide a baseline algorithm in [https://github.com/ZZWaang/mirex2025-musecoco this repository]. This is modified from the model MuseCoco (Lu, P., et al. 2023). Please also refer to this code repository to check data format and generation protocol.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T07:15:00Z

Zizzi wang: /* Description */

=Description=
Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

Please refer to [https://github.com/ZZWaang/mirex2025-musecoco this repository] to access the baseline method and know more about the submission format.

=Data Format=
Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==
* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As described in the Evaluation and Competition Format, there are four types of submissions. Below is a list of them:

{| class="wikitable"
|-
! Task !! Submission Method !! Deadline
|-
| 2 Prompts for the test set || Email JSON files to organizers. || Aug 15, 2025
|-
| Algorithm || Email code/github link/docker to organizers. Check Algorithm Submission below. || Aug 21, 2025
|-
| Cherry-picked IDs || Email IDS to organizers. || Aug 28, 2025
|-
| Evaluation metric (optional) || Email organizers. || Aug 21, 2025
|}

==Algorithm Submission==
Participants must include a <code>generation.sh</code> script in their submission. The task captain will use the script to generate output files using the following format:

<pre>
./generation.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: path to the input .json file.
* Output Folder: path to the folder where the generated output files will be saved.
* n_sample: number of samples to generate.
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

=Baseline=

We provide a baseline algorithm in [https://github.com/ZZWaang/mirex2025-musecoco this repository]. This is modified from the model MuseCoco (Lu, P., et al. 2023). Please also refer to this code repository to check data format and generation protocol.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

MIREX HOME

2025-07-09T07:13:33Z

Zizzi wang: /* Task Descriptions */

==Welcome to MIREX 2025==

After a break of 3 years, we want to bring back the MIREX (Music Information Retrieval Evaluation eXchange) competition starting from 2024. We want to bring in new tasks, benchmarks, and datasets in response to the rapid development of computer music research.

The MIREX community will hold its annual meeting as part of [https://ismir.net/ The International Society for Music Information Retrieval Conference]. This year, the conference will be held in [https://ismir2025.ismir.net/ Daejeon, South Korea] from September 21-25, 2025.

In a long run, we want to make MIREX a platform for researchers to share their latest research results, to compare their systems with others, and to promote the development of the field.

==Task Descriptions==

Traditional MIR tasks
* [[2025:Audio Chord Estimation]] <TC: [mailto:jj2731@nyu.edu Junyan Jiang]>
* [[2025:Lyrics Transcription]] <TC: [mailto:ruibiny@alumni.cmu.edu Ruibin Yuan] & [mailto:jj2731@nyu.edu Junyan Jiang]>
* [[2025:Cover Song Identification]] <TC: [mailto:x.du@rochester.edu Xingjian Du] & [mailto:ruibiny@alumni.cmu.edu Ruibin Yuan]>
* [[2025:Music Structure Analysis]] <TC: [mailto:yixiao.zhang@qmul.ac.uk Yixiao Zhang] & [mailto:ruibiny@alumni.cmu.edu Ruibin Yuan]>
* [[2025:Audio Beat Tracking]] <TC: [mailto:mwysjtu@gmail.com Wenye Ma] & [mailto:yinghao.ma@qmul.ac.uk Yinghao Ma]>
* [[2025:Audio Key Detection]] <TC: [mailto:mwysjtu@gmail.com Wenye Ma] & [mailto:yinghao.ma@qmul.ac.uk Yinghao Ma]>

Modern MIR Tasks
* [[2025:Symbolic Music Generation]] <TC: [mailto:ziyu.wang@nyu.edu Ziyu Wang] & [mailto:jzhao@u.nus.edu Jingwei Zhao]>
* [[2025:Music Audio Generation]] <TC: [mailto:ruibiny@alumni.cmu.edu Ruibin Yuan]>
* [[2025:Music Description & Captioning]] <TC: [mailto:yixiao.zhang@qmul.ac.uk Yixiao Zhang] & [mailto:ruibiny@alumni.cmu.edu Ruibin Yuan]>
* [[2025:Polyphonic Transcription]] <TC: [mailto:yunglu@purdue.edu Yung-Hsiang Lu], [mailto:yun98@purdue.edu Kristen Yeon-Ji Yun], [mailto:ziyu.wang@nyu.edu Ziyu Wang], [mailto:yujia.yan@rochester.edu Yujia Yan]>
* [[2025:Song Deepfake Detection]] <TC: [mailto:you.zhang@rochester.edu Neil Zhang]>
* [[2025:Music Reasoning QA]] <TC: [mailto:yinghao.ma@qmul.ac.uk Yinghao Ma]>
* [[2025:RenCon]] (Expressive Piano Performance Rendering Competition) <TC: [mailto:huan.zhang@qmul.ac.uk Huan Zhang]>

==Call for Challenges==

Starting with MIREX 2024, we invite the ISMIR community to propose new research challenges that address cutting-edge problems in Music Information Retrieval (MIR). These challenges should aim to push the boundaries of current research and foster innovation in the field.

We also welcome challenge sponsors from both industry and research institutions, particularly those willing to contribute datasets and computational resources to support the competition.

For the format and requirements for the challenge proposal, please go to [[2025:Call for Challenges]].

===What's new:===

Starting with MIREX 2025, we invite the ISMIR community to participate in shaping the future of Music Information Retrieval (MIR) by either '''proposing new research challenges''' or '''volunteering as task captains''' for existing ones.

* '''New challenge proposals''' should aim to address cutting-edge problems and push the boundaries of current MIR research.
* '''Task captains for established tasks''' are encouraged to help revitalize previous tasks—potentially by updating evaluation methodologies, datasets, or other aspects to reflect recent advances in the field.

Task Captain Responsibilities:

* Register on the [https://www.music-ir.org/mirex MIREX Wiki] and maintain a task description page.
* Collect submissions via the MIREX submission server (or provide customized submission instructions).
* Execute and evaluate the submissions.
* Report results to MIREX and create a results page on the MIREX Wiki.
* (Optional) Present a MIREX task captain poster at the Late-Breaking and Demo (LBD) session at ISMIR 2025.

==How to Participate==

See also the general [[Submission Guidelines]].

* Read the [[Participant Agreement]] and task description carefully.
* Program your system.
* Write a 2-4 page extended abstract PDF describing your system.
* Submit your system and extended abstract to the [http://futuremirex.com/submission MIREX submission site].
* Top-performing teams will have the opportunity to present their MIREX posters at the LBD session at ISMIR 2025.

==Important Dates==

* <del>Challenge proposals due: May 9, 2025</del>
* <del>Notification of acceptance: May 16, 2025</del>
* Submission open: May 31, 2025
* Submission close: Sept 1, 2025 (Some tasks may have a different deadline; see task descriptions)
* Result published: Sept 10, 2025 (Some tasks may have a different deadline; see task descriptions)

==Contact Us==

====Email====

For general questions, feedback, and suggestions, please send messages to our mailing list [mailto:future-mirex@googlegroups.com future-mirex@googlegroups.com].

For task-specific questions, we have listed the email for each task captain [[MIREX_HOME#Task_Descriptions|here]].

====Discord Server====

For real-time discussion with the MIREX organizers or task captains, you may join our [https://discord.gg/vC2YWX29sC discord server].

Notice: some task captains are not in the discord server.

====LinkedIn Organization Page====

You may visit our LinkedIn organization page [https://www.linkedin.com/company/future-mirex/ here].

We are looking forward to seeing you at MIREX 2025!

Future MIREX Team, 2025

MIREX 2025 Organizers:
* Gus Xia, MBZUAI
* Junyan Jiang, New York University
* Akira Maezawa, Yamaha
* Ziyu Wang, New York University
* Yixiao Zhang, ByteDance Inc.
* Ruibin Yuan, Hong Kong University of Science and Technology
* J. Stephen Downie, University of Illinois

MIREX HOME

2025-07-09T07:13:05Z

Zizzi wang: /* Task Descriptions */

==Welcome to MIREX 2025==

After a break of 3 years, we want to bring back the MIREX (Music Information Retrieval Evaluation eXchange) competition starting from 2024. We want to bring in new tasks, benchmarks, and datasets in response to the rapid development of computer music research.

The MIREX community will hold its annual meeting as part of [https://ismir.net/ The International Society for Music Information Retrieval Conference]. This year, the conference will be held in [https://ismir2025.ismir.net/ Daejeon, South Korea] from September 21-25, 2025.

In a long run, we want to make MIREX a platform for researchers to share their latest research results, to compare their systems with others, and to promote the development of the field.

==Task Descriptions==

Traditional MIR tasks
* [[2025:Audio Chord Estimation]] <TC: [mailto:jj2731@nyu.edu Junyan Jiang]>
* [[2025:Lyrics Transcription]] <TC: [mailto:ruibiny@alumni.cmu.edu Ruibin Yuan] & [mailto:jj2731@nyu.edu Junyan Jiang]>
* [[2025:Cover Song Identification]] <TC: [mailto:x.du@rochester.edu Xingjian Du] & [mailto:ruibiny@alumni.cmu.edu Ruibin Yuan]>
* [[2025:Music Structure Analysis]] <TC: [mailto:yixiao.zhang@qmul.ac.uk Yixiao Zhang] & [mailto:ruibiny@alumni.cmu.edu Ruibin Yuan]>
* [[2025:Audio Beat Tracking]] <TC: [mailto:mwysjtu@gmail.com Wenye Ma] & [mailto:yinghao.ma@qmul.ac.uk Yinghao Ma]>
* [[2025:Audio Key Detection]] <TC: [mailto:mwysjtu@gmail.com Wenye Ma] & [mailto:yinghao.ma@qmul.ac.uk Yinghao Ma]>

Modern MIR Tasks
* [[2025:Symbolic Music Generation]] <TC: [mailto:ziyu.wang@nyu.edu Ziyu Wang] & [mailto: jzhao@u.nus.edu Jingwei Zhao]>
* [[2025:Music Audio Generation]] <TC: [mailto:ruibiny@alumni.cmu.edu Ruibin Yuan]>
* [[2025:Music Description & Captioning]] <TC: [mailto:yixiao.zhang@qmul.ac.uk Yixiao Zhang] & [mailto:ruibiny@alumni.cmu.edu Ruibin Yuan]>
* [[2025:Polyphonic Transcription]] <TC: [mailto:yunglu@purdue.edu Yung-Hsiang Lu], [mailto:yun98@purdue.edu Kristen Yeon-Ji Yun], [mailto:ziyu.wang@nyu.edu Ziyu Wang], [mailto:yujia.yan@rochester.edu Yujia Yan]>
* [[2025:Song Deepfake Detection]] <TC: [mailto:you.zhang@rochester.edu Neil Zhang]>
* [[2025:Music Reasoning QA]] <TC: [mailto:yinghao.ma@qmul.ac.uk Yinghao Ma]>
* [[2025:RenCon]] (Expressive Piano Performance Rendering Competition) <TC: [mailto:huan.zhang@qmul.ac.uk Huan Zhang]>

==Call for Challenges==

Starting with MIREX 2024, we invite the ISMIR community to propose new research challenges that address cutting-edge problems in Music Information Retrieval (MIR). These challenges should aim to push the boundaries of current research and foster innovation in the field.

We also welcome challenge sponsors from both industry and research institutions, particularly those willing to contribute datasets and computational resources to support the competition.

For the format and requirements for the challenge proposal, please go to [[2025:Call for Challenges]].

===What's new:===

Starting with MIREX 2025, we invite the ISMIR community to participate in shaping the future of Music Information Retrieval (MIR) by either '''proposing new research challenges''' or '''volunteering as task captains''' for existing ones.

* '''New challenge proposals''' should aim to address cutting-edge problems and push the boundaries of current MIR research.
* '''Task captains for established tasks''' are encouraged to help revitalize previous tasks—potentially by updating evaluation methodologies, datasets, or other aspects to reflect recent advances in the field.

Task Captain Responsibilities:

* Register on the [https://www.music-ir.org/mirex MIREX Wiki] and maintain a task description page.
* Collect submissions via the MIREX submission server (or provide customized submission instructions).
* Execute and evaluate the submissions.
* Report results to MIREX and create a results page on the MIREX Wiki.
* (Optional) Present a MIREX task captain poster at the Late-Breaking and Demo (LBD) session at ISMIR 2025.

==How to Participate==

See also the general [[Submission Guidelines]].

* Read the [[Participant Agreement]] and task description carefully.
* Program your system.
* Write a 2-4 page extended abstract PDF describing your system.
* Submit your system and extended abstract to the [http://futuremirex.com/submission MIREX submission site].
* Top-performing teams will have the opportunity to present their MIREX posters at the LBD session at ISMIR 2025.

==Important Dates==

* <del>Challenge proposals due: May 9, 2025</del>
* <del>Notification of acceptance: May 16, 2025</del>
* Submission open: May 31, 2025
* Submission close: Sept 1, 2025 (Some tasks may have a different deadline; see task descriptions)
* Result published: Sept 10, 2025 (Some tasks may have a different deadline; see task descriptions)

==Contact Us==

====Email====

For general questions, feedback, and suggestions, please send messages to our mailing list [mailto:future-mirex@googlegroups.com future-mirex@googlegroups.com].

For task-specific questions, we have listed the email for each task captain [[MIREX_HOME#Task_Descriptions|here]].

====Discord Server====

For real-time discussion with the MIREX organizers or task captains, you may join our [https://discord.gg/vC2YWX29sC discord server].

Notice: some task captains are not in the discord server.

====LinkedIn Organization Page====

You may visit our LinkedIn organization page [https://www.linkedin.com/company/future-mirex/ here].

We are looking forward to seeing you at MIREX 2025!

Future MIREX Team, 2025

MIREX 2025 Organizers:
* Gus Xia, MBZUAI
* Junyan Jiang, New York University
* Akira Maezawa, Yamaha
* Ziyu Wang, New York University
* Yixiao Zhang, ByteDance Inc.
* Ruibin Yuan, Hong Kong University of Science and Technology
* J. Stephen Downie, University of Illinois

2025:Symbolic Music Generation

2025-07-09T07:10:07Z

Zizzi wang: /* Description */

=Description=
Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections. Please refer to [https://github.com/ZZWaang/mirex2025-musecoco this repository] to access the baseline method and submission format.

Please check [https://github.com/ZZWaang/mirex2025-musecoco this repository] to access the baseline method and know more about the submission format.

=Data Format=
Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==
* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As described in the Evaluation and Competition Format, there are four types of submissions. Below is a list of them:

{| class="wikitable"
|-
! Task !! Submission Method !! Deadline
|-
| 2 Prompts for the test set || Email JSON files to organizers. || Aug 15, 2025
|-
| Algorithm || Email code/github link/docker to organizers. Check Algorithm Submission below. || Aug 21, 2025
|-
| Cherry-picked IDs || Email IDS to organizers. || Aug 28, 2025
|-
| Evaluation metric (optional) || Email organizers. || Aug 21, 2025
|}

==Algorithm Submission==
Participants must include a <code>generation.sh</code> script in their submission. The task captain will use the script to generate output files using the following format:

<pre>
./generation.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: path to the input .json file.
* Output Folder: path to the folder where the generated output files will be saved.
* n_sample: number of samples to generate.
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

=Baseline=

We provide a baseline algorithm in [https://github.com/ZZWaang/mirex2025-musecoco this repository]. This is modified from the model MuseCoco (Lu, P., et al. 2023). Please also refer to this code repository to check data format and generation protocol.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T07:09:14Z

Zizzi wang: /* Baseline */

=Description=
Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections. Please refer to [https://github.com/ZZWaang/mirex2025-musecoco this repository] to access the baseline method and submission format.

=Data Format=
Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==
* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As described in the Evaluation and Competition Format, there are four types of submissions. Below is a list of them:

{| class="wikitable"
|-
! Task !! Submission Method !! Deadline
|-
| 2 Prompts for the test set || Email JSON files to organizers. || Aug 15, 2025
|-
| Algorithm || Email code/github link/docker to organizers. Check Algorithm Submission below. || Aug 21, 2025
|-
| Cherry-picked IDs || Email IDS to organizers. || Aug 28, 2025
|-
| Evaluation metric (optional) || Email organizers. || Aug 21, 2025
|}

==Algorithm Submission==
Participants must include a <code>generation.sh</code> script in their submission. The task captain will use the script to generate output files using the following format:

<pre>
./generation.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: path to the input .json file.
* Output Folder: path to the folder where the generated output files will be saved.
* n_sample: number of samples to generate.
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

=Baseline=

We provide a baseline algorithm in [https://github.com/ZZWaang/mirex2025-musecoco this repository]. This is modified from the model MuseCoco (Lu, P., et al. 2023). Please also refer to this code repository to check data format and generation protocol.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T07:07:58Z

Zizzi wang: /* Description */

=Description=
Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections. Please refer to [https://github.com/ZZWaang/mirex2025-musecoco this repository] to access the baseline method and submission format.

=Data Format=
Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==
* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As described in the Evaluation and Competition Format, there are four types of submissions. Below is a list of them:

{| class="wikitable"
|-
! Task !! Submission Method !! Deadline
|-
| 2 Prompts for the test set || Email JSON files to organizers. || Aug 15, 2025
|-
| Algorithm || Email code/github link/docker to organizers. Check Algorithm Submission below. || Aug 21, 2025
|-
| Cherry-picked IDs || Email IDS to organizers. || Aug 28, 2025
|-
| Evaluation metric (optional) || Email organizers. || Aug 21, 2025
|}

==Algorithm Submission==
Participants must include a <code>generation.sh</code> script in their submission. The task captain will use the script to generate output files using the following format:

<pre>
./generation.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: path to the input .json file.
* Output Folder: path to the folder where the generated output files will be saved.
* n_sample: number of samples to generate.
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

=Baselines=

We provide a baseline algorithm in [https://github.com/ZZWaang/mirex2025-musecoco this repository]. This is modified from the model MuseCoco (Lu, P., et al. 2023). Please also refer to this code repository to check data format and generation protocol.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T07:07:30Z

Zizzi wang: /* Description */

=Description=
Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

Please refer to [https://github.com/ZZWaang/mirex2025-musecoco this repository] to access the baseline method and submission format.

=Data Format=
Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==
* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As described in the Evaluation and Competition Format, there are four types of submissions. Below is a list of them:

{| class="wikitable"
|-
! Task !! Submission Method !! Deadline
|-
| 2 Prompts for the test set || Email JSON files to organizers. || Aug 15, 2025
|-
| Algorithm || Email code/github link/docker to organizers. Check Algorithm Submission below. || Aug 21, 2025
|-
| Cherry-picked IDs || Email IDS to organizers. || Aug 28, 2025
|-
| Evaluation metric (optional) || Email organizers. || Aug 21, 2025
|}

==Algorithm Submission==
Participants must include a <code>generation.sh</code> script in their submission. The task captain will use the script to generate output files using the following format:

<pre>
./generation.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: path to the input .json file.
* Output Folder: path to the folder where the generated output files will be saved.
* n_sample: number of samples to generate.
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

=Baselines=

We provide a baseline algorithm in [https://github.com/ZZWaang/mirex2025-musecoco this repository]. This is modified from the model MuseCoco (Lu, P., et al. 2023). Please also refer to this code repository to check data format and generation protocol.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T07:05:54Z

Zizzi wang: /* Baselines */

=Description=
Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

=Data Format=
Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==
* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As described in the Evaluation and Competition Format, there are four types of submissions. Below is a list of them:

{| class="wikitable"
|-
! Task !! Submission Method !! Deadline
|-
| 2 Prompts for the test set || Email JSON files to organizers. || Aug 15, 2025
|-
| Algorithm || Email code/github link/docker to organizers. Check Algorithm Submission below. || Aug 21, 2025
|-
| Cherry-picked IDs || Email IDS to organizers. || Aug 28, 2025
|-
| Evaluation metric (optional) || Email organizers. || Aug 21, 2025
|}

==Algorithm Submission==
Participants must include a <code>generation.sh</code> script in their submission. The task captain will use the script to generate output files using the following format:

<pre>
./generation.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: path to the input .json file.
* Output Folder: path to the folder where the generated output files will be saved.
* n_sample: number of samples to generate.
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

=Baselines=

We provide a baseline algorithm in [https://github.com/ZZWaang/mirex2025-musecoco this repository]. This is modified from the model MuseCoco (Lu, P., et al. 2023). Please also refer to this code repository to check data format and generation protocol.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T07:03:44Z

Zizzi wang: /* Baselines */

=Description=
Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

=Data Format=
Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==
* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As described in the Evaluation and Competition Format, there are four types of submissions. Below is a list of them:

{| class="wikitable"
|-
! Task !! Submission Method !! Deadline
|-
| 2 Prompts for the test set || Email JSON files to organizers. || Aug 15, 2025
|-
| Algorithm || Email code/github link/docker to organizers. Check Algorithm Submission below. || Aug 21, 2025
|-
| Cherry-picked IDs || Email IDS to organizers. || Aug 28, 2025
|-
| Evaluation metric (optional) || Email organizers. || Aug 21, 2025
|}

==Algorithm Submission==
Participants must include a <code>generation.sh</code> script in their submission. The task captain will use the script to generate output files using the following format:

<pre>
./generation.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: path to the input .json file.
* Output Folder: path to the folder where the generated output files will be saved.
* n_sample: number of samples to generate.
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

=Baselines=

We provide a baseline algorithm in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main this repo].

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2024:Symbolic Music Generation

2025-07-09T07:02:59Z

Zizzi wang: /* Algorithm Submission */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, and chord progression. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

==Important Dates==

* '''Oct 8, 2024''': Submit two lead sheets as a part of the test set.
* '''Oct 15, 2024''': Submit the main algorithm.
* '''Oct 22, 2024''': Return the generated samples. The cherry-picking phase begins.
* '''Oct 25, 2024''': Submit the cherry-picked sample ids.
* '''Oct 31 - Nov 3, 2024''': Online subjective evaluation.
* '''Nov 5, 2024''': Announce the final result.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

'''Accepted submission form'''
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Baselines=

To establish a benchmark for this task, we consider the three baseline models in their official implementations:

'''WholeSongGen''' (Wang et al., 2024)
* A denoising diffusion probabilistic model (DDPM) generating piano accompaniments as piano-roll images.

'''Compose & Embellish''' (Wu and Yang, 2023)
* A Transformer-based architecture generating piano performances in beat-based event sequences.

'''AccoMontage''' (Zhao and Xia, 2021)
* A hybrid algorithm generating piano accompaniments by rule-based search and music representation learning.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte, C. Towards automatic extraction of harmony information from music signals. PhD Diss. 2010.
* Lu, P., et al. Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110 (2023).
* Wang, Z., et al. Whole-song hierarchical generation of symbolic music using cascaded diffusion models, in ICLR 2024.
* Wu, S.-L., & Yang, Y.-H. Compose & Embellish: Well-structured piano performance generation via a two-stage approach, in ICASSP 2023.
* Zhao, J., & Xia, G. Accomontage: Accompaniment arrangement via phrase selection and style transfer, in ISMIR 2021.

* Code and data format samples: [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main]

2025:Symbolic Music Generation

2025-07-09T07:01:33Z

Zizzi wang: /* Algorithm Submission */

=Description=
Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

=Data Format=
Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==
* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As described in the Evaluation and Competition Format, there are four types of submissions. Below is a list of them:

{| class="wikitable"
|-
! Task !! Submission Method !! Deadline
|-
| 2 Prompts for the test set || Email JSON files to organizers. || Aug 15, 2025
|-
| Algorithm || Email code/github link/docker to organizers. Check Algorithm Submission below. || Aug 21, 2025
|-
| Cherry-picked IDs || Email IDS to organizers. || Aug 28, 2025
|-
| Evaluation metric (optional) || Email organizers. || Aug 21, 2025
|}

==Algorithm Submission==
Participants must include a <code>generation.sh</code> script in their submission. The task captain will use the script to generate output files using the following format:

<pre>
./generation.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: path to the input .json file.
* Output Folder: path to the folder where the generated output files will be saved.
* n_sample: number of samples to generate.
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

=Baselines=

To be announced later.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T06:58:56Z

Zizzi wang: /* Submission */

=Description=
Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

=Data Format=
Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==
* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As described in the Evaluation and Competition Format, there are four types of submissions. Below is a list of them:

{| class="wikitable"
|-
! Task !! Submission Method !! Deadline
|-
| 2 Prompts for the test set || Email JSON files to organizers. || Aug 15, 2025
|-
| Algorithm || Email code/github link/docker to organizers. Check Algorithm Submission below. || Aug 21, 2025
|-
| Cherry-picked IDs || Email IDS to organizers. || Aug 28, 2025
|-
| Evaluation metric (optional) || Email organizers. || Aug 21, 2025
|}

==Algorithm Submission==

=Baselines=

To be announced later.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T06:57:21Z

Zizzi wang: /* Submission */

=Description=
Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

=Data Format=
Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==
* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As described in the Evaluation and Competition Format, there are four types of submissions. Below is a list of them:

{| class="wikitable"
|-
! Task !! Submission Method !! Deadline
|-
| 2 Prompts for the test set || Email JSON files to organizers ||
|-
| Algorithm || Email code/github link/docker to organizers ||
|-
| Cherry-picked IDs || Email IDS to organizers || Example
|-
| Evaluation metric (optional) || Email organizers ||
|}

==Algorithm Submission==
To be announced later.

=Baselines=

To be announced later.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T06:52:00Z

Zizzi wang:

=Description=
Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

=Data Format=
Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==
* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As described in the Evaluation and Competition Format, there are four types of submissions. Below is a list of them:

{| class="wikitable"
|+ Caption text
|-
! Header text !! Header text !! Header text
|-
| Example || Example || Example
|-
| Example || Example || Example
|-
| Example || Example || Example
|}

# Test set submission
# Algorithm submission
# Cherry-picked sample IDs submission
# Evaluation form submission
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
To be announced later.

=Baselines=

To be announced later.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T06:47:59Z

Zizzi wang: /* Evaluation and Competition Format */

=Description=

Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

=Data Format=

Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=

We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==

* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission
# Algorithm submission
# Cherry-picked sample IDs submission
# Evaluation form submission
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
To be announced later.

=Baselines=

To be announced later.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T06:47:25Z

Zizzi wang: /* Evaluation and Competition Format */

=Description=

Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

=Data Format=

Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=

We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==

* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission
# Algorithm submission
# Cherry-picked sample IDs submission
# Evaluation form submission
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
To be announced later.

=Baselines=

To be announced later.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T06:47:01Z

Zizzi wang: /* Evaluation and Competition Format */

=Description=

Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

=Data Format=

Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=

We will evaluate the submitted algorithms through an '''online subjective double-blind test'''. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* '''We welcome both challenge participants and non-participants to submit plans for objective evaluation.''' Evaluation methods may be incorporated as reference benchmarks and could inform the development of future evaluation metrics.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''8 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency to the prompt (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates==

* '''Aug 15, 2025''': Submit two prompts as a part of the test set.
* '''Aug 21, 2025''': Submit the main algorithm.
* '''Aug 26, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 28, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2025''': Online subjective evaluation.
* '''Sep 6, 2025''': Announce the final result.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission
# Algorithm submission
# Cherry-picked sample IDs submission
# Evaluation form submission
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
To be announced later.

=Baselines=

To be announced later.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T06:35:39Z

Zizzi wang: /* Data Example */

=Description=

Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

=Data Format=

Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates (Tentative)==

* '''Aug 7, 2025''': Submit two lead sheets as a part of the test set.
* '''Aug 15, 2025''': Submit the main algorithm.
* '''Aug 20, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 24, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2024''': Online subjective evaluation.
* '''Sep 6, 2024''': Announce the final result.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission
# Algorithm submission
# Cherry-picked sample IDs submission
# Evaluation form submission
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
To be announced later.

=Baselines=

To be announced later.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T06:35:22Z

Zizzi wang: /* Data Format */

=Description=

Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

=Data Format=

Both the input prompt and output generation should be stored in JSON format. Specifically, music is represented by a list of notes, which contains <code>start</code>, <code>pitch</code>, and <code>duration</code> attributes.

The prompt is stored under the key <code>prompt</code> and lasts 5 measures (the first measure is the pickup measure). Below is an example prompt:

<pre>
{
"prompt": [
{
"start": 16,
"pitch": 72,
"duration": 6
},
{
"start": 16,
"pitch": 57,
"duration": 14
},
...
]
}
</pre>

The generation is stored under the key <code>generation</code> and lasts 12 measures. Below is an example generation:
<pre>
# Generation
{
"generation": [
{
"start": 80,
"pitch": 40,
"duration": 4
},
{
"start": 80,
"pitch": 40,
"duration": 4
},
...
]
}

</pre>

In the above examples, <code>start</code> and <code>duration</code> attributes are counted in sixteenth notes. Since the data is assumed to be in 4/4 meter and quantized to a sixteenth note resolution, the <code>start</code> of the prompt should range from 0-79 (0-15 is the pickup measure) and <code>start</code> of the generation should range from 80-271. The <code>pitch</code> property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"prompt": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
}
</pre>

This is an example of the generated continuation. Note that the generation starts from the fifth measure (time step 80).

<pre>
{
"acc": [
{"start": 80, "pitch": 41, "duration": 12},
{"start": 80, "pitch": 65, "duration": 5},
...
]
}
</pre>

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates (Tentative)==

* '''Aug 7, 2025''': Submit two lead sheets as a part of the test set.
* '''Aug 15, 2025''': Submit the main algorithm.
* '''Aug 20, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 24, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2024''': Online subjective evaluation.
* '''Sep 6, 2024''': Announce the final result.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission
# Algorithm submission
# Cherry-picked sample IDs submission
# Evaluation form submission
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
To be announced later.

=Baselines=

To be announced later.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-07-09T06:14:31Z

Zizzi wang: /* Description */

=Description=

Symbolic music generation covers a wide range of tasks and settings, including varying types of control, generation objectives (e.g., continuation, inpainting), and representations (e.g., score, performance, single- or multi-track). In MIREX, we narrow this scope each year to focus on a specific subtask.

For this year’s challenge, the selected task is '''Piano Music Continuation'''. Given a 4-measure piano prompt (plus an optional pickup measure), the goal is to generate a 12-measure continuation that is musically coherent with the prompt, forming a complete 16-measure piece. All music is assumed to be in 4/4 time and quantized to sixteenth-note resolution. The continuation should match the style of the prompt, which may vary across classical, pop, jazz, or other existing styles. Further details are provided in the following sections.

=Data Format=
The input prompt consists of 4 bars of piano music, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing a property: <code>prompt</code>:

* <code>prompt</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>output</code>:

* <code>output</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both prompt and output, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 17 * 16 - 1 = 271. Notes that end later than the sixteenth measure (i.e., 17 * 16 = 272th time step) will be truncated to the end of the ninth measure.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"prompt": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
}
</pre>

This is an example of the generated continuation. Note that the generation starts from the fifth measure (time step 80).

<pre>
{
"acc": [
{"start": 80, "pitch": 41, "duration": 12},
{"start": 80, "pitch": 65, "duration": 5},
...
]
}
</pre>

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates (Tentative)==

* '''Aug 7, 2025''': Submit two lead sheets as a part of the test set.
* '''Aug 15, 2025''': Submit the main algorithm.
* '''Aug 20, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 24, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2024''': Online subjective evaluation.
* '''Sep 6, 2024''': Announce the final result.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission
# Algorithm submission
# Cherry-picked sample IDs submission
# Evaluation form submission
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
To be announced later.

=Baselines=

To be announced later.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-05-30T15:14:39Z

Zizzi wang:

=Description=

This year, the selected task is '''piano music continuation'''. Given a 4-measure piano prompt, the goal is to generate a 12-measure continuation that is musically coherent with the prompt, resulting in a complete 16-measure piece. All music is assumed to be in 4/4 meter, quantized to sixteenth-note resolution, and may belong to any existing piano style, including classical, pop, jazz, and others. Further details on the data structure are provided in the Data Format section.

=Data Format=
The input prompt consists of 4 bars of piano music, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing a property: <code>prompt</code>:

* <code>prompt</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>output</code>:

* <code>output</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both prompt and output, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 17 * 16 - 1 = 271. Notes that end later than the sixteenth measure (i.e., 17 * 16 = 272th time step) will be truncated to the end of the ninth measure.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"prompt": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
}
</pre>

This is an example of the generated continuation. Note that the generation starts from the fifth measure (time step 80).

<pre>
{
"acc": [
{"start": 80, "pitch": 41, "duration": 12},
{"start": 80, "pitch": 65, "duration": 5},
...
]
}
</pre>

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Coherency (5-point scale)
:* Creativity (5-point scale)
:* Structuredness (5-point scale)
:* Overall musicality (5-point scale)

==Important Dates (Tentative)==

* '''Aug 7, 2025''': Submit two lead sheets as a part of the test set.
* '''Aug 15, 2025''': Submit the main algorithm.
* '''Aug 20, 2025''': Return the generated samples. The cherry-picking phase begins.
* '''Aug 24, 2025''': Submit the cherry-picked sample ids.
* '''Aug 30 - Sep 5, 2024''': Online subjective evaluation.
* '''Sep 6, 2024''': Announce the final result.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission
# Algorithm submission
# Cherry-picked sample IDs submission
# Evaluation form submission
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
To be announced later.

=Baselines=

To be announced later.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

2025:Symbolic Music Generation

2025-05-30T14:08:15Z

Zizzi wang: Created page with "=Description= Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have..."

2024:Symbolic Music Generation

2024-09-15T14:21:11Z

Zizzi wang: /* Description */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, and chord progression. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

==Important Dates==

* '''Oct 8, 2024''': Submit two lead sheets as a part of the test set.
* '''Oct 15, 2024''': Submit the main algorithm.
* '''Oct 22, 2024''': Return the generated samples. The cherry-picking phase begins.
* '''Oct 25, 2024''': Submit the cherry-picked sample ids.
* '''Oct 31 - Nov 3, 2024''': Online subjective evaluation.
* '''Nov 5, 2024''': Announce the final result.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

'''Accepted submission form'''
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Baselines=

'''WholeSongGen''' (Wang et al., 2023)

This model generates accompaniment using the diffusion model.

'''Accomontage''' (Zhao et al., 2020)

This algorithm generates accompaniment using a combination of rule-based search and deep representation learning.

'''Anticipatory Music Transformer''' (Thickstun et al., 2024)

This model generates accompaniment using a Transformer-based model.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte, C. (2010). Towards automatic extraction of harmony information from music signals (Doctoral dissertation).* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.
* Wang, Z., Min, L., & Xia, G. Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models. In The Twelfth International Conference on Learning Representations.
* Jingwei Zhao, & Gus Xia (2021). AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021 (pp. 833–840).
* Thickstun, J., Hall, D., Donahue, C., & Liang, P. (2023). Anticipatory music transformer. arXiv preprint arXiv:2306.08620.
* Code and data format samples: [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main]

2024:Symbolic Music Generation

2024-09-15T10:18:30Z

Zizzi wang: /* References */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

==Important Dates==

* '''Oct 8, 2024''': Submit two lead sheets as a part of the test set.
* '''Oct 15, 2024''': Submit the main algorithm.
* '''Oct 22, 2024''': Return the generated samples. The cherry-picking phase begins.
* '''Oct 25, 2024''': Submit the cherry-picked sample ids.
* '''Oct 31 - Nov 3, 2024''': Online subjective evaluation.
* '''Nov 5, 2024''': Announce the final result.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

'''Accepted submission form'''
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Baselines=

'''WholeSongGen''' (Wang et al., 2023)

This model generates accompaniment using the diffusion model.

'''Accomontage''' (Zhao et al., 2020)

This algorithm generates accompaniment using a combination of rule-based search and deep representation learning.

'''Anticipatory Music Transformer''' (Thickstun et al., 2024)

This model generates accompaniment using a Transformer-based model.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte, C. (2010). Towards automatic extraction of harmony information from music signals (Doctoral dissertation).* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.
* Wang, Z., Min, L., & Xia, G. Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models. In The Twelfth International Conference on Learning Representations.
* Jingwei Zhao, & Gus Xia (2021). AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021 (pp. 833–840).
* Thickstun, J., Hall, D., Donahue, C., & Liang, P. (2023). Anticipatory music transformer. arXiv preprint arXiv:2306.08620.
* Code and data format samples: [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main]

2024:Symbolic Music Generation

2024-09-15T10:18:15Z

Zizzi wang: /* References */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

==Important Dates==

* '''Oct 8, 2024''': Submit two lead sheets as a part of the test set.
* '''Oct 15, 2024''': Submit the main algorithm.
* '''Oct 22, 2024''': Return the generated samples. The cherry-picking phase begins.
* '''Oct 25, 2024''': Submit the cherry-picked sample ids.
* '''Oct 31 - Nov 3, 2024''': Online subjective evaluation.
* '''Nov 5, 2024''': Announce the final result.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

'''Accepted submission form'''
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Baselines=

'''WholeSongGen''' (Wang et al., 2023)

This model generates accompaniment using the diffusion model.

'''Accomontage''' (Zhao et al., 2020)

This algorithm generates accompaniment using a combination of rule-based search and deep representation learning.

'''Anticipatory Music Transformer''' (Thickstun et al., 2024)

This model generates accompaniment using a Transformer-based model.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte, C. (2010). Towards automatic extraction of harmony information from music signals (Doctoral dissertation).* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.
* Wang, Z., Min, L., & Xia, G. Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models. In The Twelfth International Conference on Learning Representations.
* Jingwei Zhao, & Gus Xia (2021). AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021 (pp. 833–840).
* Thickstun, J., Hall, D., Donahue, C., & Liang, P. (2023). Anticipatory music transformer. arXiv preprint arXiv:2306.08620.
* Code and data format samples: [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main link]

2024:Symbolic Music Generation

2024-09-15T10:16:26Z

Zizzi wang:

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

==Important Dates==

* '''Oct 8, 2024''': Submit two lead sheets as a part of the test set.
* '''Oct 15, 2024''': Submit the main algorithm.
* '''Oct 22, 2024''': Return the generated samples. The cherry-picking phase begins.
* '''Oct 25, 2024''': Submit the cherry-picked sample ids.
* '''Oct 31 - Nov 3, 2024''': Online subjective evaluation.
* '''Nov 5, 2024''': Announce the final result.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

'''Accepted submission form'''
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Baselines=

'''WholeSongGen''' (Wang et al., 2023)

This model generates accompaniment using the diffusion model.

'''Accomontage''' (Zhao et al., 2020)

This algorithm generates accompaniment using a combination of rule-based search and deep representation learning.

'''Anticipatory Music Transformer''' (Thickstun et al., 2024)

This model generates accompaniment using a Transformer-based model.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte, C. (2010). Towards automatic extraction of harmony information from music signals (Doctoral dissertation).* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.
* Wang, Z., Min, L., & Xia, G. Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models. In The Twelfth International Conference on Learning Representations.
* Jingwei Zhao, & Gus Xia (2021). AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021 (pp. 833–840).
* Thickstun, J., Hall, D., Donahue, C., & Liang, P. (2023). Anticipatory music transformer. arXiv preprint arXiv:2306.08620.

2024:Symbolic Music Generation

2024-09-15T10:15:26Z

Zizzi wang: /* Evaluation */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation and Competition Format=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

'''Accepted submission form'''
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Important Dates=

* '''Oct 8, 2024''': Submit two lead sheets as a part of the test set.
* '''Oct 15, 2024''': Submit the main algorithm.
* '''Oct 22, 2024''': Return the generated samples. The cherry-picking phase begins.
* '''Oct 25, 2024''': Submit the cherry-picked sample ids.
* '''Oct 31 - Nov 3, 2024''': Online subjective evaluation.
* '''Nov 5, 2024''': Announce the final result.

=Baselines=

'''WholeSongGen''' (Wang et al., 2023)

This model generates accompaniment using the diffusion model.

'''Accomontage''' (Zhao et al., 2020)

This algorithm generates accompaniment using a combination of rule-based search and deep representation learning.

'''Anticipatory Music Transformer''' (Thickstun et al., 2024)

This model generates accompaniment using a Transformer-based model.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte, C. (2010). Towards automatic extraction of harmony information from music signals (Doctoral dissertation).* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.
* Wang, Z., Min, L., & Xia, G. Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models. In The Twelfth International Conference on Learning Representations.
* Jingwei Zhao, & Gus Xia (2021). AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021 (pp. 833–840).
* Thickstun, J., Hall, D., Donahue, C., & Liang, P. (2023). Anticipatory music transformer. arXiv preprint arXiv:2306.08620.

2024:Symbolic Music Generation

2024-09-15T10:14:26Z

Zizzi wang:

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

'''Accepted submission form'''
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Important Dates=

* '''Oct 8, 2024''': Submit two lead sheets as a part of the test set.
* '''Oct 15, 2024''': Submit the main algorithm.
* '''Oct 22, 2024''': Return the generated samples. The cherry-picking phase begins.
* '''Oct 25, 2024''': Submit the cherry-picked sample ids.
* '''Oct 31 - Nov 3, 2024''': Online subjective evaluation.
* '''Nov 5, 2024''': Announce the final result.

=Baselines=

'''WholeSongGen''' (Wang et al., 2023)

This model generates accompaniment using the diffusion model.

'''Accomontage''' (Zhao et al., 2020)

This algorithm generates accompaniment using a combination of rule-based search and deep representation learning.

'''Anticipatory Music Transformer''' (Thickstun et al., 2024)

This model generates accompaniment using a Transformer-based model.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte, C. (2010). Towards automatic extraction of harmony information from music signals (Doctoral dissertation).* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.
* Wang, Z., Min, L., & Xia, G. Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models. In The Twelfth International Conference on Learning Representations.
* Jingwei Zhao, & Gus Xia (2021). AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021 (pp. 833–840).
* Thickstun, J., Hall, D., Donahue, C., & Liang, P. (2023). Anticipatory music transformer. arXiv preprint arXiv:2306.08620.

2024:Symbolic Music Generation

2024-09-15T10:14:07Z

Zizzi wang:

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

'''Accepted submission form'''
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Important Dates=

* '''Oct 8, 2024''': Submit two lead sheets as a part of the test set.
* '''Oct 15, 2024''': Submit the main algorithm.
* '''Oct 22, 2024''': Return the generated samples. The cherry-picking phase begins.
* '''Oct 25, 2024''': Submit the cherry-picked sample ids.
* '''Oct 31 - Nov 3, 2024''': Online subjective evaluation.
* '''Nov 5, 2024''': Announce the final result.

=Baselines=

'''WholeSongGen''' (Wang et al., 2023)

This model generates accompaniment using the diffusion model.

'''Accomontage''' (Zhao et al., 2020)

This algorithm generates accompaniment using a combination of rule-based search and deep representation learning.

'''Anticipatory Music Transformer''' (Thickstun et al., 2024)

This model generates accompaniment using a Transformer-based model.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte, C. (2010). Towards automatic extraction of harmony information from music signals (Doctoral dissertation).* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.
* Wang, Z., Min, L., & Xia, G. Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models. In The Twelfth International Conference on Learning Representations.
* Jingwei Zhao, & Gus Xia (2021). AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021 (pp. 833–840).
* Thickstun, J., Hall, D., Donahue, C., & Liang, P. (2023). Anticipatory music transformer. arXiv preprint arXiv:2306.08620.

2024:Symbolic Music Generation

2024-09-15T10:13:29Z

Zizzi wang: /* References */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

'''Accepted submission form'''
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Important Dates=

* '''Oct 8, 2024''': Submit two lead sheets as a part of the test set.
* '''Oct 15, 2024''': Submit the main algorithm.
* '''Oct 22, 2024''': Return the generated samples. The cherry-picking phase begins.
* '''Oct 25, 2024''': Submit the cherry-picked sample ids.
* '''Oct 31 - Nov 3, 2024''': Online subjective evaluation.
* '''Nov 5, 2024''': Announce the final result.

=Baselines=

'''WholeSongGen''' (Wang et al., 2023)

This model generates accompaniment using the diffusion model.

'''Accomontage''' (Zhao et al., 2020)

This algorithm generates accompaniment using a combination of rule-based search and deep representation learning.

'''Anticipatory Music Transformer''' (Thickstun et al., 2024)

This model generates accompaniment using a Transformer-based model.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte, C. (2010). Towards automatic extraction of harmony information from music signals (Doctoral dissertation).* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.
* Wang, Z., Min, L., & Xia, G. Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models. In The Twelfth International Conference on Learning Representations.
* Jingwei Zhao, & Gus Xia (2021). AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021 (pp. 833–840).
* Thickstun, J., Hall, D., Donahue, C., & Liang, P. (2023). Anticipatory music transformer. arXiv preprint arXiv:2306.08620.

2024:Symbolic Music Generation

2024-09-15T10:09:46Z

Zizzi wang: /* Baselines */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

'''Accepted submission form'''
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Important Dates=

* '''Oct 8, 2024''': Submit two lead sheets as a part of the test set.
* '''Oct 15, 2024''': Submit the main algorithm.
* '''Oct 22, 2024''': Return the generated samples. The cherry-picking phase begins.
* '''Oct 25, 2024''': Submit the cherry-picked sample ids.
* '''Oct 31 - Nov 3, 2024''': Online subjective evaluation.
* '''Nov 5, 2024''': Announce the final result.

=Baselines=

'''WholeSongGen''' (Wang et al., 2023)

This model generates accompaniment using the diffusion model.

'''Accomontage''' (Zhao et al., 2020)

This algorithm generates accompaniment using a combination of rule-based search and deep representation learning.

'''Anticipatory Music Transformer''' (Thickstun et al., 2024)

This model generates accompaniment using a Transformer-based model.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.
* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.

2024:Symbolic Music Generation

2024-09-15T10:09:32Z

Zizzi wang: /* Baselines */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

'''Accepted submission form'''
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Important Dates=

* '''Oct 8, 2024''': Submit two lead sheets as a part of the test set.
* '''Oct 15, 2024''': Submit the main algorithm.
* '''Oct 22, 2024''': Return the generated samples. The cherry-picking phase begins.
* '''Oct 25, 2024''': Submit the cherry-picked sample ids.
* '''Oct 31 - Nov 3, 2024''': Online subjective evaluation.
* '''Nov 5, 2024''': Announce the final result.

=Baselines=

'''WholeSongGen''' (Wang et al., 2023)
This model generates accompaniment using the diffusion model.

'''Accomontage''' (Zhao et al., 2020)
This algorithm generates accompaniment using a combination of rule-based search and deep representation learning.

'''Anticipatory Music Transformer''' (Thickstun et al., 2024)
This model generates accompaniment using a Transformer-based model.

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.
* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.

2024:Symbolic Music Generation

2024-09-15T10:05:07Z

Zizzi wang: /* Important Dates */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

'''Accepted submission form'''
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Important Dates=

* '''Oct 8, 2024''': Submit two lead sheets as a part of the test set.
* '''Oct 15, 2024''': Submit the main algorithm.
* '''Oct 22, 2024''': Return the generated samples. The cherry-picking phase begins.
* '''Oct 25, 2024''': Submit the cherry-picked sample ids.
* '''Oct 31 - Nov 3, 2024''': Online subjective evaluation.
* '''Nov 5, 2024''': Announce the final result.

=Baselines=
TBD

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.
* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.

2024:Symbolic Music Generation

2024-09-15T10:01:09Z

Zizzi wang: /* Algorithm Submission */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

'''Accepted submission form'''
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

==Important Dates==
The submission process is '''tentative'''.
* Oct 8, 2024: submission of two lead sheets as a part of the test set. This is also a confirmation of participation.
* Oct. 15, 2024: submission of the algorithm in docker.
* Oct. 22, 2024: return of the generated samples. Start of the cherry-picking phase.
* Oct. 25, 2024: submission of the cherry-picked sample ids.
* Oct. 31 - Nov. 3, 2024: subjective test.
* Nov. 5, 2024: announcement of the final result.

=Baselines=
TBD

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.
* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.

2024:Symbolic Music Generation

2024-09-15T10:00:48Z

Zizzi wang: /* Packaging Submissions */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

'''Packaging Submissions'''
* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

Accepted submission form:
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

==Important Dates==
The submission process is '''tentative'''.
* Oct 8, 2024: submission of two lead sheets as a part of the test set. This is also a confirmation of participation.
* Oct. 15, 2024: submission of the algorithm in docker.
* Oct. 22, 2024: return of the generated samples. Start of the cherry-picking phase.
* Oct. 25, 2024: submission of the cherry-picked sample ids.
* Oct. 31 - Nov. 3, 2024: subjective test.
* Nov. 5, 2024: announcement of the final result.

=Baselines=
TBD

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.
* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.

2024:Symbolic Music Generation

2024-09-15T10:00:15Z

Zizzi wang: /* I/O Format */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==Algorithm Submission==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

==Packaging Submissions==

* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

Accepted submission form:
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

==Important Dates==
The submission process is '''tentative'''.
* Oct 8, 2024: submission of two lead sheets as a part of the test set. This is also a confirmation of participation.
* Oct. 15, 2024: submission of the algorithm in docker.
* Oct. 22, 2024: return of the generated samples. Start of the cherry-picking phase.
* Oct. 25, 2024: submission of the cherry-picked sample ids.
* Oct. 31 - Nov. 3, 2024: subjective test.
* Nov. 5, 2024: announcement of the final result.

=Baselines=
TBD

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.
* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.

2024:Symbolic Music Generation

2024-09-15T09:59:50Z

Zizzi wang:

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==I/O Format==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

==Packaging Submissions==

* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

Accepted submission form:
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

==Important Dates==
The submission process is '''tentative'''.
* Oct 8, 2024: submission of two lead sheets as a part of the test set. This is also a confirmation of participation.
* Oct. 15, 2024: submission of the algorithm in docker.
* Oct. 22, 2024: return of the generated samples. Start of the cherry-picking phase.
* Oct. 25, 2024: submission of the cherry-picked sample ids.
* Oct. 31 - Nov. 3, 2024: subjective test.
* Nov. 5, 2024: announcement of the final result.

=Baselines=
TBD

=Contacts=
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

=References=
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.
* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.

2024:Symbolic Music Generation

2024-09-15T09:59:12Z

Zizzi wang: /* Evaluation */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==I/O Format==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

==Packaging Submissions==

* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

Accepted submission form:
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

==Important Dates==
The submission process is '''tentative'''.
* Oct 8, 2024: submission of two lead sheets as a part of the test set. This is also a confirmation of participation.
* Oct. 15, 2024: submission of the algorithm in docker.
* Oct. 22, 2024: return of the generated samples. Start of the cherry-picking phase.
* Oct. 25, 2024: submission of the cherry-picked sample ids.
* Oct. 31 - Nov. 3, 2024: subjective test.
* Nov. 5, 2024: announcement of the final result.

=Baselines=
TBD

==Contacts==
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

==References==
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.
* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.

2024:Symbolic Music Generation

2024-09-15T09:58:49Z

Zizzi wang: /* Submission */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=

As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
# Test set submission (Oct 8, 2024)
# Algorithm submission (Oct 15, 2024)
# Cherry-picked sample IDs submission (Oct 25, 2024)
# Evaluation form submission (Nov 3, 2024)
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''

==I/O Format==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

==Packaging Submissions==

* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

Accepted submission form:
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

==Important Dates==
The submission process is '''tentative'''.
* Oct 8, 2024: submission of two lead sheets as a part of the test set. This is also a confirmation of participation.
* Oct. 15, 2024: submission of the algorithm in docker.
* Oct. 22, 2024: return of the generated samples. Start of the cherry-picking phase.
* Oct. 25, 2024: submission of the cherry-picked sample ids.
* Oct. 31 - Nov. 3, 2024: subjective test.
* Nov. 5, 2024: announcement of the final result.

=Baselines=
TBD

==Contacts==
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

==References==
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.
* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.

2024:Symbolic Music Generation

2024-09-15T09:50:36Z

Zizzi wang: /* Contacts */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=
==Important Dates==
The submission process is '''tentative'''.
* Oct 8, 2024: submission of two lead sheets as a part of the test set. This is also a confirmation of participation.
* Oct. 15, 2024: submission of the algorithm in docker.
* Oct. 22, 2024: return of the generated samples. Start of the cherry-picking phase.
* Oct. 25, 2024: submission of the cherry-picked sample ids.
* Oct. 31 - Nov. 3, 2024: subjective test.
* Nov. 5, 2024: announcement of the final result.

==I/O Format==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

==Packaging Submissions==

* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

Accepted submission form:
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Baselines=
TBD

==Contacts==
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao: jzhao<at>u.nus.edu

==References==
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.
* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.

2024:Symbolic Music Generation

2024-09-15T09:49:34Z

Zizzi wang:

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=
==Important Dates==
The submission process is '''tentative'''.
* Oct 8, 2024: submission of two lead sheets as a part of the test set. This is also a confirmation of participation.
* Oct. 15, 2024: submission of the algorithm in docker.
* Oct. 22, 2024: return of the generated samples. Start of the cherry-picking phase.
* Oct. 25, 2024: submission of the cherry-picked sample ids.
* Oct. 31 - Nov. 3, 2024: subjective test.
* Nov. 5, 2024: announcement of the final result.

==I/O Format==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

==Packaging Submissions==

* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

Accepted submission form:
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Baselines=
TBD

==Contacts==
If you any questions or suggestions about the task, please contact:
* Ziyu Wang: ziyu.wang<at>nyu.edu
* Jingwei Zhao:

==References==
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.
* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.

2024:Symbolic Music Generation

2024-09-15T09:48:00Z

Zizzi wang: /* Objective Measurements */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
* We welcome proposals of the objective measurements.

=Submission=
==Important Dates==
The submission process is '''tentative'''.
* Oct 8, 2024: submission of two lead sheets as a part of the test set. This is also a confirmation of participation.
* Oct. 15, 2024: submission of the algorithm in docker.
* Oct. 22, 2024: return of the generated samples. Start of the cherry-picking phase.
* Oct. 25, 2024: submission of the cherry-picked sample ids.
* Oct. 31 - Nov. 3, 2024: subjective test.
* Nov. 5, 2024: announcement of the final result.

==I/O Format==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

==Packaging Submissions==

* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

Accepted submission form:
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Baselines=
TBD

==References==
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.
* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.

2024:Symbolic Music Generation

2024-09-15T09:47:11Z

Zizzi wang: /* References */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023)scwill currently use only the likelihood.

=Submission=
==Important Dates==
The submission process is '''tentative'''.
* Oct 8, 2024: submission of two lead sheets as a part of the test set. This is also a confirmation of participation.
* Oct. 15, 2024: submission of the algorithm in docker.
* Oct. 22, 2024: return of the generated samples. Start of the cherry-picking phase.
* Oct. 25, 2024: submission of the cherry-picked sample ids.
* Oct. 31 - Nov. 3, 2024: subjective test.
* Nov. 5, 2024: announcement of the final result.

==I/O Format==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

==Packaging Submissions==

* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

Accepted submission form:
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Baselines=
TBD

==References==
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.
* Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., & Bian, J. (2023). Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.

2024:Symbolic Music Generation

2024-09-15T09:46:58Z

Zizzi wang: /* Subjective Evaluation Format */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked to rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)

==Objective Measurements==
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023)scwill currently use only the likelihood.

=Submission=
==Important Dates==
The submission process is '''tentative'''.
* Oct 8, 2024: submission of two lead sheets as a part of the test set. This is also a confirmation of participation.
* Oct. 15, 2024: submission of the algorithm in docker.
* Oct. 22, 2024: return of the generated samples. Start of the cherry-picking phase.
* Oct. 25, 2024: submission of the cherry-picked sample ids.
* Oct. 31 - Nov. 3, 2024: subjective test.
* Nov. 5, 2024: announcement of the final result.

==I/O Format==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

==Packaging Submissions==

* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

Accepted submission form:
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Baselines=
TBD

==References==
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.

2024:Symbolic Music Generation

2024-09-15T09:43:03Z

Zizzi wang: /* Evaluation */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.
* We hope to compute some objective measurements as well, but these will only be reported as a reference.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate 16 arrangements for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference. We will currently use only the likelihood.

=Submission=
==Important Dates==
The submission process is '''tentative'''.
* Oct 8, 2024: submission of two lead sheets as a part of the test set. This is also a confirmation of participation.
* Oct. 15, 2024: submission of the algorithm in docker.
* Oct. 22, 2024: return of the generated samples. Start of the cherry-picking phase.
* Oct. 25, 2024: submission of the cherry-picked sample ids.
* Oct. 31 - Nov. 3, 2024: subjective test.
* Nov. 5, 2024: announcement of the final result.

==I/O Format==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

==Packaging Submissions==

* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

Accepted submission form:
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Baselines=
TBD

==References==
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.

2024:Symbolic Music Generation

2024-09-15T09:35:58Z

Zizzi wang: /* Data Format */

=Description=
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.

This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, chord progression, and optional phrase labels. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).

=Data Format=
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:

* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.

The output generation should also follow the JSON format containing one property <code>acc</code>:

* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.

'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''

# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
# Melody notes are not allowed to overlap with one another.
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.
# The accompaniment of the pick-up measure should be blank.

'''Detailed explanation of the <code>pitch</code> attribute.'''

# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.

'''Detailed explanation of the chord <code>symbol</code> attribute.'''

# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.

=Data Example=
Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of ''Hey Jude'' by The Beatles.

<pre>
{
"melody": [
{"start": 12, "pitch": 72, "duration": 4},
{"start": 16, "pitch": 69, "duration": 8},
...
],
"chords": [
{"start": 0, "symbol": "N", "duration": 16},
{"start": 16, "symbol": "F", "duration": 16},
...
]
}
</pre>

This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).

<pre>
{
"acc": [
{"start": 16, "pitch": 41, "duration": 12},
{"start": 16, "pitch": 65, "duration": 5},
...
]
}
</pre>
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.

=Evaluation=
We will evaluate the submitted algorithms through an online subjective double-blind test. '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.

==Subjective Evaluation Format==
* After each team submits the algorithm, the organizer team will use the algorithm to generate 10 arrangements for each test sample. The generated results will be returned to each team for cherry-picking.
* Only a subset of the test set will be used for subjective evaluation.
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
* The subject will be asked rate each arrangement based on the following criteria:
:* Harmony correctness (5-point scale)
:* Creativity (5-point scale)
:* Naturalness (5-point scale)
:* Overall musicality (5-point scale)
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference. We will currently use only the likelihood.

=Submission=
==Important Dates==
The submission process is '''tentative'''.
* Oct 8, 2024: submission of two lead sheets as a part of the test set. This is also a confirmation of participation.
* Oct. 15, 2024: submission of the algorithm in docker.
* Oct. 22, 2024: return of the generated samples. Start of the cherry-picking phase.
* Oct. 25, 2024: submission of the cherry-picked sample ids.
* Oct. 31 - Nov. 3, 2024: subjective test.
* Nov. 5, 2024: announcement of the final result.

==I/O Format==
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:

'''Usage'''

<pre>
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
</pre>

* Input File: Path to the input .json file.
* Output Folder: Path to the folder where the generated output files will be saved.
* n_sample: Number of samples to generate.

'''Output'''
* The script should generate n_sample output files in the specified output folder.
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.

Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.

==Packaging Submissions==

* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

Accepted submission form:
* Link to public or private Github repository
* Link to public or private docker hub
* Shared google drive links
* If the repository is private, an access token is also required

=Baselines=
TBD

==References==
* Harte. Towards Automatic Extraction of Harmony Information from Music Signals. PhD thesis, Queen Mary University of London, August 2010.