2026:Music Evaluation via CMI-RewardBench - Revision history

Yinghao Ma: /* A Simple Example */

2026-10-07T09:39:07Z

‎A Simple Example

2026-10-05T22:33:54Z

‎Timeline and Submission Instructions

2026-10-05T22:33:31Z

‎Timeline and Submission Instructions

2026-10-05T22:32:50Z

‎Test Dataset

2026-10-05T22:32:03Z

‎Test Dataset

2026-10-05T22:31:28Z

‎A Private Test Set

2026-09-22T17:45:56Z

‎Timeline and Submission Instructions

2026-06-28T23:00:52Z

‎Overview

2026-06-28T22:59:27Z

‎Metrics

2026-06-25T14:58:47Z

@@ Line 34: / Line 34: @@
 <pre>
+{
-   "sample_id": "toy_example_001",
+   "battle_uuid": "toy_example_001",
    "preferred_candidate": "A"
+}
 </pre>
 == Submission and Environment Requirements ==

@@ Line 114: / Line 114: @@
 The evaluation follows a rapid-turnaround test phase window in October:
-* '''[updated] October 6th:''' The evaluation team will release the official music audio files and the corresponding evaluation JSONL file. See private test set subsection above.
+* '''[updated] October 6th:''' The evaluation team will release the official music audio files and the corresponding evaluation JSONL file. '''See private test set subsection above.'''
 * '''[updated] October 7th (AOE):''' Deadline to generate and upload your predictions. You must submit the completed JSONL file containing the preference predictions for each of the given audio pairs.

@@ Line 114: / Line 114: @@
 The evaluation follows a rapid-turnaround test phase window in October:
-* '''[updated] October 6th:''' The evaluation team will release the official music audio files and the corresponding evaluation JSONL file.
+* '''[updated] October 6th:''' The evaluation team will release the official music audio files and the corresponding evaluation JSONL file. See private test set subsection above.
 * '''[updated] October 7th (AOE):''' Deadline to generate and upload your predictions. You must submit the completed JSONL file containing the preference predictions for each of the given audio pairs.

@@ Line 88: / Line 88: @@
 crowdsourced platform over the months leading up to the evaluation. Because this data is kept completely private and released continuously, it ensures zero data leakage and prevents benchmark saturation.
-* URL: https://drive.google.com/file/d/14X46JD1W_mN-UOtVSupvrVPsaR5jep1a/view?usp=sharing
+* URL (available on 6th and 7th Oct): https://drive.google.com/file/d/14X46JD1W_mN-UOtVSupvrVPsaR5jep1a/view?usp=sharing
 = Baselines =

@@ Line 88: / Line 88: @@
 crowdsourced platform over the months leading up to the evaluation. Because this data is kept completely private and released continuously, it ensures zero data leakage and prevents benchmark saturation.
-https://drive.google.com/file/d/14X46JD1W_mN-UOtVSupvrVPsaR5jep1a/view?usp=sharing
+* URL: https://drive.google.com/file/d/14X46JD1W_mN-UOtVSupvrVPsaR5jep1a/view?usp=sharing
 = Baselines =

@@ Line 112: / Line 112: @@
 The evaluation follows a rapid-turnaround test phase window in October:
-* '''October 1st:''' The evaluation team will release the official music audio files and the corresponding evaluation JSONL file.
+* '''[updated] October 6th:''' The evaluation team will release the official music audio files and the corresponding evaluation JSONL file.
-* '''October 2nd (AOE):''' Deadline to generate and upload your predictions. You must submit the completed JSONL file containing the preference predictions for each of the given audio pairs.
+* '''[updated] October 7th (AOE):''' Deadline to generate and upload your predictions. You must submit the completed JSONL file containing the preference predictions for each of the given audio pairs.
 All final prediction files must be uploaded through the official portal:

@@ Line 10: / Line 10: @@
 * '''Input:''' A JSON or JSONL file containing data items structured similarly to the [[https://huggingface.co/datasets/music-arena/music-arena-dataset|Music Arena Dataset]]. Each entry includes a unique identifier, a text prompt description, and paths or URLs for two audio samples (Audio A and Audio B).
-* '''Output:''' A prediction file containing a preference score indicating the relative quality and alignment, along with a declaration of the preferred candidate.
+* '''Output:''' A prediction file containing a preference score indicating the relative quality and alignment [optional], along with a declaration of the preferred candidate (A or B).
 == A Simple Example ==

@@ Line 97: / Line 97: @@
 Submissions will be evaluated quantitatively using standard alignment and efficiency metrics defined in the CMI-RewardBench framework ([https://arxiv.org/html/2603.00610v2](https://www.google.com/search?q=https%3A%2F%2Farxiv.org%2Fhtml%2F2603.00610v2)). The main evaluation criteria include:
-* Accuracy: Measurement of the model's preference output correlation with crowdsourced human Elo ratings and pairwise preference consensus labels.
+* Accuracy: Measurement of the model's preference output correlation with crowdsourced human ratings (pairwise preference consensus labels).
 = Rules =

@@ Line 35: / Line 35: @@
+{
    "sample_id": "toy_example_001",
    "preferred_candidate": "A"
+}
@@ Line 112: / Line 111: @@
-= Submission =
+== Timeline and Submission Instructions ==
-* Participants must submit either a containerised system (e.g., Docker container) containing their complete evaluation script, runtime environments, and model weights; or an API entry which must expose a standardised API, either way shall be capable of accepting an audio pair alongside its generation prompt, returning a definitive scalar preference score.
+The evaluation follows a rapid-turnaround test phase window in October:
-* Technical Documentation: Submissions must include a brief technical report summarising model architecture, total training data volume, model parameter size, and computational overhead.
+All final prediction files must be uploaded through the official portal:
+* '''Submission Site:''' [https://futuremirex.com/submission/ https://futuremirex.com/submission/]
 = Paper =