2026:Music Evaluation via CMI-RewardBench Results - Revision history

Junyan: /* Results */ PDF links for submissions (abstracts archived on futuremirex.com)

2026-10-14T15:23:38Z

‎Results: PDF links for submissions (abstracts archived on futuremirex.com)

Junyan: Junyan moved page 2026:YOUR TASK Results to 2026:Music Evaluation via CMI-RewardBench Results: Rename placeholder title to the task name

2026-10-14T15:19:57Z

Junyan moved page 2026:YOUR TASK Results to 2026:Music Evaluation via CMI-RewardBench Results: Rename placeholder title to the task name

Yinghao Ma: /* Results */

2026-10-14T14:32:54Z

‎Results

Yinghao Ma at 14:30, 14 October 2026

2026-10-14T14:30:40Z

Yinghao Ma: Created page with " =Music Evaluation via CMI-RewardBench Results= ==Task Description== The MIREX 2026 Music Evaluation via CMI-RewardBench task evaluates automatic music reward models and eva..."

2026-10-14T14:21:40Z

Created page with " =Music Evaluation via CMI-RewardBench Results= ==Task Description== The MIREX 2026 Music Evaluation via CMI-RewardBench task evaluates automatic music reward models and eva..."

New page

=Music Evaluation via CMI-RewardBench Results=

==Task Description==

The MIREX 2026 Music Evaluation via CMI-RewardBench task evaluates automatic music reward models and evaluation metrics according to their agreement with human preferences.

Given a text prompt and two generated music candidates (Audio A and Audio B), each system predicts which candidate is preferred by human listeners. Systems are evaluated using pairwise preference accuracy.

The evaluation reports results for the 249-sample core dataset and vocal and instrumental subsets of the 618-sample extended dataset.

For details, see [[2026:Music_Evaluation_via_CMI-RewardBench|MIREX 2026 Music Evaluation via CMI-RewardBench]].

==Results==

Systems are ranked by overall accuracy on the 249-sample core dataset in descending order. All scores are percentages. Bold indicates the highest accuracy in each column.

{| class="wikitable sortable" style="vertical-align:bottom;"
|- style="font-weight:bold;"
! Rank
! System
! style="text-align:right;" | Core Overall (249)
! style="text-align:right;" | Vocal Extension (300+)
! style="text-align:right;" | Instrumental Extension (300+)
|-
| 1
| CloudEar-overall (User ID: 32)
| style="text-align:right;" | '''73.9'''
| style="text-align:right;" | 60.9
| style="text-align:right;" | 72.6
|-
| 2
| SongEval-overall [https://arxiv.org/abs/2505.10793 [2]]
| style="text-align:right;" | 72.7
| style="text-align:right;" | 62.0
| style="text-align:right;" | 72.0
|-
| 3
| first_submission (User ID: 59)
| style="text-align:right;" | 72.3
| style="text-align:right;" | '''63.3'''
| style="text-align:right;" | '''75.4'''
|-
| 4
| CloudEar-musicality (User ID: 32)
| style="text-align:right;" | 69.9
| style="text-align:right;" | 60.6
| style="text-align:right;" | 70.7
|-
| 5
| CloudEar-alignment (User ID: 32)
| style="text-align:right;" | 69.5
| style="text-align:right;" | 61.6
| style="text-align:right;" | 69.5
|-
| 6
| Baseline: CMI-RM [https://proceedings.mlr.press/v306/ma26am.html [1]]
| style="text-align:right;" | 65.9
| style="text-align:right;" | 57.2
| style="text-align:right;" | 69.2
|-
| 7
| aimusic (User ID: 68)
| style="text-align:right;" | 65.5
| style="text-align:right;" | 55.9
| style="text-align:right;" | 70.7
|-
| 8
| TuneJury [https://arxiv.org/abs/2606.17006 [3]]
| style="text-align:right;" | 64.3
| style="text-align:right;" | 53.9
| style="text-align:right;" | 66.0
|-
| 9
| Random guess (always B)
| style="text-align:right;" | 59.0
| style="text-align:right;" | 55.2
| style="text-align:right;" | 56.1
|-
| 10
| MuseCritic [https://arxiv.org/abs/2608.11755 [4]]
| style="text-align:right;" | 55.4
| style="text-align:right;" | 59.9
| style="text-align:right;" | 46.7
|-
| 11
| SYSTASYS Resource-Lite V2 - Blind Freeze 2026 (User ID: 72)
| style="text-align:right;" | 39.8
| style="text-align:right;" | 43.4
| style="text-align:right;" | 40.2
|-
| 11
| Flip of "SYSTASYS Resource-Lite V2 - Blind Freeze 2026"
| style="text-align:right;" | 60.2
| style="text-align:right;" | 56.6
| style="text-align:right;" | 59.8
|}

==References==

# Ma, Y., Xia, H., Gao, H., et al. (2026). ''CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction.'' Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), PMLR Vol. 306, pp. 85099–85130. [https://proceedings.mlr.press/v306/ma26am.html Paper]
# Yao, J., Ma, G., Xue, H., et al. (2025). ''SongEval: A Benchmark Dataset for Song Aesthetics Evaluation.'' arXiv:2505.10793. [https://arxiv.org/abs/2505.10793 Paper]
# Kim, Y., Lee, J., Xia, H., et al. (2026). ''TuneJury: An Open Metric for Improving Music Generation Preference Alignment.'' arXiv:2606.17006. [https://arxiv.org/abs/2606.17006 Paper]
# Zhuang, J., Jiang, C., Wang, H., et al. (2026). ''MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques.'' arXiv:2608.11755. [https://arxiv.org/abs/2608.11755 Paper]

@@ Line 25: / Line 25: @@
 |-
 | 1
-| CloudEar-overall (User ID: 32)
+| CloudEar-overall (User ID: 32) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/CloudEar.pdf PDF])
 | style="text-align:right;" | '''73.9'''
 | style="text-align:right;" | 60.9
@@ Line 37: / Line 37: @@
 |-
 | 3
-| first_submission (User ID: 59)
+| first_submission (User ID: 59) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/first_submission.pdf PDF])
 | style="text-align:right;" | 72.3
 | style="text-align:right;" | '''63.3'''
@@ Line 43: / Line 43: @@
 |-
 | 4
-| CloudEar (Musicality) (User ID: 32)
+| CloudEar (Musicality) (User ID: 32) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/CloudEar.pdf PDF])
 | style="text-align:right;" | 69.9
 | style="text-align:right;" | 60.6
@@ Line 49: / Line 49: @@
 |-
 | 5
-| CloudEar (Alignment) (User ID: 32)
+| CloudEar (Alignment) (User ID: 32) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/CloudEar.pdf PDF])
 | style="text-align:right;" | 69.5
 | style="text-align:right;" | 61.6
@@ Line 73: / Line 73: @@
 |-
 | 9
-| aimusic (User ID: 68)
+| aimusic (User ID: 68) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/aimusic.pdf PDF])
 | style="text-align:right;" | 65.5
 | style="text-align:right;" | 55.9
@@ Line 97: / Line 97: @@
 |-
 | 13
-| SYSTASYS Resource-Lite V2 - Blind Freeze 2026 (User ID: 72)
+| SYSTASYS Resource-Lite V2 - Blind Freeze 2026 (User ID: 72) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/SYSTASYS.pdf PDF])
 | style="text-align:right;" | 39.8
 | style="text-align:right;" | 43.4

← Older revision	Revision as of 15:19, 14 October 2026
(No difference)

@@ Line 21: / Line 21: @@
 ! System
 ! style="text-align:right;" | Core Overall (249)
-! style="text-align:right;" | Vocal Extension (618)
+! style="text-align:right;" | Vocal Extension (300+)
-! style="text-align:right;" | Instrumental Extension (618)
+! style="text-align:right;" | Instrumental Extension (300+)
 |-
 | 1

@@ Line 21: / Line 21: @@
 ! System
 ! style="text-align:right;" | Core Overall (249)
-! style="text-align:right;" | Vocal Extension (300+)
+! style="text-align:right;" | Vocal Extension (618)
-! style="text-align:right;" | Instrumental Extension (300+)
+! style="text-align:right;" | Instrumental Extension (618)
 |-
 | 1
@@ Line 31: / Line 31: @@
 |-
 | 2
-| SongEval-overall [https://arxiv.org/abs/2505.10793 &#91;2&#93;]
+| SongEval (Average) [https://arxiv.org/abs/2505.10793 &#91;2&#93;]
 | style="text-align:right;" | 72.7
 | style="text-align:right;" | 62.0
@@ Line 43: / Line 43: @@
 |-
 | 4
-| CloudEar-musicality (User ID: 32)
+| CloudEar (Musicality) (User ID: 32)
 | style="text-align:right;" | 69.9
 | style="text-align:right;" | 60.6
@@ Line 49: / Line 49: @@
 |-
 | 5
-| CloudEar-alignment (User ID: 32)
+| CloudEar (Alignment) (User ID: 32)
 | style="text-align:right;" | 69.5
 | style="text-align:right;" | 61.6
@@ Line 55: / Line 55: @@
 |-
 | 6
 | Baseline: CMI-RM [https://proceedings.mlr.press/v306/ma26am.html &#91;1&#93;]
 | style="text-align:right;" | 65.9
@@ Line 60: / Line 72: @@
 | style="text-align:right;" | 69.2
 |-
-| 7
+| 9
 | aimusic (User ID: 68)
 | style="text-align:right;" | 65.5
@@ Line 66: / Line 78: @@
 | style="text-align:right;" | 70.7
 |-
-| 8
+| 10
 | TuneJury [https://arxiv.org/abs/2606.17006 &#91;3&#93;]
 | style="text-align:right;" | 64.3
@@ Line 72: / Line 84: @@
 | style="text-align:right;" | 66.0
 |-
-| 9
+| 11
 | Random guess (always B)
 | style="text-align:right;" | 59.0
@@ Line 78: / Line 90: @@
 | style="text-align:right;" | 56.1
 |-
-| 10
+| 12
 | MuseCritic [https://arxiv.org/abs/2608.11755 &#91;4&#93;]
 | style="text-align:right;" | 55.4
@@ Line 84: / Line 96: @@
 | style="text-align:right;" | 46.7
 |-
-| 11
+| 13
 | SYSTASYS Resource-Lite V2 - Blind Freeze 2026 (User ID: 72)
 | style="text-align:right;" | 39.8
@@ Line 90: / Line 102: @@
 | style="text-align:right;" | 40.2
 |-
-| 11
+| 13
 | Flip of "SYSTASYS Resource-Lite V2 - Blind Freeze 2026"
 | style="text-align:right;" | 60.2
@@ Line 103: / Line 115: @@
 # Kim, Y., Lee, J., Xia, H., et al. (2026). ''TuneJury: An Open Metric for Improving Music Generation Preference Alignment.'' arXiv:2606.17006. [https://arxiv.org/abs/2606.17006 Paper]
 # Zhuang, J., Jiang, C., Wang, H., et al. (2026). ''MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques.'' arXiv:2608.11755. [https://arxiv.org/abs/2608.11755 Paper]