2026:Music Evaluation via CMI-RewardBench Results
Contents
Music Evaluation via CMI-RewardBench Results
Task Description
The MIREX 2026 Music Evaluation via CMI-RewardBench task evaluates automatic music reward models and evaluation metrics according to their agreement with human preferences.
Given a text prompt and two generated music candidates (Audio A and Audio B), each system predicts which candidate is preferred by human listeners. Systems are evaluated using pairwise preference accuracy.
The evaluation reports results for the 249-sample core dataset and vocal and instrumental subsets of the 618-sample extended dataset.
For details, see MIREX 2026 Music Evaluation via CMI-RewardBench.
Results
Systems are ranked by overall accuracy on the 249-sample core dataset in descending order. All scores are percentages. Bold indicates the highest accuracy in each column.
| Rank | System | Core Overall (249) | Vocal Extension (300+) | Instrumental Extension (300+) |
|---|---|---|---|---|
| 1 | CloudEar-overall (User ID: 32) | 73.9 | 60.9 | 72.6 |
| 2 | SongEval (Average) [2] | 72.7 | 62.0 | 72.0 |
| 3 | first_submission (User ID: 59) | 72.3 | 63.3 | 75.4 |
| 4 | CloudEar (Musicality) (User ID: 32) | 69.9 | 60.6 | 70.7 |
| 5 | CloudEar (Alignment) (User ID: 32) | 69.5 | 61.6 | 69.5 |
| 6 | SongBench (Musicality) [5] | 69.1 | 62.0 | 68.8 |
| 7 | SongBench (Average) [5] | 69.1 | 59.6 | 72.6 |
| 8 | Baseline: CMI-RM [1] | 65.9 | 57.2 | 69.2 |
| 9 | aimusic (User ID: 68) | 65.5 | 55.9 | 70.7 |
| 10 | TuneJury [3] | 64.3 | 53.9 | 66.0 |
| 11 | Random guess (always B) | 59.0 | 55.2 | 56.1 |
| 12 | MuseCritic [4] | 55.4 | 59.9 | 46.7 |
| 13 | SYSTASYS Resource-Lite V2 - Blind Freeze 2026 (User ID: 72) | 39.8 | 43.4 | 40.2 |
| 13 | Flip of "SYSTASYS Resource-Lite V2 - Blind Freeze 2026" | 60.2 | 56.6 | 59.8 |
References
- Ma, Y., Xia, H., Gao, H., et al. (2026). CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), PMLR Vol. 306, pp. 85099–85130. Paper
- Yao, J., Ma, G., Xue, H., et al. (2025). SongEval: A Benchmark Dataset for Song Aesthetics Evaluation. arXiv:2505.10793. Paper
- Kim, Y., Lee, J., Xia, H., et al. (2026). TuneJury: An Open Metric for Improving Music Generation Preference Alignment. arXiv:2606.17006. Paper
- Zhuang, J., Jiang, C., Wang, H., et al. (2026). MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques. arXiv:2608.11755. Paper
- Wu, D., Lei, S., Tan, W., Li, G., Wang, Y., Zhang, H., Zuo, L., and Wu, Z. (2026). SongBench: A Fine-Grained Multi-Aspect Benchmark for Song Quality Assessment. arXiv:2604.25937. Paper Code
