Difference between revisions of "2026:Music Evaluation via CMI-RewardBench Results"
From MIREX Wiki
Yinghao Ma (talk | contribs) |
(→Results: PDF links for submissions (abstracts archived on futuremirex.com)) |
||
| (2 intermediate revisions by 2 users not shown) | |||
| Line 21: | Line 21: | ||
! System | ! System | ||
! style="text-align:right;" | Core Overall (249) | ! style="text-align:right;" | Core Overall (249) | ||
| − | ! style="text-align:right;" | Vocal Extension ( | + | ! style="text-align:right;" | Vocal Extension (300+) |
| − | ! style="text-align:right;" | Instrumental Extension ( | + | ! style="text-align:right;" | Instrumental Extension (300+) |
|- | |- | ||
| 1 | | 1 | ||
| − | | CloudEar-overall (User ID: 32) | + | | CloudEar-overall (User ID: 32) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/CloudEar.pdf PDF]) |
| style="text-align:right;" | '''73.9''' | | style="text-align:right;" | '''73.9''' | ||
| style="text-align:right;" | 60.9 | | style="text-align:right;" | 60.9 | ||
| Line 37: | Line 37: | ||
|- | |- | ||
| 3 | | 3 | ||
| − | | first_submission (User ID: 59) | + | | first_submission (User ID: 59) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/first_submission.pdf PDF]) |
| style="text-align:right;" | 72.3 | | style="text-align:right;" | 72.3 | ||
| style="text-align:right;" | '''63.3''' | | style="text-align:right;" | '''63.3''' | ||
| Line 43: | Line 43: | ||
|- | |- | ||
| 4 | | 4 | ||
| − | | CloudEar (Musicality) (User ID: 32) | + | | CloudEar (Musicality) (User ID: 32) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/CloudEar.pdf PDF]) |
| style="text-align:right;" | 69.9 | | style="text-align:right;" | 69.9 | ||
| style="text-align:right;" | 60.6 | | style="text-align:right;" | 60.6 | ||
| Line 49: | Line 49: | ||
|- | |- | ||
| 5 | | 5 | ||
| − | | CloudEar (Alignment) (User ID: 32) | + | | CloudEar (Alignment) (User ID: 32) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/CloudEar.pdf PDF]) |
| style="text-align:right;" | 69.5 | | style="text-align:right;" | 69.5 | ||
| style="text-align:right;" | 61.6 | | style="text-align:right;" | 61.6 | ||
| Line 73: | Line 73: | ||
|- | |- | ||
| 9 | | 9 | ||
| − | | aimusic (User ID: 68) | + | | aimusic (User ID: 68) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/aimusic.pdf PDF]) |
| style="text-align:right;" | 65.5 | | style="text-align:right;" | 65.5 | ||
| style="text-align:right;" | 55.9 | | style="text-align:right;" | 55.9 | ||
| Line 97: | Line 97: | ||
|- | |- | ||
| 13 | | 13 | ||
| − | | SYSTASYS Resource-Lite V2 - Blind Freeze 2026 (User ID: 72) | + | | SYSTASYS Resource-Lite V2 - Blind Freeze 2026 (User ID: 72) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/SYSTASYS.pdf PDF]) |
| style="text-align:right;" | 39.8 | | style="text-align:right;" | 39.8 | ||
| style="text-align:right;" | 43.4 | | style="text-align:right;" | 43.4 | ||
Latest revision as of 10:23, 14 October 2026
Contents
Music Evaluation via CMI-RewardBench Results
Task Description
The MIREX 2026 Music Evaluation via CMI-RewardBench task evaluates automatic music reward models and evaluation metrics according to their agreement with human preferences.
Given a text prompt and two generated music candidates (Audio A and Audio B), each system predicts which candidate is preferred by human listeners. Systems are evaluated using pairwise preference accuracy.
The evaluation reports results for the 249-sample core dataset and vocal and instrumental subsets of the 618-sample extended dataset.
For details, see MIREX 2026 Music Evaluation via CMI-RewardBench.
Results
Systems are ranked by overall accuracy on the 249-sample core dataset in descending order. All scores are percentages. Bold indicates the highest accuracy in each column.
| Rank | System | Core Overall (249) | Vocal Extension (300+) | Instrumental Extension (300+) |
|---|---|---|---|---|
| 1 | CloudEar-overall (User ID: 32) (PDF) | 73.9 | 60.9 | 72.6 |
| 2 | SongEval (Average) [2] | 72.7 | 62.0 | 72.0 |
| 3 | first_submission (User ID: 59) (PDF) | 72.3 | 63.3 | 75.4 |
| 4 | CloudEar (Musicality) (User ID: 32) (PDF) | 69.9 | 60.6 | 70.7 |
| 5 | CloudEar (Alignment) (User ID: 32) (PDF) | 69.5 | 61.6 | 69.5 |
| 6 | SongBench (Musicality) [5] | 69.1 | 62.0 | 68.8 |
| 7 | SongBench (Average) [5] | 69.1 | 59.6 | 72.6 |
| 8 | Baseline: CMI-RM [1] | 65.9 | 57.2 | 69.2 |
| 9 | aimusic (User ID: 68) (PDF) | 65.5 | 55.9 | 70.7 |
| 10 | TuneJury [3] | 64.3 | 53.9 | 66.0 |
| 11 | Random guess (always B) | 59.0 | 55.2 | 56.1 |
| 12 | MuseCritic [4] | 55.4 | 59.9 | 46.7 |
| 13 | SYSTASYS Resource-Lite V2 - Blind Freeze 2026 (User ID: 72) (PDF) | 39.8 | 43.4 | 40.2 |
| 13 | Flip of "SYSTASYS Resource-Lite V2 - Blind Freeze 2026" | 60.2 | 56.6 | 59.8 |
References
- Ma, Y., Xia, H., Gao, H., et al. (2026). CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), PMLR Vol. 306, pp. 85099–85130. Paper
- Yao, J., Ma, G., Xue, H., et al. (2025). SongEval: A Benchmark Dataset for Song Aesthetics Evaluation. arXiv:2505.10793. Paper
- Kim, Y., Lee, J., Xia, H., et al. (2026). TuneJury: An Open Metric for Improving Music Generation Preference Alignment. arXiv:2606.17006. Paper
- Zhuang, J., Jiang, C., Wang, H., et al. (2026). MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques. arXiv:2608.11755. Paper
- Wu, D., Lei, S., Tan, W., Li, G., Wang, Y., Zhang, H., Zuo, L., and Wu, Z. (2026). SongBench: A Fine-Grained Multi-Aspect Benchmark for Song Quality Assessment. arXiv:2604.25937. Paper Code
