Difference between revisions of "2026:Music Evaluation via CMI-RewardBench Results"
From MIREX Wiki
Yinghao Ma (talk | contribs) (Created page with " =Music Evaluation via CMI-RewardBench Results= ==Task Description== The MIREX 2026 Music Evaluation via CMI-RewardBench task evaluates automatic music reward models and eva...") |
(→Results: PDF links for submissions (abstracts archived on futuremirex.com)) |
||
| (3 intermediate revisions by 2 users not shown) | |||
| Line 25: | Line 25: | ||
|- | |- | ||
| 1 | | 1 | ||
| − | | CloudEar-overall (User ID: 32) | + | | CloudEar-overall (User ID: 32) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/CloudEar.pdf PDF]) |
| style="text-align:right;" | '''73.9''' | | style="text-align:right;" | '''73.9''' | ||
| style="text-align:right;" | 60.9 | | style="text-align:right;" | 60.9 | ||
| Line 31: | Line 31: | ||
|- | |- | ||
| 2 | | 2 | ||
| − | | SongEval | + | | SongEval (Average) [https://arxiv.org/abs/2505.10793 [2]] |
| style="text-align:right;" | 72.7 | | style="text-align:right;" | 72.7 | ||
| style="text-align:right;" | 62.0 | | style="text-align:right;" | 62.0 | ||
| Line 37: | Line 37: | ||
|- | |- | ||
| 3 | | 3 | ||
| − | | first_submission (User ID: 59) | + | | first_submission (User ID: 59) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/first_submission.pdf PDF]) |
| style="text-align:right;" | 72.3 | | style="text-align:right;" | 72.3 | ||
| style="text-align:right;" | '''63.3''' | | style="text-align:right;" | '''63.3''' | ||
| Line 43: | Line 43: | ||
|- | |- | ||
| 4 | | 4 | ||
| − | | CloudEar | + | | CloudEar (Musicality) (User ID: 32) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/CloudEar.pdf PDF]) |
| style="text-align:right;" | 69.9 | | style="text-align:right;" | 69.9 | ||
| style="text-align:right;" | 60.6 | | style="text-align:right;" | 60.6 | ||
| Line 49: | Line 49: | ||
|- | |- | ||
| 5 | | 5 | ||
| − | | CloudEar | + | | CloudEar (Alignment) (User ID: 32) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/CloudEar.pdf PDF]) |
| style="text-align:right;" | 69.5 | | style="text-align:right;" | 69.5 | ||
| style="text-align:right;" | 61.6 | | style="text-align:right;" | 61.6 | ||
| Line 55: | Line 55: | ||
|- | |- | ||
| 6 | | 6 | ||
| + | | SongBench (Musicality) [https://arxiv.org/abs/2604.25937 [5]] | ||
| + | | style="text-align:right;" | 69.1 | ||
| + | | style="text-align:right;" | 62.0 | ||
| + | | style="text-align:right;" | 68.8 | ||
| + | |- | ||
| + | | 7 | ||
| + | | SongBench (Average) [https://arxiv.org/abs/2604.25937 [5]] | ||
| + | | style="text-align:right;" | 69.1 | ||
| + | | style="text-align:right;" | 59.6 | ||
| + | | style="text-align:right;" | 72.6 | ||
| + | |- | ||
| + | | 8 | ||
| Baseline: CMI-RM [https://proceedings.mlr.press/v306/ma26am.html [1]] | | Baseline: CMI-RM [https://proceedings.mlr.press/v306/ma26am.html [1]] | ||
| style="text-align:right;" | 65.9 | | style="text-align:right;" | 65.9 | ||
| Line 60: | Line 72: | ||
| style="text-align:right;" | 69.2 | | style="text-align:right;" | 69.2 | ||
|- | |- | ||
| − | | | + | | 9 |
| − | | aimusic (User ID: 68) | + | | aimusic (User ID: 68) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/aimusic.pdf PDF]) |
| style="text-align:right;" | 65.5 | | style="text-align:right;" | 65.5 | ||
| style="text-align:right;" | 55.9 | | style="text-align:right;" | 55.9 | ||
| style="text-align:right;" | 70.7 | | style="text-align:right;" | 70.7 | ||
|- | |- | ||
| − | | | + | | 10 |
| TuneJury [https://arxiv.org/abs/2606.17006 [3]] | | TuneJury [https://arxiv.org/abs/2606.17006 [3]] | ||
| style="text-align:right;" | 64.3 | | style="text-align:right;" | 64.3 | ||
| Line 72: | Line 84: | ||
| style="text-align:right;" | 66.0 | | style="text-align:right;" | 66.0 | ||
|- | |- | ||
| − | | | + | | 11 |
| Random guess (always B) | | Random guess (always B) | ||
| style="text-align:right;" | 59.0 | | style="text-align:right;" | 59.0 | ||
| Line 78: | Line 90: | ||
| style="text-align:right;" | 56.1 | | style="text-align:right;" | 56.1 | ||
|- | |- | ||
| − | | | + | | 12 |
| MuseCritic [https://arxiv.org/abs/2608.11755 [4]] | | MuseCritic [https://arxiv.org/abs/2608.11755 [4]] | ||
| style="text-align:right;" | 55.4 | | style="text-align:right;" | 55.4 | ||
| Line 84: | Line 96: | ||
| style="text-align:right;" | 46.7 | | style="text-align:right;" | 46.7 | ||
|- | |- | ||
| − | | | + | | 13 |
| − | | SYSTASYS Resource-Lite V2 - Blind Freeze 2026 (User ID: 72) | + | | SYSTASYS Resource-Lite V2 - Blind Freeze 2026 (User ID: 72) ([http://futuremirex.com/portal/wp-content/uploads/2026/music-evaluation-via-cmi-rewardbench/SYSTASYS.pdf PDF]) |
| style="text-align:right;" | 39.8 | | style="text-align:right;" | 39.8 | ||
| style="text-align:right;" | 43.4 | | style="text-align:right;" | 43.4 | ||
| style="text-align:right;" | 40.2 | | style="text-align:right;" | 40.2 | ||
|- | |- | ||
| − | | | + | | 13 |
| Flip of "SYSTASYS Resource-Lite V2 - Blind Freeze 2026" | | Flip of "SYSTASYS Resource-Lite V2 - Blind Freeze 2026" | ||
| style="text-align:right;" | 60.2 | | style="text-align:right;" | 60.2 | ||
| Line 103: | Line 115: | ||
# Kim, Y., Lee, J., Xia, H., et al. (2026). ''TuneJury: An Open Metric for Improving Music Generation Preference Alignment.'' arXiv:2606.17006. [https://arxiv.org/abs/2606.17006 Paper] | # Kim, Y., Lee, J., Xia, H., et al. (2026). ''TuneJury: An Open Metric for Improving Music Generation Preference Alignment.'' arXiv:2606.17006. [https://arxiv.org/abs/2606.17006 Paper] | ||
# Zhuang, J., Jiang, C., Wang, H., et al. (2026). ''MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques.'' arXiv:2608.11755. [https://arxiv.org/abs/2608.11755 Paper] | # Zhuang, J., Jiang, C., Wang, H., et al. (2026). ''MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques.'' arXiv:2608.11755. [https://arxiv.org/abs/2608.11755 Paper] | ||
| + | # Wu, D., Lei, S., Tan, W., Li, G., Wang, Y., Zhang, H., Zuo, L., and Wu, Z. (2026). ''SongBench: A Fine-Grained Multi-Aspect Benchmark for Song Quality Assessment.'' arXiv:2604.25937. [https://arxiv.org/abs/2604.25937 Paper] [https://github.com/Tencent/SongBench Code] | ||
Latest revision as of 10:23, 14 October 2026
Contents
Music Evaluation via CMI-RewardBench Results
Task Description
The MIREX 2026 Music Evaluation via CMI-RewardBench task evaluates automatic music reward models and evaluation metrics according to their agreement with human preferences.
Given a text prompt and two generated music candidates (Audio A and Audio B), each system predicts which candidate is preferred by human listeners. Systems are evaluated using pairwise preference accuracy.
The evaluation reports results for the 249-sample core dataset and vocal and instrumental subsets of the 618-sample extended dataset.
For details, see MIREX 2026 Music Evaluation via CMI-RewardBench.
Results
Systems are ranked by overall accuracy on the 249-sample core dataset in descending order. All scores are percentages. Bold indicates the highest accuracy in each column.
| Rank | System | Core Overall (249) | Vocal Extension (300+) | Instrumental Extension (300+) |
|---|---|---|---|---|
| 1 | CloudEar-overall (User ID: 32) (PDF) | 73.9 | 60.9 | 72.6 |
| 2 | SongEval (Average) [2] | 72.7 | 62.0 | 72.0 |
| 3 | first_submission (User ID: 59) (PDF) | 72.3 | 63.3 | 75.4 |
| 4 | CloudEar (Musicality) (User ID: 32) (PDF) | 69.9 | 60.6 | 70.7 |
| 5 | CloudEar (Alignment) (User ID: 32) (PDF) | 69.5 | 61.6 | 69.5 |
| 6 | SongBench (Musicality) [5] | 69.1 | 62.0 | 68.8 |
| 7 | SongBench (Average) [5] | 69.1 | 59.6 | 72.6 |
| 8 | Baseline: CMI-RM [1] | 65.9 | 57.2 | 69.2 |
| 9 | aimusic (User ID: 68) (PDF) | 65.5 | 55.9 | 70.7 |
| 10 | TuneJury [3] | 64.3 | 53.9 | 66.0 |
| 11 | Random guess (always B) | 59.0 | 55.2 | 56.1 |
| 12 | MuseCritic [4] | 55.4 | 59.9 | 46.7 |
| 13 | SYSTASYS Resource-Lite V2 - Blind Freeze 2026 (User ID: 72) (PDF) | 39.8 | 43.4 | 40.2 |
| 13 | Flip of "SYSTASYS Resource-Lite V2 - Blind Freeze 2026" | 60.2 | 56.6 | 59.8 |
References
- Ma, Y., Xia, H., Gao, H., et al. (2026). CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), PMLR Vol. 306, pp. 85099–85130. Paper
- Yao, J., Ma, G., Xue, H., et al. (2025). SongEval: A Benchmark Dataset for Song Aesthetics Evaluation. arXiv:2505.10793. Paper
- Kim, Y., Lee, J., Xia, H., et al. (2026). TuneJury: An Open Metric for Improving Music Generation Preference Alignment. arXiv:2606.17006. Paper
- Zhuang, J., Jiang, C., Wang, H., et al. (2026). MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques. arXiv:2608.11755. Paper
- Wu, D., Lei, S., Tan, W., Li, G., Wang, Y., Zhang, H., Zuo, L., and Wu, Z. (2026). SongBench: A Fine-Grained Multi-Aspect Benchmark for Song Quality Assessment. arXiv:2604.25937. Paper Code
