Difference between revisions of "2026:Rhythm Game Chart Generation Results"
From MIREX Wiki
(→Algorithmic Evaluation Results) |
(→Expert Evaluation Results) |
||
| Line 340: | Line 340: | ||
== Expert Evaluation Results == | == Expert Evaluation Results == | ||
| − | 5 osu!taiko mappers evaluated submissions across 5 songs: '''Silver Bullet''', '''A fight in Tunisia''', '''YUBIKIRI-GENMAN''', '''Something Fun''', and '''closer to you'''. Each judge was assigned 3 songs based on their familiarity and interest of the songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models on 4 categories (Musical Representation, Creativity, Gameplay, and Human-Likeness) using an integer scale from 1 to 10. We follow [https://osu.ppy.sh/wiki/en/Contests/Monthly_Beatmapping_Contest Monthly Beatmapping Contest] and adopt '''within-song normalization'''. For each category, judge X's final score = (judge X's raw score - judge X's average raw score) / judge X's standard deviation within that song. | + | 5 osu!taiko mappers evaluated submissions across 5 songs: '''Silver Bullet''', '''A fight in Tunisia''', '''YUBIKIRI-GENMAN''', '''Something Fun''', and '''closer to you'''. Each judge was assigned 3 songs based on their familiarity and interest of the songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models on 4 categories (Musical Representation, Creativity, Gameplay, and Human-Likeness) using an integer scale from 1 to 10. We follow [https://osu.ppy.sh/wiki/en/Contests/Monthly_Beatmapping_Contest Monthly Beatmapping Contest] and adopt '''within-song normalization'''. For each category, judge X's final score = (judge X's raw score - judge X's average raw score) / judge X's standard deviation within that song, and a submission's score is the sum of each judge's final score. |
== Community Evaluation Results == | == Community Evaluation Results == | ||
Revision as of 09:24, 11 October 2026
Contents
Submissions
| Submission | Title | Authors | |
|---|---|---|---|
| DOSU | DOSU: DIFFICULTY-CONDITIONED ON-GRID SEQUENCE UNMASKING FOR OSU!TAIKO CHART GENERATION | Junyoung Koh, Hao-Wen Dong | |
| FumenLab | FUMENLAB: PLAN-CONDITIONED JOINT GENERATION AND REFINEMENT OF OSU!TAIKO CHARTS | Ryo Mitoma | |
| HiTaiko | HiTaiko: Hierarchical osu!Taiko Chart Generation via Metrical Candidate Selection | Sungjung Park | |
| MALer | RHYTHM GAME CHART GENERATION WITH PRE-TRAINED MUSIC TRANSCRIPTION MODEL | Sojeong An, Chris Donahue, Dasaem Jeong |
Test Set
Song Selection
| Title | Artist | Genre | Language | Tags | BPM |
|---|---|---|---|---|---|
| Silver Bullet | Ennnn | Electronic | Instrumental | breakcore, neurofunk, tech | 89-379 (mostly 175) |
| A fight in Tunisia | Sobrem & NIWASHI | Jazz | Instrumental | piano | 148 |
| Something Fun | JOYLESS | Rock | English | male vocal | 194 |
| closer to you (than ever before) | xiiiac13 | Electronic | Instrumental | lo-fi | 110 |
| YUBIKIRI-GENMAN | Mili | Pop | Japanese | j-pop, progressive, female vocal | 59-146 (mostly 146) |
| Himitsu feat. KAFU | MIMI | Pop | Japanese | j-pop, vocaloid | 168 |
Default Settings
| Parameter | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Overall Difficulty | 3 | 4 | 5 | 5 | 6 |
| HP Drain Rate | 8 | 7 | 6 | 5 | 5 |
Results
Algorithmic Evaluation Results
120 beatmaps were checked using AiMod across 4 models, 6 songs, and 5 difficulties.
Difficulty ordering warnings:
- DOSU - closer to you - Inner Oni < Oni
- MALer - YUBIKIRI-GENMAN - Inner Oni < Oni
- MALer - closer to you - Kantan < Oni < Futsuu < Inner Oni < Muzukashii
DOSU
| Song | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Silver Bullet | OK | OK | OK | OK | OK |
| A fight in Tunisia | OK | OK | OK | OK | OK |
| Himitsu | OK | OK | OK | OK | OK |
| yubikiri-genman | OK | OK | OK | OK | OK |
| something fun | OK | OK | OK | OK | OK |
| closer to you | OK | OK | OK | OK | OK |
FumenLab
| Song | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Silver Bullet | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | Objects <10ms apart and concurrent objects | Objects <10ms apart and concurrent objects |
| A fight in Tunisia | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | OK | OK | OK |
| Himitsu | OK | OK | OK | OK | Minor snapping issue on drumrolls and swells |
| yubikiri-genman | OK | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | OK | OK |
| something fun | OK | OK | OK | OK | OK |
| closer to you | OK | OK | OK | OK | OK |
HiTaiko
| Song | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Silver Bullet | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | OK |
| A fight in Tunisia | OK | OK | OK | OK | OK |
| Himitsu | OK | OK | OK | OK | OK |
| yubikiri-genman | OK | OK | OK | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells |
| something fun | OK | OK | OK | OK | OK |
| closer to you | Unused audio at the end | Unused audio at the end | Unused audio at the end | Unused audio at the end | OK |
MALer
| Song | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Silver Bullet | OK | OK | OK | OK | OK |
| A fight in Tunisia | OK | OK | OK | OK | OK |
| Himitsu | OK | OK | OK | OK | OK |
| yubikiri-genman | OK | OK | OK | OK | OK |
| something fun | OK | OK | OK | OK | OK |
| closer to you | Unused audio at the end | Unused audio at the end | Unused audio at the end | Unused audio at the end | Unused audio at the end |
Expert Evaluation Results
5 osu!taiko mappers evaluated submissions across 5 songs: Silver Bullet, A fight in Tunisia, YUBIKIRI-GENMAN, Something Fun, and closer to you. Each judge was assigned 3 songs based on their familiarity and interest of the songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models on 4 categories (Musical Representation, Creativity, Gameplay, and Human-Likeness) using an integer scale from 1 to 10. We follow Monthly Beatmapping Contest and adopt within-song normalization. For each category, judge X's final score = (judge X's raw score - judge X's average raw score) / judge X's standard deviation within that song, and a submission's score is the sum of each judge's final score.
Community Evaluation Results
| Mapper | Musical Representation | Creativity | Gameplay | Human-Likeness | Overall Mean |
|---|---|---|---|---|---|
| DOSU | |||||
| FumenLab | |||||
| MALer | |||||
| HiTaiko | |||||
| Amasugi (human mapper) |
