Difference between revisions of "2026:Rhythm Game Chart Generation Results"
(→Expert Evaluation Results) |
(→Expert Evaluation Results) |
||
| Line 340: | Line 340: | ||
== Expert Evaluation Results == | == Expert Evaluation Results == | ||
| − | 5 osu!taiko mappers evaluated submissions across 5 songs: Silver Bullet, A fight in Tunisia, YUBIKIRI-GENMAN, Something Fun, and closer to you. Each judge was assigned 3 songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all | + | 5 osu!taiko mappers evaluated submissions across 5 songs: Silver Bullet, A fight in Tunisia, YUBIKIRI-GENMAN, Something Fun, and closer to you. Each judge was assigned 3 songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models. |
Each submission was rated on 4 categories, scores can be integers from 1 to 10: Musical Representation, Creativity, Gameplay, and Human-Likeness. Category scores are reported separately as the mean of the three judges’ ratings. | Each submission was rated on 4 categories, scores can be integers from 1 to 10: Musical Representation, Creativity, Gameplay, and Human-Likeness. Category scores are reported separately as the mean of the three judges’ ratings. | ||
Revision as of 08:32, 11 October 2026
Contents
Submissions
| Submission | Title | Authors | |
|---|---|---|---|
| DOSU | DOSU: DIFFICULTY-CONDITIONED ON-GRID SEQUENCE UNMASKING FOR OSU!TAIKO CHART GENERATION | Junyoung Koh, Hao-Wen Dong | |
| FumenLab | FUMENLAB: PLAN-CONDITIONED JOINT GENERATION AND REFINEMENT OF OSU!TAIKO CHARTS | Ryo Mitoma | |
| HiTaiko | HiTaiko: Hierarchical osu!Taiko Chart Generation via Metrical Candidate Selection | Sungjung Park | |
| MALer | RHYTHM GAME CHART GENERATION WITH PRE-TRAINED MUSIC TRANSCRIPTION MODEL | Sojeong An, Chris Donahue, Dasaem Jeong |
Test Set
Song Selection
| Title | Artist | Genre | Language | Tags | BPM |
|---|---|---|---|---|---|
| Silver Bullet | Ennnn | Electronic | Instrumental | breakcore, neurofunk, tech | 89-379 (mostly 175) |
| A fight in Tunisia | Sobrem & NIWASHI | Jazz | Instrumental | piano | 148 |
| Something Fun | JOYLESS | Rock | English | male vocal | 194 |
| closer to you (than ever before) | xiiiac13 | Electronic | Instrumental | lo-fi | 110 |
| YUBIKIRI-GENMAN | Mili | Pop | Japanese | j-pop, progressive, female vocal | 59-146 (mostly 146) |
| Himitsu feat. KAFU | MIMI | Pop | Japanese | j-pop, vocaloid | 168 |
Default Settings
| Parameter | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Overall Difficulty | 3 | 4 | 5 | 5 | 6 |
| HP Drain Rate | 8 | 7 | 6 | 5 | 5 |
Results
Algorithmic Evaluation Results
120 beatmaps were checked using AiMod across 4 models, 6 songs, and 5 difficulties.
Difficulty ordering warnings:
- DOSU - closer to you - Inner Oni < Oni
- MALer - YUBIKIRI-GENMAN - Inner Oni < Oni
- MALer - closer to you - Kantan < Oni < Futsuu < Inner Oni < Muzukashii
DOSU
| Song | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Silver Bullet | OK | OK | OK | OK | OK |
| A fight in Tunisia | OK | OK | OK | OK | OK |
| Himitsu | OK | OK | OK | OK | OK |
| yubikiri-genman | OK | OK | OK | OK | OK |
| something fun | OK | OK | OK | OK | OK |
| closer to you | OK | OK | OK | OK | OK |
FumenLab
| Song | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Silver Bullet | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | Objects <10ms apart and concurrent objects | Objects <10ms apart and concurrent objects |
| A fight in Tunisia | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | OK | OK | OK |
| Himitsu | OK | OK | OK | OK | Minor snapping issue on drumrolls and swells |
| yubikiri-genman | OK | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | OK | OK |
| something fun | OK | OK | OK | OK | OK |
| closer to you | OK | OK | OK | OK | OK |
HiTaiko
| Song | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Silver Bullet | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | OK |
| A fight in Tunisia | OK | OK | OK | OK | OK |
| Himitsu | OK | OK | OK | OK | OK |
| yubikiri-genman | OK | OK | OK | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells |
| something fun | OK | OK | OK | OK | OK |
| closer to you | Unused audio at the end | Unused audio at the end | Unused audio at the end | Unused audio at the end | OK |
MALer
| Song | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Silver Bullet | OK | OK | OK | OK | OK |
| A fight in Tunisia | OK | OK | OK | OK | OK |
| Himitsu | OK | OK | OK | OK | OK |
| yubikiri-genman | OK | OK | OK | OK | OK |
| something fun | OK | OK | OK | OK | OK |
| closer to you | Unused audio at the end | Unused audio at the end | Unused audio at the end | Unused audio at the end | Unused audio at the end |
Expert Evaluation Results
5 osu!taiko mappers evaluated submissions across 5 songs: Silver Bullet, A fight in Tunisia, YUBIKIRI-GENMAN, Something Fun, and closer to you. Each judge was assigned 3 songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models.
Each submission was rated on 4 categories, scores can be integers from 1 to 10: Musical Representation, Creativity, Gameplay, and Human-Likeness. Category scores are reported separately as the mean of the three judges’ ratings.
For the combined quality score, each judge’s raw score is the sum of Musical Representation, Creativity, and Gameplay, with a maximum of 30. Human-Likeness is excluded from the combined score and reported separately.
To adjust for differences in judges’ scoring ranges, raw scores are normalized:
Judge’s normalized score = (raw score − judge’s mean raw score) / judge’s standard deviation
We report results under two normalization methods:
- Across-song normalization: The mean and standard deviation are calculated across all 12 submissions evaluated by each judge: 4 models across 3 assigned songs. - Within-song normalization: The mean and standard deviation are calculated separately across the 4 submissions evaluated by each judge for a given song.
Both methods use population standard deviation. A submission’s combined quality score is the sum of its 3 judges’ normalized scores. A model’s overall quality score is the sum of its submission scores across all five songs. Higher scores indicate stronger performance relative to the relevant normalization pool.
For each model, we also report an overall score for each of the 4 categories: Musical Representation, Creativity, Gameplay, and Human-Likeness. Each overall category score is the mean of its 5 per-song category scores, giving every song equal weight. These scores remain on the original 1–10 scale and are reported separately from the normalized combined quality scores.
For each song and scoring method, the highest score is highlighted, including all ties. If a normalization pool has zero standard deviation, its normalized scores are reported as unavailable.
Community Evaluation Results
| Mapper | Musical Representation | Creativity | Gameplay | Human-Likeness | Overall Mean |
|---|---|---|---|---|---|
| DOSU | |||||
| FumenLab | |||||
| MALer | |||||
| HiTaiko | |||||
| Amasugi (human mapper) |
