Difference between revisions of "2026:Rhythm Game Chart Generation Results"
(→Expert Evaluation Results) |
|||
| Line 339: | Line 339: | ||
== Expert Evaluation Results == | == Expert Evaluation Results == | ||
| + | |||
| + | 5 osu!taiko mappers evaluated submissions across 5 songs: Silver Bullet, A fight in Tunisia, YUBIKIRI-GENMAN, Something Fun, and closer to you. Each judge was assigned three songs, and each song was evaluated by three judges. For every assigned song, judges assessed submissions from all four models. | ||
| + | |||
| + | Each submission was rated on four categories, each scored out of 10: **Musical Representation, Creativity, Gameplay, and Human-Likeness**. Category scores are reported separately as the mean of the three judges’ ratings. | ||
| + | |||
| + | For the combined quality score, each judge’s **raw score** is the sum of Musical Representation, Creativity, and Gameplay, with a maximum of 30. Human-Likeness is excluded from the combined score and reported separately. | ||
| + | |||
| + | To adjust for differences in judges’ scoring ranges, raw scores are normalized: | ||
| + | |||
| + | **Judge’s normalized score = (raw score − judge’s mean raw score) / judge’s standard deviation** | ||
| + | |||
| + | We report results under two normalization methods: | ||
| + | |||
| + | - **Across-song normalization:** The mean and standard deviation are calculated across all 12 submissions evaluated by each judge: four models across three assigned songs. | ||
| + | - **Within-song normalization:** The mean and standard deviation are calculated separately across the four submissions evaluated by each judge for a given song. | ||
| + | |||
| + | Both methods use population standard deviation. A submission’s combined quality score is the **sum of its three judges’ normalized scores**. A model’s overall quality score is the sum of its submission scores across all five songs. Higher scores indicate stronger performance relative to the relevant normalization pool. | ||
| + | |||
| + | Overall category scores are calculated by averaging the per-song category scores equally across the five songs. **Human-Likeness rankings use the original mean ratings out of 10**, independently of either normalization method. | ||
| + | |||
| + | For each song and scoring method, the highest score is highlighted, including all ties. If a normalization pool has zero standard deviation, its normalized scores are reported as unavailable. | ||
== Community Evaluation Results == | == Community Evaluation Results == | ||
Revision as of 08:21, 11 October 2026
Contents
Submissions
| Submission | Title | Authors | |
|---|---|---|---|
| DOSU | DOSU: DIFFICULTY-CONDITIONED ON-GRID SEQUENCE UNMASKING FOR OSU!TAIKO CHART GENERATION | Junyoung Koh, Hao-Wen Dong | |
| FumenLab | FUMENLAB: PLAN-CONDITIONED JOINT GENERATION AND REFINEMENT OF OSU!TAIKO CHARTS | Ryo Mitoma | |
| HiTaiko | HiTaiko: Hierarchical osu!Taiko Chart Generation via Metrical Candidate Selection | Sungjung Park | |
| MALer | RHYTHM GAME CHART GENERATION WITH PRE-TRAINED MUSIC TRANSCRIPTION MODEL | Sojeong An, Chris Donahue, Dasaem Jeong |
Test Set
Song Selection
| Title | Artist | Genre | Language | Tags | BPM |
|---|---|---|---|---|---|
| Silver Bullet | Ennnn | Electronic | Instrumental | breakcore, neurofunk, tech | 89-379 (mostly 175) |
| A fight in Tunisia | Sobrem & NIWASHI | Jazz | Instrumental | piano | 148 |
| Something Fun | JOYLESS | Rock | English | male vocal | 194 |
| closer to you (than ever before) | xiiiac13 | Electronic | Instrumental | lo-fi | 110 |
| YUBIKIRI-GENMAN | Mili | Pop | Japanese | j-pop, progressive, female vocal | 59-146 (mostly 146) |
| Himitsu feat. KAFU | MIMI | Pop | Japanese | j-pop, vocaloid | 168 |
Default Settings
| Parameter | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Overall Difficulty | 3 | 4 | 5 | 5 | 6 |
| HP Drain Rate | 8 | 7 | 6 | 5 | 5 |
Results
Algorithmic Evaluation Results
120 beatmaps were checked using AiMod across 4 models, 6 songs, and 5 difficulties.
Difficulty ordering warnings:
- DOSU - closer to you - Inner Oni < Oni
- MALer - YUBIKIRI-GENMAN - Inner Oni < Oni
- MALer - closer to you - Kantan < Oni < Futsuu < Inner Oni < Muzukashii
DOSU
| Song | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Silver Bullet | OK | OK | OK | OK | OK |
| A fight in Tunisia | OK | OK | OK | OK | OK |
| Himitsu | OK | OK | OK | OK | OK |
| yubikiri-genman | OK | OK | OK | OK | OK |
| something fun | OK | OK | OK | OK | OK |
| closer to you | OK | OK | OK | OK | OK |
FumenLab
| Song | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Silver Bullet | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | Objects <10ms apart and concurrent objects | Objects <10ms apart and concurrent objects |
| A fight in Tunisia | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | OK | OK | OK |
| Himitsu | OK | OK | OK | OK | Minor snapping issue on drumrolls and swells |
| yubikiri-genman | OK | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | OK | OK |
| something fun | OK | OK | OK | OK | OK |
| closer to you | OK | OK | OK | OK | OK |
HiTaiko
| Song | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Silver Bullet | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells | OK |
| A fight in Tunisia | OK | OK | OK | OK | OK |
| Himitsu | OK | OK | OK | OK | OK |
| yubikiri-genman | OK | OK | OK | Minor snapping issue on drumrolls and swells | Minor snapping issue on drumrolls and swells |
| something fun | OK | OK | OK | OK | OK |
| closer to you | Unused audio at the end | Unused audio at the end | Unused audio at the end | Unused audio at the end | OK |
MALer
| Song | Kantan | Futsuu | Muzukashii | Oni | Inner Oni |
|---|---|---|---|---|---|
| Silver Bullet | OK | OK | OK | OK | OK |
| A fight in Tunisia | OK | OK | OK | OK | OK |
| Himitsu | OK | OK | OK | OK | OK |
| yubikiri-genman | OK | OK | OK | OK | OK |
| something fun | OK | OK | OK | OK | OK |
| closer to you | Unused audio at the end | Unused audio at the end | Unused audio at the end | Unused audio at the end | Unused audio at the end |
Expert Evaluation Results
5 osu!taiko mappers evaluated submissions across 5 songs: Silver Bullet, A fight in Tunisia, YUBIKIRI-GENMAN, Something Fun, and closer to you. Each judge was assigned three songs, and each song was evaluated by three judges. For every assigned song, judges assessed submissions from all four models.
Each submission was rated on four categories, each scored out of 10: **Musical Representation, Creativity, Gameplay, and Human-Likeness**. Category scores are reported separately as the mean of the three judges’ ratings.
For the combined quality score, each judge’s **raw score** is the sum of Musical Representation, Creativity, and Gameplay, with a maximum of 30. Human-Likeness is excluded from the combined score and reported separately.
To adjust for differences in judges’ scoring ranges, raw scores are normalized:
- Judge’s normalized score = (raw score − judge’s mean raw score) / judge’s standard deviation**
We report results under two normalization methods:
- **Across-song normalization:** The mean and standard deviation are calculated across all 12 submissions evaluated by each judge: four models across three assigned songs. - **Within-song normalization:** The mean and standard deviation are calculated separately across the four submissions evaluated by each judge for a given song.
Both methods use population standard deviation. A submission’s combined quality score is the **sum of its three judges’ normalized scores**. A model’s overall quality score is the sum of its submission scores across all five songs. Higher scores indicate stronger performance relative to the relevant normalization pool.
Overall category scores are calculated by averaging the per-song category scores equally across the five songs. **Human-Likeness rankings use the original mean ratings out of 10**, independently of either normalization method.
For each song and scoring method, the highest score is highlighted, including all ties. If a normalization pool has zero standard deviation, its normalized scores are reported as unavailable.
Community Evaluation Results
| Mapper | Musical Representation | Creativity | Gameplay | Human-Likeness | Overall Mean |
|---|---|---|---|---|---|
| DOSU | |||||
| FumenLab | |||||
| MALer | |||||
| HiTaiko | |||||
| Amasugi (human mapper) |
