Difference between revisions of "2026:Rhythm Game Chart Generation Results"

From MIREX Wiki
(→‎Expert Evaluation Results)
(→‎Expert Evaluation Results)
Line 341: Line 341:
  
 
5 osu!taiko mappers evaluated submissions across 5 songs: Silver Bullet, A fight in Tunisia, YUBIKIRI-GENMAN, Something Fun, and closer to you. Each judge was assigned 3 songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models.
 
5 osu!taiko mappers evaluated submissions across 5 songs: Silver Bullet, A fight in Tunisia, YUBIKIRI-GENMAN, Something Fun, and closer to you. Each judge was assigned 3 songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models.
−
 
−
Each submission was rated on 4 categories, scores can be integers from 1 to 10: Musical Representation, Creativity, Gameplay, and Human-Likeness. Category scores are reported separately as the mean of the three judges’ ratings.
 
−
 
−
For the combined quality score, each judge’s raw score is the sum of Musical Representation, Creativity, and Gameplay, with a maximum of 30. Human-Likeness is excluded from the combined score and reported separately.
 
−
 
−
To adjust for differences in judges’ scoring ranges, raw scores are normalized:
 
−
 
−
Judge’s normalized score = (raw score − judge’s mean raw score) / judge’s standard deviation
 
−
 
−
We report results under two normalization methods:
 
−
 
−
- Across-song normalization: The mean and standard deviation are calculated across all 12 submissions evaluated by each judge: 4 models across 3 assigned songs.
 
−
- Within-song normalization: The mean and standard deviation are calculated separately across the 4 submissions evaluated by each judge for a given song.
 
−
 
−
Both methods use population standard deviation. A submission’s combined quality score is the sum of its 3 judges’ normalized scores. A model’s overall quality score is the sum of its submission scores across all five songs. Higher scores indicate stronger performance relative to the relevant normalization pool.
 
−
 
−
For each model, we also report an overall score for each of the 4 categories: Musical Representation, Creativity, Gameplay, and Human-Likeness. Each overall category score is the mean of its 5 per-song category scores, giving every song equal weight. These scores remain on the original 1–10 scale and are reported separately from the normalized combined quality scores.
 
−
 
−
For each song and scoring method, the highest score is highlighted, including all ties. If a normalization pool has zero standard deviation, its normalized scores are reported as unavailable.
 
  
 
== Community Evaluation Results ==
 
== Community Evaluation Results ==

Revision as of 08:36, 11 October 2026

Submissions

Submission Title PDF Authors
DOSU DOSU: DIFFICULTY-CONDITIONED ON-GRID SEQUENCE UNMASKING FOR OSU!TAIKO CHART GENERATION Junyoung Koh, Hao-Wen Dong
FumenLab FUMENLAB: PLAN-CONDITIONED JOINT GENERATION AND REFINEMENT OF OSU!TAIKO CHARTS Ryo Mitoma
HiTaiko HiTaiko: Hierarchical osu!Taiko Chart Generation via Metrical Candidate Selection Sungjung Park
MALer RHYTHM GAME CHART GENERATION WITH PRE-TRAINED MUSIC TRANSCRIPTION MODEL Sojeong An, Chris Donahue, Dasaem Jeong

Test Set

Song Selection

Title Artist Genre Language Tags BPM
Silver Bullet Ennnn Electronic Instrumental breakcore, neurofunk, tech 89-379 (mostly 175)
A fight in Tunisia Sobrem & NIWASHI Jazz Instrumental piano 148
Something Fun JOYLESS Rock English male vocal 194
closer to you (than ever before) xiiiac13 Electronic Instrumental lo-fi 110
YUBIKIRI-GENMAN Mili Pop Japanese j-pop, progressive, female vocal 59-146 (mostly 146)
Himitsu feat. KAFU MIMI Pop Japanese j-pop, vocaloid 168

Default Settings

Parameter Kantan Futsuu Muzukashii Oni Inner Oni
Overall Difficulty 3 4 5 5 6
HP Drain Rate 8 7 6 5 5

Results

Algorithmic Evaluation Results

120 beatmaps were checked using AiMod across 4 models, 6 songs, and 5 difficulties.

Difficulty ordering warnings:

  • DOSU - closer to you - Inner Oni < Oni
  • MALer - YUBIKIRI-GENMAN - Inner Oni < Oni
  • MALer - closer to you - Kantan < Oni < Futsuu < Inner Oni < Muzukashii

DOSU

Song Kantan Futsuu Muzukashii Oni Inner Oni
Silver Bullet OK OK OK OK OK
A fight in Tunisia OK OK OK OK OK
Himitsu OK OK OK OK OK
yubikiri-genman OK OK OK OK OK
something fun OK OK OK OK OK
closer to you OK OK OK OK OK

FumenLab

Song Kantan Futsuu Muzukashii Oni Inner Oni
Silver Bullet Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells Objects <10ms apart and concurrent objects Objects <10ms apart and concurrent objects
A fight in Tunisia Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells OK OK OK
Himitsu OK OK OK OK Minor snapping issue on drumrolls and swells
yubikiri-genman OK Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells OK OK
something fun OK OK OK OK OK
closer to you OK OK OK OK OK

HiTaiko

Song Kantan Futsuu Muzukashii Oni Inner Oni
Silver Bullet Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells OK
A fight in Tunisia OK OK OK OK OK
Himitsu OK OK OK OK OK
yubikiri-genman OK OK OK Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells
something fun OK OK OK OK OK
closer to you Unused audio at the end Unused audio at the end Unused audio at the end Unused audio at the end OK

MALer

Song Kantan Futsuu Muzukashii Oni Inner Oni
Silver Bullet OK OK OK OK OK
A fight in Tunisia OK OK OK OK OK
Himitsu OK OK OK OK OK
yubikiri-genman OK OK OK OK OK
something fun OK OK OK OK OK
closer to you Unused audio at the end Unused audio at the end Unused audio at the end Unused audio at the end Unused audio at the end

Expert Evaluation Results

5 osu!taiko mappers evaluated submissions across 5 songs: Silver Bullet, A fight in Tunisia, YUBIKIRI-GENMAN, Something Fun, and closer to you. Each judge was assigned 3 songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models.

Community Evaluation Results

Mapper Musical Representation Creativity Gameplay Human-Likeness Overall Mean
DOSU
FumenLab
MALer
HiTaiko
Amasugi (human mapper)
Retrieved from "https://music-ir.org/mirex/w/index.php?title=2026:Rhythm_Game_Chart_Generation_Results&oldid=15121"