Difference between revisions of "2026:Rhythm Game Chart Generation Results"

From MIREX Wiki
(→‎Expert Evaluation Results)
(→‎Expert Evaluation Results)
Line 341: Line 341:
  
 
5 osu!taiko mappers evaluated submissions across 5 songs: '''Silver Bullet''', '''A fight in Tunisia''', '''YUBIKIRI-GENMAN''', '''Something Fun''', and '''closer to you'''. Each judge was assigned 3 songs based on their familiarity and interest of the songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models on 4 categories (Musical Representation, Creativity, Gameplay, and Human-Likeness) using an integer scale from 1 to 10. We follow [https://osu.ppy.sh/wiki/en/Contests/Monthly_Beatmapping_Contest Monthly Beatmapping Contest] and adopt '''within-song normalization'''. For each category, judge X's final score = (judge X's raw score - judge X's average raw score) / judge X's standard deviation within that song, and a submission's score is the sum of each judge's final score.
 
5 osu!taiko mappers evaluated submissions across 5 songs: '''Silver Bullet''', '''A fight in Tunisia''', '''YUBIKIRI-GENMAN''', '''Something Fun''', and '''closer to you'''. Each judge was assigned 3 songs based on their familiarity and interest of the songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models on 4 categories (Musical Representation, Creativity, Gameplay, and Human-Likeness) using an integer scale from 1 to 10. We follow [https://osu.ppy.sh/wiki/en/Contests/Monthly_Beatmapping_Contest Monthly Beatmapping Contest] and adopt '''within-song normalization'''. For each category, judge X's final score = (judge X's raw score - judge X's average raw score) / judge X's standard deviation within that song, and a submission's score is the sum of each judge's final score.
 +
 +
{| class="wikitable"
 +
|+ Silver Bullet
 +
! Model
 +
! Musical Representation
 +
! Creativity
 +
! Gameplay
 +
! Human-Likeness
 +
|-
 +
| DOSU
 +
| '''1.5872'''
 +
| −2.6304
 +
| '''2.0099'''
 +
| −0.4552
 +
|-
 +
| FumenLab
 +
| 0.3005
 +
| 0.9401
 +
| −3.5863
 +
| −1.9478
 +
|-
 +
| HiTaiko
 +
| 1.2778
 +
| '''1.8629'''
 +
| '''2.0099'''
 +
| '''4.1634'''
 +
|-
 +
| MALer
 +
| −3.1656
 +
| −0.1726
 +
| −0.4335
 +
| −1.7604
 +
|}
  
 
== Community Evaluation Results ==
 
== Community Evaluation Results ==

Revision as of 09:25, 11 October 2026

Submissions

Submission Title PDF Authors
DOSU DOSU: DIFFICULTY-CONDITIONED ON-GRID SEQUENCE UNMASKING FOR OSU!TAIKO CHART GENERATION Junyoung Koh, Hao-Wen Dong
FumenLab FUMENLAB: PLAN-CONDITIONED JOINT GENERATION AND REFINEMENT OF OSU!TAIKO CHARTS Ryo Mitoma
HiTaiko HiTaiko: Hierarchical osu!Taiko Chart Generation via Metrical Candidate Selection Sungjung Park
MALer RHYTHM GAME CHART GENERATION WITH PRE-TRAINED MUSIC TRANSCRIPTION MODEL Sojeong An, Chris Donahue, Dasaem Jeong

Test Set

Song Selection

Title Artist Genre Language Tags BPM
Silver Bullet Ennnn Electronic Instrumental breakcore, neurofunk, tech 89-379 (mostly 175)
A fight in Tunisia Sobrem & NIWASHI Jazz Instrumental piano 148
Something Fun JOYLESS Rock English male vocal 194
closer to you (than ever before) xiiiac13 Electronic Instrumental lo-fi 110
YUBIKIRI-GENMAN Mili Pop Japanese j-pop, progressive, female vocal 59-146 (mostly 146)
Himitsu feat. KAFU MIMI Pop Japanese j-pop, vocaloid 168

Default Settings

Parameter Kantan Futsuu Muzukashii Oni Inner Oni
Overall Difficulty 3 4 5 5 6
HP Drain Rate 8 7 6 5 5

Results

Algorithmic Evaluation Results

120 beatmaps were checked using AiMod across 4 models, 6 songs, and 5 difficulties.

Difficulty ordering warnings:

  • DOSU - closer to you - Inner Oni < Oni
  • MALer - YUBIKIRI-GENMAN - Inner Oni < Oni
  • MALer - closer to you - Kantan < Oni < Futsuu < Inner Oni < Muzukashii

DOSU

Song Kantan Futsuu Muzukashii Oni Inner Oni
Silver Bullet OK OK OK OK OK
A fight in Tunisia OK OK OK OK OK
Himitsu OK OK OK OK OK
yubikiri-genman OK OK OK OK OK
something fun OK OK OK OK OK
closer to you OK OK OK OK OK

FumenLab

Song Kantan Futsuu Muzukashii Oni Inner Oni
Silver Bullet Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells Objects <10ms apart and concurrent objects Objects <10ms apart and concurrent objects
A fight in Tunisia Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells OK OK OK
Himitsu OK OK OK OK Minor snapping issue on drumrolls and swells
yubikiri-genman OK Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells OK OK
something fun OK OK OK OK OK
closer to you OK OK OK OK OK

HiTaiko

Song Kantan Futsuu Muzukashii Oni Inner Oni
Silver Bullet Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells OK
A fight in Tunisia OK OK OK OK OK
Himitsu OK OK OK OK OK
yubikiri-genman OK OK OK Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells
something fun OK OK OK OK OK
closer to you Unused audio at the end Unused audio at the end Unused audio at the end Unused audio at the end OK

MALer

Song Kantan Futsuu Muzukashii Oni Inner Oni
Silver Bullet OK OK OK OK OK
A fight in Tunisia OK OK OK OK OK
Himitsu OK OK OK OK OK
yubikiri-genman OK OK OK OK OK
something fun OK OK OK OK OK
closer to you Unused audio at the end Unused audio at the end Unused audio at the end Unused audio at the end Unused audio at the end

Expert Evaluation Results

5 osu!taiko mappers evaluated submissions across 5 songs: Silver Bullet, A fight in Tunisia, YUBIKIRI-GENMAN, Something Fun, and closer to you. Each judge was assigned 3 songs based on their familiarity and interest of the songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models on 4 categories (Musical Representation, Creativity, Gameplay, and Human-Likeness) using an integer scale from 1 to 10. We follow Monthly Beatmapping Contest and adopt within-song normalization. For each category, judge X's final score = (judge X's raw score - judge X's average raw score) / judge X's standard deviation within that song, and a submission's score is the sum of each judge's final score.

Silver Bullet
Model Musical Representation Creativity Gameplay Human-Likeness
DOSU 1.5872 −2.6304 2.0099 −0.4552
FumenLab 0.3005 0.9401 −3.5863 −1.9478
HiTaiko 1.2778 1.8629 2.0099 4.1634
MALer −3.1656 −0.1726 −0.4335 −1.7604

Community Evaluation Results

Mapper Musical Representation Creativity Gameplay Human-Likeness Overall Mean
DOSU
FumenLab
MALer
HiTaiko
Amasugi (human mapper)
Retrieved from "https://music-ir.org/mirex/w/index.php?title=2026:Rhythm_Game_Chart_Generation_Results&oldid=15127"