Difference between revisions of "2026:Rhythm Game Chart Generation Results"

From MIREX Wiki
(→‎Community Evaluation Results)
(→‎Community Evaluation Results)
 
(15 intermediate revisions by 2 users not shown)
Line 147: Line 147:
 
| OK
 
| OK
 
|-
 
|-
−
| Himitsu
+
| Himitsu feat. KAFU
 
| OK
 
| OK
 
| OK
 
| OK
Line 154: Line 154:
 
| OK
 
| OK
 
|-
 
|-
−
| yubikiri-genman
+
| YUBIKIRI-GENMAN
 
| OK
 
| OK
 
| OK
 
| OK
Line 161: Line 161:
 
| OK
 
| OK
 
|-
 
|-
−
| something fun
+
| Something Fun
 
| OK
 
| OK
 
| OK
 
| OK
Line 201: Line 201:
 
| OK
 
| OK
 
|-
 
|-
−
| Himitsu
+
| Himitsu feat. KAFU
 
| OK
 
| OK
 
| OK
 
| OK
Line 208: Line 208:
 
| Minor snapping issue on drumrolls and swells
 
| Minor snapping issue on drumrolls and swells
 
|-
 
|-
−
| yubikiri-genman
+
| YUBIKIRI-GENMAN
 
| OK
 
| OK
 
| Minor snapping issue on drumrolls and swells
 
| Minor snapping issue on drumrolls and swells
Line 215: Line 215:
 
| OK
 
| OK
 
|-
 
|-
−
| something fun
+
| Something Fun
 
| OK
 
| OK
 
| OK
 
| OK
Line 255: Line 255:
 
| OK
 
| OK
 
|-
 
|-
−
| Himitsu
+
| Himitsu feat. KAFU
 
| OK
 
| OK
 
| OK
 
| OK
Line 262: Line 262:
 
| OK
 
| OK
 
|-
 
|-
−
| yubikiri-genman
+
| YUBIKIRI-GENMAN
 
| OK
 
| OK
 
| OK
 
| OK
Line 269: Line 269:
 
| Minor snapping issue on drumrolls and swells
 
| Minor snapping issue on drumrolls and swells
 
|-
 
|-
−
| something fun
+
| Something Fun
 
| OK
 
| OK
 
| OK
 
| OK
Line 309: Line 309:
 
| OK
 
| OK
 
|-
 
|-
−
| Himitsu
+
| Himitsu feat. KAFU
 
| OK
 
| OK
 
| OK
 
| OK
Line 316: Line 316:
 
| OK
 
| OK
 
|-
 
|-
−
| yubikiri-genman
+
| YUBIKIRI-GENMAN
 
| OK
 
| OK
 
| OK
 
| OK
Line 323: Line 323:
 
| OK
 
| OK
 
|-
 
|-
−
| something fun
+
| Something Fun
 
| OK
 
| OK
 
| OK
 
| OK
Line 342: Line 342:
 
5 osu!taiko mappers evaluated submissions across 5 songs: '''Silver Bullet''', '''A fight in Tunisia''', '''YUBIKIRI-GENMAN''', '''Something Fun''', and '''closer to you'''. Each judge was assigned 3 songs based on their familiarity and interest of the songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models on 4 categories (Musical Representation, Creativity, Gameplay, and Human-Likeness) using an integer scale from 1 to 10.
 
5 osu!taiko mappers evaluated submissions across 5 songs: '''Silver Bullet''', '''A fight in Tunisia''', '''YUBIKIRI-GENMAN''', '''Something Fun''', and '''closer to you'''. Each judge was assigned 3 songs based on their familiarity and interest of the songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models on 4 categories (Musical Representation, Creativity, Gameplay, and Human-Likeness) using an integer scale from 1 to 10.
  
−
We follow [https://osu.ppy.sh/wiki/en/Contests/Monthly_Beatmapping_Contest Monthly Beatmapping Contest] and adopt '''within-song normalization''' for per-song report. For each category, judge X's final score = (judge X's raw score - judge X's average raw score) / judge X's standard deviation within that song, and a submission's score is the sum of each judge's final score.
+
We adopt the judge-score normalization approach used by the [https://osu.ppy.sh/wiki/en/Contests/Monthly_Beatmapping_Contest Monthly Beatmapping Contest] and report both '''within-song normalization''' and '''across-song normalization''' scores. For each category, a judge’s normalized score is calculated as '''(raw rating − judge’s mean rating) / judge’s population standard deviation'''. Within-song normalization uses the four submissions evaluated by that judge for the same song; across-song normalization uses all 12 submissions assigned to that judge across three songs. A submission’s category score is the '''sum of its three judges’ normalized scores'''. '''Combined Quality''' is calculated by first summing each judge’s raw ratings for Musical Representation, Creativity, and Gameplay, then normalizing these totals using the same procedure and summing the three judges’ normalized scores. '''Human-Likeness is evaluated separately and excluded from Combined Quality'''.
  
 
=== Silver Bullet ===
 
=== Silver Bullet ===
 +
 +
{| class="wikitable"
 +
! rowspan="2" | Model
 +
! colspan="2" | Musical Representation
 +
! colspan="2" | Creativity
 +
! colspan="2" | Gameplay
 +
! colspan="2" | Combined Quality
 +
! colspan="2" | Human-Likeness
 +
|-
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
|-
 +
| DOSU
 +
| '''1.5872''' || 1.9303
 +
| −2.6304 || −0.6119
 +
| '''2.0099''' || '''1.2101'''
 +
| −0.9236 || 0.8593
 +
| −0.4552 || −0.4745
 +
|-
 +
| FumenLab
 +
| 0.3005 || 1.8878
 +
| 0.9401 || 1.1426
 +
| −3.5863 || −2.9908
 +
| −1.5011 || −0.0563
 +
| −1.9478 || −1.6081
 +
|-
 +
| HiTaiko
 +
| 1.2778 || '''2.6483'''
 +
| '''1.8629''' || '''2.8452'''
 +
| '''2.0099''' || '''1.2101'''
 +
| '''3.6945''' || '''2.5621'''
 +
| '''4.1634''' || '''3.0147'''
 +
|-
 +
| MALer
 +
| −3.1656 || −0.6386
 +
| −0.1726 || 0.6706
 +
| −0.4335 || −1.0840
 +
| −1.2698 || −0.3204
 +
| −1.7604 || −1.4425
 +
|}
  
 
{| class="wikitable"
 
{| class="wikitable"
 
! Model
 
! Model
−
! Musical Representation
 
−
! Creativity
 
−
! Gameplay
 
−
! Human-Likeness
 
 
! Qualitative Feedback
 
! Qualitative Feedback
 
|-
 
|-
 
| DOSU
 
| DOSU
−
| '''1.5872'''
 
−
| −2.6304
 
−
| '''2.0099'''
 
−
| −0.4552
 
 
| Musical representation drew mixed assessments. Oni and Muzukashii showed effective sound selection, but Inner Oni was conservative early and uneven near the ending. Inconsistent special-object use and variable-BPM handling weakened coherence across difficulties.
 
| Musical representation drew mixed assessments. Oni and Muzukashii showed effective sound selection, but Inner Oni was conservative early and uneven near the ending. Inconsistent special-object use and variable-BPM handling weakened coherence across difficulties.
 
|-
 
|-
 
| FumenLab
 
| FumenLab
−
| 0.3005
 
−
| 0.9401
 
−
| −3.5863
 
−
| −1.9478
 
 
| Oni and Inner Oni showed promising sound selection, but awkward bursts and near-overlapping notes compromised gameplay near the ending. Kantan was excessively sparse, while intermediate difficulties were comparatively plain. Repeated musical phrases received inconsistent treatment.
 
| Oni and Inner Oni showed promising sound selection, but awkward bursts and near-overlapping notes compromised gameplay near the ending. Kantan was excessively sparse, while intermediate difficulties were comparatively plain. Repeated musical phrases received inconsistent treatment.
 
|-
 
|-
 
| HiTaiko
 
| HiTaiko
−
| 1.2778
 
−
| '''1.8629'''
 
−
| '''2.0099'''
 
−
| '''4.1634'''
 
 
| Coherent musical representation and expressive color patterns were recurring strengths. Inner Oni made quieter passages disproportionately dense, while Oni was comparatively restrained. Futsuu and Muzukashii showed promising structure, although assessments of the lower-tier ending differed. Finisher placement remained inconsistent.
 
| Coherent musical representation and expressive color patterns were recurring strengths. Inner Oni made quieter passages disproportionately dense, while Oni was comparatively restrained. Futsuu and Muzukashii showed promising structure, although assessments of the lower-tier ending differed. Finisher placement remained inconsistent.
 
|-
 
|-
 
| MALer
 
| MALer
−
| −3.1656
 
−
| −0.1726
 
−
| −0.4335
 
−
| −1.7604
 
 
| Imaginative Oni patterns and distinctive sectioning were offset by rhythms that often felt disconnected from the music. Inner Oni contained timing and overlap problems; unusual subdivisions in Muzukashii and questionable sound selection in Futsuu weakened the difficulty spread.
 
| Imaginative Oni patterns and distinctive sectioning were offset by rhythms that often felt disconnected from the music. Inner Oni contained timing and overlap problems; unusual subdivisions in Muzukashii and questionable sound selection in Futsuu weakened the difficulty spread.
 
|}
 
|}
  
 
=== A fight in Tunisia ===
 
=== A fight in Tunisia ===
 +
 +
{| class="wikitable"
 +
! rowspan="2" | Model
 +
! colspan="2" | Musical Representation
 +
! colspan="2" | Creativity
 +
! colspan="2" | Gameplay
 +
! colspan="2" | Combined Quality
 +
! colspan="2" | Human-Likeness
 +
|-
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
|-
 +
| DOSU
 +
| −4.7037 || −5.3766
 +
| −3.6090 || −4.6191
 +
| 0.4677 || −0.1418
 +
| −4.0148 || −4.1326
 +
| −1.5881 || −2.3767
 +
|-
 +
| FumenLab
 +
| 0.2720 || −1.1810
 +
| −0.2284 || −1.7695
 +
| −2.4438 || −2.4506
 +
| −0.7583 || −2.0492
 +
| −0.1690 || −0.8568
 +
|-
 +
| HiTaiko
 +
| 2.0747 || 0.3217
 +
| 0.9776 || −0.6639
 +
| '''2.2563''' || '''1.3928'''
 +
| 2.0694 || 0.1551
 +
| '''2.3546''' || '''1.7015'''
 +
|-
 +
| MALer
 +
| '''2.3570''' || '''0.9179'''
 +
| '''2.8598''' || '''1.3301'''
 +
| −0.2803 || −0.2017
 +
| '''2.7037''' || '''0.7954'''
 +
| −0.5975 || −1.2836
 +
|}
  
 
{| class="wikitable"
 
{| class="wikitable"
 
! Model
 
! Model
−
! Musical Representation
 
−
! Creativity
 
−
! Gameplay
 
−
! Human-Likeness
 
 
! Qualitative Feedback
 
! Qualitative Feedback
 
|-
 
|-
 
| DOSU
 
| DOSU
−
| −4.7037
 
−
| −3.6090
 
−
| 0.4677
 
−
| −1.5881
 
 
| The mapping frequently missed prominent musical events across all difficulties. Conservative patterns supported playability, but omitted piano passages and notes in musically empty sections weakened representation. Isolated Inner Oni patterns showed potential without fitting the song.
 
| The mapping frequently missed prominent musical events across all difficulties. Conservative patterns supported playability, but omitted piano passages and notes in musically empty sections weakened representation. Isolated Inner Oni patterns showed potential without fitting the song.
 
|-
 
|-
 
| FumenLab
 
| FumenLab
−
| 0.2720
 
−
| −0.2284
 
−
| −2.4438
 
−
| −0.1690
 
 
| Awkward bursts, excessive local density and neglected piano passages weakened musical clarity. Kantan–Muzukashii lacked coherent sound selection; Oni was sometimes too demanding for its tier. Inner Oni was relatively stronger but remained rhythmically awkward.
 
| Awkward bursts, excessive local density and neglected piano passages weakened musical clarity. Kantan–Muzukashii lacked coherent sound selection; Oni was sometimes too demanding for its tier. Inner Oni was relatively stronger but remained rhythmically awkward.
 
|-
 
|-
 
| HiTaiko
 
| HiTaiko
−
| 2.0747
 
−
| 0.9776
 
−
| '''2.2563'''
 
−
| '''2.3546'''
 
 
| Oni stood out for coherent sound selection and structure. Inner Oni was more conservative and relied on abrupt doublets, while Kantan–Muzukashii often omitted material or mixed musical layers inconsistently. Piano representation and repeated special-object placement remained weaknesses.
 
| Oni stood out for coherent sound selection and structure. Inner Oni was more conservative and relied on abrupt doublets, while Kantan–Muzukashii often omitted material or mixed musical layers inconsistently. Piano representation and repeated special-object placement remained weaknesses.
 
|-
 
|-
 
| MALer
 
| MALer
−
| '''2.3570'''
 
−
| '''2.8598'''
 
−
| −0.2803
 
−
| −0.5975
 
 
| Expressive color patterns attracted praise, but overall assessments differed. Complex snapping and inconsistent musical-layer selection reduced readability, particularly in Kantan–Muzukashii. Inner Oni offered some musical representation, although its structure and rhythmic emphasis remained uneven.
 
| Expressive color patterns attracted praise, but overall assessments differed. Complex snapping and inconsistent musical-layer selection reduced readability, particularly in Kantan–Muzukashii. Inner Oni offered some musical representation, although its structure and rhythmic emphasis remained uneven.
 
|}
 
|}
  
 
=== YUBIKIRI-GENMAN ===
 
=== YUBIKIRI-GENMAN ===
 +
 +
{| class="wikitable"
 +
! rowspan="2" | Model
 +
! colspan="2" | Musical Representation
 +
! colspan="2" | Creativity
 +
! colspan="2" | Gameplay
 +
! colspan="2" | Combined Quality
 +
! colspan="2" | Human-Likeness
 +
|-
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
|-
 +
| DOSU
 +
| −1.6143 || −1.9648
 +
| '''0.6706''' || −0.1198
 +
| −1.2916 || −1.3750
 +
| −1.0346 || −0.8634
 +
| −1.5202 || −1.6127
 +
|-
 +
| FumenLab
 +
| 1.3093 || '''1.3220'''
 +
| 0.3325 || '''1.5965'''
 +
| '''2.5865''' || '''2.5448'''
 +
| '''1.6890''' || '''2.3056'''
 +
| 0.6172 || 0.7949
 +
|-
 +
| HiTaiko
 +
| '''1.3093''' || −0.1391
 +
| −0.2970 || −1.0083
 +
| 1.8815 || 1.2769
 +
| 1.3451 || −0.2158
 +
| '''4.0095''' || '''2.8074'''
 +
|-
 +
| MALer
 +
| −1.0043 || −1.5591
 +
| −0.7060 || −1.4543
 +
| −3.1764 || −3.0761
 +
| −1.9995 || −2.1248
 +
| −3.1065 || −2.6061
 +
|}
 +
 +
''FumenLab and HiTaiko’s within-song Musical Representation scores round to the same value. HiTaiko has the slightly higher unrounded score.''
  
 
{| class="wikitable"
 
{| class="wikitable"
 
! Model
 
! Model
−
! Musical Representation
 
−
! Creativity
 
−
! Gameplay
 
−
! Human-Likeness
 
 
! Qualitative Feedback
 
! Qualitative Feedback
 
|-
 
|-
 
| DOSU
 
| DOSU
−
| −1.6143
 
−
| '''0.6706'''
 
−
| −1.2916
 
−
| −1.5202
 
 
| Excessive rhythmic improvisation disrupted the song’s gentle character. Muzukashii, Oni and Inner Oni often emphasized verses over choruses, weakening emotional progression. Kantan’s simpler rhythms were more acceptable, but consistency, object placement and musical correspondence remained concerns.
 
| Excessive rhythmic improvisation disrupted the song’s gentle character. Muzukashii, Oni and Inner Oni often emphasized verses over choruses, weakening emotional progression. Kantan’s simpler rhythms were more acceptable, but consistency, object placement and musical correspondence remained concerns.
 
|-
 
|-
 
| FumenLab
 
| FumenLab
−
| 1.3093
 
−
| 0.3325
 
−
| '''2.5865'''
 
−
| 0.6172
 
 
| Assessments of musical coherence differed. Muzukashii and Oni sometimes followed the vocals effectively, but repeated phrases received inconsistent treatment. Inner Oni’s complex climax disrupted the gentle atmosphere. Kantan was sparse, and hitsound volume and effect choices required refinement.
 
| Assessments of musical coherence differed. Muzukashii and Oni sometimes followed the vocals effectively, but repeated phrases received inconsistent treatment. Inner Oni’s complex climax disrupted the gentle atmosphere. Kantan was sparse, and hitsound volume and effect choices required refinement.
 
|-
 
|-
 
| HiTaiko
 
| HiTaiko
−
| '''1.3093'''
 
−
| −0.2970
 
−
| 1.8815
 
−
| '''4.0095'''
 
 
| Oni and Inner Oni showed effective vocal following and a convincing build toward the final chorus, although some feedback found them too conservative. Muzukashii’s density allocation was inconsistent; Kantan and Futsuu offered little content. Special-object use and omissions remained concerns.
 
| Oni and Inner Oni showed effective vocal following and a convincing build toward the final chorus, although some feedback found them too conservative. Muzukashii’s density allocation was inconsistent; Kantan and Futsuu offered little content. Special-object use and omissions remained concerns.
 
|-
 
|-
 
| MALer
 
| MALer
−
| −1.0043
 
−
| −0.7060
 
−
| −3.1764
 
−
| −3.1065
 
 
| Irregular snapping and abrupt density changes were poorly suited to the gentle vocal/piano arrangement. Muzukashii, Oni and Inner Oni contained isolated difficulty spikes, with Oni reportedly harder than Inner Oni. Some color patterns and brief rhythm choices worked but were not sustained.
 
| Irregular snapping and abrupt density changes were poorly suited to the gentle vocal/piano arrangement. Muzukashii, Oni and Inner Oni contained isolated difficulty spikes, with Oni reportedly harder than Inner Oni. Some color patterns and brief rhythm choices worked but were not sustained.
 
|}
 
|}
−
''Musical Representation scores for FumenLab and HiTaiko round to the same value; HiTaiko has the higher unrounded score.''
 
  
 
=== Something Fun ===
 
=== Something Fun ===
 +
 +
{| class="wikitable"
 +
! rowspan="2" | Model
 +
! colspan="2" | Musical Representation
 +
! colspan="2" | Creativity
 +
! colspan="2" | Gameplay
 +
! colspan="2" | Combined Quality
 +
! colspan="2" | Human-Likeness
 +
|-
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
|-
 +
| DOSU
 +
| −4.1362 || −4.6148
 +
| −3.0094 || −2.8557
 +
| −2.8967 || −3.2500
 +
| −3.5170 || −3.8992
 +
| −3.9851 || −3.5107
 +
|-
 +
| FumenLab
 +
| 1.7808 || 0.8973
 +
| 2.0497 || 1.3931
 +
| '''2.0645''' || '''1.7150'''
 +
| '''2.3657''' || 1.5878
 +
| '''2.1521''' || 1.6169
 +
|-
 +
| HiTaiko
 +
| −1.0707 || −1.7031
 +
| −1.3013 || −0.9950
 +
| −0.3393 || −1.1085
 +
| −1.1090 || −1.4505
 +
| 0.6620 || 0.4912
 +
|-
 +
| MALer
 +
| '''3.4261''' || '''2.4004'''
 +
| '''2.2610''' || '''2.7960'''
 +
| 1.1716 || 0.9334
 +
| 2.2602 || '''2.0269'''
 +
| 1.1710 || '''2.1783'''
 +
|}
  
 
{| class="wikitable"
 
{| class="wikitable"
 
! Model
 
! Model
−
! Musical Representation
 
−
! Creativity
 
−
! Gameplay
 
−
! Human-Likeness
 
 
! Qualitative Feedback
 
! Qualitative Feedback
 
|-
 
|-
 
| DOSU
 
| DOSU
−
| −4.1362
 
−
| −3.0094
 
−
| −2.8967
 
−
| −3.9851
 
 
| Rhythmic choices and sound selection were poorly aligned with the music. Missing notes, excessive finishers and misplaced spinners weakened consistency. Both Muzukashii and Inner Oni contained passages whose musical intent was unclear.
 
| Rhythmic choices and sound selection were poorly aligned with the music. Missing notes, excessive finishers and misplaced spinners weakened consistency. Both Muzukashii and Inner Oni contained passages whose musical intent was unclear.
 
|-
 
|-
 
| FumenLab
 
| FumenLab
−
| 1.7808
 
−
| 2.0497
 
−
| '''2.0645'''
 
−
| '''2.1521'''
 
 
| The set was broadly serviceable, but repetitive color patterns and inconsistent finishers limited expressiveness. Kantan and Futsuu were weak, while some Inner Oni streams added unnecessary complexity. Isolated passages also raised timing concerns.
 
| The set was broadly serviceable, but repetitive color patterns and inconsistent finishers limited expressiveness. Kantan and Futsuu were weak, while some Inner Oni streams added unnecessary complexity. Isolated passages also raised timing concerns.
 
|-
 
|-
 
| HiTaiko
 
| HiTaiko
−
| −1.0707
 
−
| −1.3013
 
−
| −0.3393
 
−
| 0.6620
 
 
| Some patterns followed the melody effectively, but prolonged simple rhythms, questionable drumrolls and uneven sectional difficulty made the experience repetitive. Finisher placement lacked consistency; no particular difficulty tier emerged as a clear strength.
 
| Some patterns followed the melody effectively, but prolonged simple rhythms, questionable drumrolls and uneven sectional difficulty made the experience repetitive. Finisher placement lacked consistency; no particular difficulty tier emerged as a clear strength.
 
|-
 
|-
 
| MALer
 
| MALer
−
| '''3.4261'''
 
−
| '''2.2610'''
 
−
| 1.1716
 
−
| 1.1710
 
 
| Creative ideas produced sharply divided assessments. Futsuu omitted a conspicuous musical passage, while Inner Oni showed uneven density and timing problems. Kantan and Muzukashii had questionable rhythmic choices; Oni was comparatively more successful but appeared too easy for its label.
 
| Creative ideas produced sharply divided assessments. Futsuu omitted a conspicuous musical passage, while Inner Oni showed uneven density and timing problems. Kantan and Muzukashii had questionable rhythmic choices; Oni was comparatively more successful but appeared too easy for its label.
 
|}
 
|}
  
 
=== closer to you ===
 
=== closer to you ===
 +
 +
{| class="wikitable"
 +
! rowspan="2" | Model
 +
! colspan="2" | Musical Representation
 +
! colspan="2" | Creativity
 +
! colspan="2" | Gameplay
 +
! colspan="2" | Combined Quality
 +
! colspan="2" | Human-Likeness
 +
|-
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
! Within-song
 +
! Across-song
 +
|-
 +
| DOSU
 +
| −0.5579 || 0.7297
 +
| 0.7683 || 1.3234
 +
| −0.7678 || 0.5162
 +
| −0.2182 || 0.8831
 +
| −0.5907 || 0.3679
 +
|-
 +
| FumenLab
 +
| '''3.3983''' || '''4.5471'''
 +
| '''4.2866''' || '''5.5175'''
 +
| '''4.4877''' || '''5.3224'''
 +
| '''4.4930''' || '''5.9265'''
 +
| '''3.5659''' || '''4.1415'''
 +
|-
 +
| HiTaiko
 +
| 1.6737 || 2.6624
 +
| −2.5275 || −2.2587
 +
| −0.1399 || 1.1821
 +
| −0.4174 || 0.6315
 +
| 1.0151 || 1.8349
 +
|-
 +
| MALer
 +
| −4.5141 || −3.0877
 +
| −2.5275 || −2.2587
 +
| −3.5800 || −1.6254
 +
| −3.8573 || −2.6216
 +
| −3.9903 || −3.1775
 +
|}
  
 
{| class="wikitable"
 
{| class="wikitable"
 
! Model
 
! Model
−
! Musical Representation
 
−
! Creativity
 
−
! Gameplay
 
−
! Human-Likeness
 
 
! Qualitative Feedback
 
! Qualitative Feedback
 
|-
 
|-
 
| DOSU
 
| DOSU
−
| −0.5579
 
−
| 0.7683
 
−
| −0.7678
 
−
| −0.5907
 
 
| Rhythmic improvisation showed promise, and Futsuu conveyed a pleasing feel. However, inconsistent finishers and uneven density weakened progression: Oni omitted the opening, while Inner Oni used fewer streams than Muzukashii.
 
| Rhythmic improvisation showed promise, and Futsuu conveyed a pleasing feel. However, inconsistent finishers and uneven density weakened progression: Oni omitted the opening, while Inner Oni used fewer streams than Muzukashii.
 
|-
 
|-
 
| FumenLab
 
| FumenLab
−
| '''3.3983'''
 
−
| '''4.2866'''
 
−
| '''4.4877'''
 
−
| '''3.5659'''
 
 
| Strong structure and effective rhythmic improvisation were recurring positives, with Inner Oni receiving particular praise. Color-pattern variety, consistency and timing-point use still needed refinement, but the set conveyed a comparatively convincing mapping approach.
 
| Strong structure and effective rhythmic improvisation were recurring positives, with Inner Oni receiving particular praise. Color-pattern variety, consistency and timing-point use still needed refinement, but the set conveyed a comparatively convincing mapping approach.
 
|-
 
|-
 
| HiTaiko
 
| HiTaiko
−
| 1.6737
 
−
| −2.5275
 
−
| −0.1399
 
−
| 1.0151
 
 
| Patterns were coherent but repetitive, with little distinction between difficulty tiers. Even Inner Oni remained sparse and closer to a much easier difficulty. Drumroll use and the ending finisher were questionable choices for the song’s calm character.
 
| Patterns were coherent but repetitive, with little distinction between difficulty tiers. Even Inner Oni remained sparse and closer to a much easier difficulty. Drumroll use and the ending finisher were questionable choices for the song’s calm character.
 
|-
 
|-
 
| MALer
 
| MALer
−
| −4.5141
 
−
| −2.5275
 
−
| −3.5800
 
−
| −3.9903
 
 
| Missing sections, timing errors and inconsistent musical logic undermined the spread. Muzukashii was reportedly the hardest tier, while easier difficulties contained unexplained breaks. Inner Oni’s awkward fast bursts did not resolve the overall lack of musical coverage.
 
| Missing sections, timing errors and inconsistent musical logic undermined the spread. Muzukashii was reportedly the hardest tier, while easier difficulties contained unexplained breaks. Inner Oni’s awkward fast bursts did not resolve the overall lack of musical coverage.
 
|}
 
|}
  
−
Qualitative feedback suggested distinct tendencies across the evaluated models. HiTaiko often favored coherent but conservative arrangements, while MALer explored more unusual rhythms with less consistent musical fit. FumenLab demonstrated effective improvisation and higher-difficulty patterning, but struggled with simplification and technical consistency. DOSU showed isolated strengths in sound selection, alongside substantial variation in musical correspondence and structural coherence. Across models, adapting musical ideas to lower difficulties remained a recurring challenge.
+
== Community Evaluation Results ==
  
−
== Community Evaluation Results ==
+
X osu!taiko players evaluated submissions on one test song: '''Himitsu feat. KAFU'''. Players assessed submissions from all 4 models, plus one beatmap created by a human mapper, on 4 categories (Musical Representation, Creativity, Gameplay, and Human-Likeness) using an integer scale from 1 to 10. They were not informed that any of the beatmaps were human-made. Every participant selected their preferred difficulty level and evaluated only the beatmaps of that difficulty.
  
 +
=== Kantan - 2 participants ===
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
Line 594: Line 736:
 
|
 
|
 
|}
 
|}
 +
 +
=== Futsuu - 2 participants ===
 +
 +
=== Muzukashii - 2 participants ===
 +
 +
=== Inner Oni - 5 participants ===

Latest revision as of 11:46, 14 October 2026

Submissions

Submission Title PDF Authors
DOSU DOSU: DIFFICULTY-CONDITIONED ON-GRID SEQUENCE UNMASKING FOR OSU!TAIKO CHART GENERATION PDF Junyoung Koh, Hao-Wen Dong
FumenLab FUMENLAB: PLAN-CONDITIONED JOINT GENERATION AND REFINEMENT OF OSU!TAIKO CHARTS PDF Ryo Mitoma
HiTaiko HiTaiko: Hierarchical osu!Taiko Chart Generation via Metrical Candidate Selection PDF Sungjung Park
MALer RHYTHM GAME CHART GENERATION WITH PRE-TRAINED MUSIC TRANSCRIPTION MODEL PDF Sojeong An, Chris Donahue, Dasaem Jeong

Test Set

Song Selection

Title Artist Genre Language Tags BPM
Silver Bullet Ennnn Electronic Instrumental breakcore, neurofunk, tech 89-379 (mostly 175)
A fight in Tunisia Sobrem & NIWASHI Jazz Instrumental piano 148
Something Fun JOYLESS Rock English male vocal 194
closer to you (than ever before) xiiiac13 Electronic Instrumental lo-fi 110
YUBIKIRI-GENMAN Mili Pop Japanese j-pop, progressive, female vocal 59-146 (mostly 146)
Himitsu feat. KAFU MIMI Pop Japanese j-pop, vocaloid 168

Default Settings

Parameter Kantan Futsuu Muzukashii Oni Inner Oni
Overall Difficulty 3 4 5 5 6
HP Drain Rate 8 7 6 5 5

Results

Algorithmic Evaluation Results

120 beatmaps were checked using AiMod across 4 models, 6 songs, and 5 difficulties.

Difficulty ordering warnings:

  • DOSU - closer to you - Inner Oni < Oni
  • MALer - YUBIKIRI-GENMAN - Inner Oni < Oni
  • MALer - closer to you - Kantan < Oni < Futsuu < Inner Oni < Muzukashii

DOSU

Song Kantan Futsuu Muzukashii Oni Inner Oni
Silver Bullet OK OK OK OK OK
A fight in Tunisia OK OK OK OK OK
Himitsu feat. KAFU OK OK OK OK OK
YUBIKIRI-GENMAN OK OK OK OK OK
Something Fun OK OK OK OK OK
closer to you OK OK OK OK OK

FumenLab

Song Kantan Futsuu Muzukashii Oni Inner Oni
Silver Bullet Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells Objects <10ms apart and concurrent objects Objects <10ms apart and concurrent objects
A fight in Tunisia Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells OK OK OK
Himitsu feat. KAFU OK OK OK OK Minor snapping issue on drumrolls and swells
YUBIKIRI-GENMAN OK Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells OK OK
Something Fun OK OK OK OK OK
closer to you OK OK OK OK OK

HiTaiko

Song Kantan Futsuu Muzukashii Oni Inner Oni
Silver Bullet Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells OK
A fight in Tunisia OK OK OK OK OK
Himitsu feat. KAFU OK OK OK OK OK
YUBIKIRI-GENMAN OK OK OK Minor snapping issue on drumrolls and swells Minor snapping issue on drumrolls and swells
Something Fun OK OK OK OK OK
closer to you Unused audio at the end Unused audio at the end Unused audio at the end Unused audio at the end OK

MALer

Song Kantan Futsuu Muzukashii Oni Inner Oni
Silver Bullet OK OK OK OK OK
A fight in Tunisia OK OK OK OK OK
Himitsu feat. KAFU OK OK OK OK OK
YUBIKIRI-GENMAN OK OK OK OK OK
Something Fun OK OK OK OK OK
closer to you Unused audio at the end Unused audio at the end Unused audio at the end Unused audio at the end Unused audio at the end

Expert Evaluation Results

5 osu!taiko mappers evaluated submissions across 5 songs: Silver Bullet, A fight in Tunisia, YUBIKIRI-GENMAN, Something Fun, and closer to you. Each judge was assigned 3 songs based on their familiarity and interest of the songs, and each song was evaluated by 3 judges. For every assigned song, judges assessed submissions from all 4 models on 4 categories (Musical Representation, Creativity, Gameplay, and Human-Likeness) using an integer scale from 1 to 10.

We adopt the judge-score normalization approach used by the Monthly Beatmapping Contest and report both within-song normalization and across-song normalization scores. For each category, a judge’s normalized score is calculated as (raw rating − judge’s mean rating) / judge’s population standard deviation. Within-song normalization uses the four submissions evaluated by that judge for the same song; across-song normalization uses all 12 submissions assigned to that judge across three songs. A submission’s category score is the sum of its three judges’ normalized scores. Combined Quality is calculated by first summing each judge’s raw ratings for Musical Representation, Creativity, and Gameplay, then normalizing these totals using the same procedure and summing the three judges’ normalized scores. Human-Likeness is evaluated separately and excluded from Combined Quality.

Silver Bullet

Model Musical Representation Creativity Gameplay Combined Quality Human-Likeness
Within-song Across-song Within-song Across-song Within-song Across-song Within-song Across-song Within-song Across-song
DOSU 1.5872 1.9303 −2.6304 −0.6119 2.0099 1.2101 −0.9236 0.8593 −0.4552 −0.4745
FumenLab 0.3005 1.8878 0.9401 1.1426 −3.5863 −2.9908 −1.5011 −0.0563 −1.9478 −1.6081
HiTaiko 1.2778 2.6483 1.8629 2.8452 2.0099 1.2101 3.6945 2.5621 4.1634 3.0147
MALer −3.1656 −0.6386 −0.1726 0.6706 −0.4335 −1.0840 −1.2698 −0.3204 −1.7604 −1.4425
Model Qualitative Feedback
DOSU Musical representation drew mixed assessments. Oni and Muzukashii showed effective sound selection, but Inner Oni was conservative early and uneven near the ending. Inconsistent special-object use and variable-BPM handling weakened coherence across difficulties.
FumenLab Oni and Inner Oni showed promising sound selection, but awkward bursts and near-overlapping notes compromised gameplay near the ending. Kantan was excessively sparse, while intermediate difficulties were comparatively plain. Repeated musical phrases received inconsistent treatment.
HiTaiko Coherent musical representation and expressive color patterns were recurring strengths. Inner Oni made quieter passages disproportionately dense, while Oni was comparatively restrained. Futsuu and Muzukashii showed promising structure, although assessments of the lower-tier ending differed. Finisher placement remained inconsistent.
MALer Imaginative Oni patterns and distinctive sectioning were offset by rhythms that often felt disconnected from the music. Inner Oni contained timing and overlap problems; unusual subdivisions in Muzukashii and questionable sound selection in Futsuu weakened the difficulty spread.

A fight in Tunisia

Model Musical Representation Creativity Gameplay Combined Quality Human-Likeness
Within-song Across-song Within-song Across-song Within-song Across-song Within-song Across-song Within-song Across-song
DOSU −4.7037 −5.3766 −3.6090 −4.6191 0.4677 −0.1418 −4.0148 −4.1326 −1.5881 −2.3767
FumenLab 0.2720 −1.1810 −0.2284 −1.7695 −2.4438 −2.4506 −0.7583 −2.0492 −0.1690 −0.8568
HiTaiko 2.0747 0.3217 0.9776 −0.6639 2.2563 1.3928 2.0694 0.1551 2.3546 1.7015
MALer 2.3570 0.9179 2.8598 1.3301 −0.2803 −0.2017 2.7037 0.7954 −0.5975 −1.2836
Model Qualitative Feedback
DOSU The mapping frequently missed prominent musical events across all difficulties. Conservative patterns supported playability, but omitted piano passages and notes in musically empty sections weakened representation. Isolated Inner Oni patterns showed potential without fitting the song.
FumenLab Awkward bursts, excessive local density and neglected piano passages weakened musical clarity. Kantan–Muzukashii lacked coherent sound selection; Oni was sometimes too demanding for its tier. Inner Oni was relatively stronger but remained rhythmically awkward.
HiTaiko Oni stood out for coherent sound selection and structure. Inner Oni was more conservative and relied on abrupt doublets, while Kantan–Muzukashii often omitted material or mixed musical layers inconsistently. Piano representation and repeated special-object placement remained weaknesses.
MALer Expressive color patterns attracted praise, but overall assessments differed. Complex snapping and inconsistent musical-layer selection reduced readability, particularly in Kantan–Muzukashii. Inner Oni offered some musical representation, although its structure and rhythmic emphasis remained uneven.

YUBIKIRI-GENMAN

Model Musical Representation Creativity Gameplay Combined Quality Human-Likeness
Within-song Across-song Within-song Across-song Within-song Across-song Within-song Across-song Within-song Across-song
DOSU −1.6143 −1.9648 0.6706 −0.1198 −1.2916 −1.3750 −1.0346 −0.8634 −1.5202 −1.6127
FumenLab 1.3093 1.3220 0.3325 1.5965 2.5865 2.5448 1.6890 2.3056 0.6172 0.7949
HiTaiko 1.3093 −0.1391 −0.2970 −1.0083 1.8815 1.2769 1.3451 −0.2158 4.0095 2.8074
MALer −1.0043 −1.5591 −0.7060 −1.4543 −3.1764 −3.0761 −1.9995 −2.1248 −3.1065 −2.6061

FumenLab and HiTaiko’s within-song Musical Representation scores round to the same value. HiTaiko has the slightly higher unrounded score.

Model Qualitative Feedback
DOSU Excessive rhythmic improvisation disrupted the song’s gentle character. Muzukashii, Oni and Inner Oni often emphasized verses over choruses, weakening emotional progression. Kantan’s simpler rhythms were more acceptable, but consistency, object placement and musical correspondence remained concerns.
FumenLab Assessments of musical coherence differed. Muzukashii and Oni sometimes followed the vocals effectively, but repeated phrases received inconsistent treatment. Inner Oni’s complex climax disrupted the gentle atmosphere. Kantan was sparse, and hitsound volume and effect choices required refinement.
HiTaiko Oni and Inner Oni showed effective vocal following and a convincing build toward the final chorus, although some feedback found them too conservative. Muzukashii’s density allocation was inconsistent; Kantan and Futsuu offered little content. Special-object use and omissions remained concerns.
MALer Irregular snapping and abrupt density changes were poorly suited to the gentle vocal/piano arrangement. Muzukashii, Oni and Inner Oni contained isolated difficulty spikes, with Oni reportedly harder than Inner Oni. Some color patterns and brief rhythm choices worked but were not sustained.

Something Fun

Model Musical Representation Creativity Gameplay Combined Quality Human-Likeness
Within-song Across-song Within-song Across-song Within-song Across-song Within-song Across-song Within-song Across-song
DOSU −4.1362 −4.6148 −3.0094 −2.8557 −2.8967 −3.2500 −3.5170 −3.8992 −3.9851 −3.5107
FumenLab 1.7808 0.8973 2.0497 1.3931 2.0645 1.7150 2.3657 1.5878 2.1521 1.6169
HiTaiko −1.0707 −1.7031 −1.3013 −0.9950 −0.3393 −1.1085 −1.1090 −1.4505 0.6620 0.4912
MALer 3.4261 2.4004 2.2610 2.7960 1.1716 0.9334 2.2602 2.0269 1.1710 2.1783
Model Qualitative Feedback
DOSU Rhythmic choices and sound selection were poorly aligned with the music. Missing notes, excessive finishers and misplaced spinners weakened consistency. Both Muzukashii and Inner Oni contained passages whose musical intent was unclear.
FumenLab The set was broadly serviceable, but repetitive color patterns and inconsistent finishers limited expressiveness. Kantan and Futsuu were weak, while some Inner Oni streams added unnecessary complexity. Isolated passages also raised timing concerns.
HiTaiko Some patterns followed the melody effectively, but prolonged simple rhythms, questionable drumrolls and uneven sectional difficulty made the experience repetitive. Finisher placement lacked consistency; no particular difficulty tier emerged as a clear strength.
MALer Creative ideas produced sharply divided assessments. Futsuu omitted a conspicuous musical passage, while Inner Oni showed uneven density and timing problems. Kantan and Muzukashii had questionable rhythmic choices; Oni was comparatively more successful but appeared too easy for its label.

closer to you

Model Musical Representation Creativity Gameplay Combined Quality Human-Likeness
Within-song Across-song Within-song Across-song Within-song Across-song Within-song Across-song Within-song Across-song
DOSU −0.5579 0.7297 0.7683 1.3234 −0.7678 0.5162 −0.2182 0.8831 −0.5907 0.3679
FumenLab 3.3983 4.5471 4.2866 5.5175 4.4877 5.3224 4.4930 5.9265 3.5659 4.1415
HiTaiko 1.6737 2.6624 −2.5275 −2.2587 −0.1399 1.1821 −0.4174 0.6315 1.0151 1.8349
MALer −4.5141 −3.0877 −2.5275 −2.2587 −3.5800 −1.6254 −3.8573 −2.6216 −3.9903 −3.1775
Model Qualitative Feedback
DOSU Rhythmic improvisation showed promise, and Futsuu conveyed a pleasing feel. However, inconsistent finishers and uneven density weakened progression: Oni omitted the opening, while Inner Oni used fewer streams than Muzukashii.
FumenLab Strong structure and effective rhythmic improvisation were recurring positives, with Inner Oni receiving particular praise. Color-pattern variety, consistency and timing-point use still needed refinement, but the set conveyed a comparatively convincing mapping approach.
HiTaiko Patterns were coherent but repetitive, with little distinction between difficulty tiers. Even Inner Oni remained sparse and closer to a much easier difficulty. Drumroll use and the ending finisher were questionable choices for the song’s calm character.
MALer Missing sections, timing errors and inconsistent musical logic undermined the spread. Muzukashii was reportedly the hardest tier, while easier difficulties contained unexplained breaks. Inner Oni’s awkward fast bursts did not resolve the overall lack of musical coverage.

Community Evaluation Results

X osu!taiko players evaluated submissions on one test song: Himitsu feat. KAFU. Players assessed submissions from all 4 models, plus one beatmap created by a human mapper, on 4 categories (Musical Representation, Creativity, Gameplay, and Human-Likeness) using an integer scale from 1 to 10. They were not informed that any of the beatmaps were human-made. Every participant selected their preferred difficulty level and evaluated only the beatmaps of that difficulty.

Kantan - 2 participants

Mapper Musical Representation Creativity Gameplay Overall Quality Score Human-Likeness Qualitative Feedback
DOSU
FumenLab
HiTaiko
MALer
Amasugi (human mapper)

Futsuu - 2 participants

Muzukashii - 2 participants

Inner Oni - 5 participants

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2026:Rhythm_Game_Chart_Generation_Results&oldid=15180"