Difference between revisions of "2026:Lyrics Transcription Results"
(Create results page ALT) |
(→Submissions: PDF links (abstracts archived on futuremirex.com)) |
||
| Line 10: | Line 10: | ||
|- | |- | ||
| Frangiadaki et al. | | Frangiadaki et al. | ||
| − | | | + | | [http://futuremirex.com/portal/wp-content/uploads/2026/lyrics-transcription/Frangiadaki.pdf PDF] |
| NA | | NA | ||
| ~132.15B | | ~132.15B | ||
| Line 17: | Line 17: | ||
|- | |- | ||
| UFAL-PMCG-Whisper | | UFAL-PMCG-Whisper | ||
| − | | | + | | [http://futuremirex.com/portal/wp-content/uploads/2026/lyrics-transcription/UFAL-PMCG-Whisper.pdf PDF] |
| NA | | NA | ||
| 1.55B | | 1.55B | ||
| Line 24: | Line 24: | ||
|- | |- | ||
| UFAL-PMCG-Qwen-ASR | | UFAL-PMCG-Qwen-ASR | ||
| − | | | + | | [http://futuremirex.com/portal/wp-content/uploads/2026/lyrics-transcription/UFAL-PMCG-Qwen-ASR.pdf PDF] |
| NA | | NA | ||
| 3.6B | | 3.6B | ||
| Line 31: | Line 31: | ||
|- | |- | ||
| UFAL-PMCG-Whisper+Qwen | | UFAL-PMCG-Whisper+Qwen | ||
| − | | | + | | [http://futuremirex.com/portal/wp-content/uploads/2026/lyrics-transcription/UFAL-PMCG-Whisper-Qwen.pdf PDF] |
| NA | | NA | ||
| 11.8B | | 11.8B | ||
| Line 38: | Line 38: | ||
|- | |- | ||
| UFAL-PMCG-Gemma | | UFAL-PMCG-Gemma | ||
| − | | | + | | [http://futuremirex.com/portal/wp-content/uploads/2026/lyrics-transcription/UFAL-PMCG-Gemma.pdf PDF] |
| NA | | NA | ||
| 11.6B | | 11.6B | ||
Latest revision as of 10:21, 14 October 2026
Submissions
| Team | Training size | Number of parameters | Training hours | Inference real-time factor | |
|---|---|---|---|---|---|
| Frangiadaki et al. | NA | ~132.15B | NA | 1.20 | |
| UFAL-PMCG-Whisper | NA | 1.55B | NA | 0.0386 | |
| UFAL-PMCG-Qwen-ASR | NA | 3.6B | NA | 0.0969 | |
| UFAL-PMCG-Whisper+Qwen | NA | 11.8B | NA | 0.1753 | |
| UFAL-PMCG-Gemma | NA | 11.6B | NA | 0.2111 |
Results
The results below are aggregate scores for each dataset. Lower is better for CER, WER, and WER'; higher is better for the F-scores. The formatting-token scores are F-scores for Punctuation (P), Parentheses (B), Line Breaks (L), and Section Breaks (S). Because all submissions were multilingual natively, we directly report complete results on the data.
Jam-ALT
| Team | CER | WER | WER' | F-score (P) | F-score (B) | F-score (L) | F-score (S) |
|---|---|---|---|---|---|---|---|
| Frangiadaki et al.(1) | 0.0881 | 0.1470 | 0.1913 | 0.5134 | 0.4932 | 0.8026 | 0.7587 |
| UFAL-PMCG-Whisper | 0.1991 | 0.2860 | 0.3317 | 0.4295 | NA | 0.7442 | 0.0000 |
| UFAL-PMCG-Qwen-ASR | 0.1598 | 0.2593 | 0.3190 | 0.2584 | NA | 0.5438 | NA |
| UFAL-PMCG-Whisper+Qwen | 0.1714 | 0.2622 | 0.3162 | 0.3210 | NA | 0.6764 | 0.2661 |
| UFAL-PMCG-Gemma | 0.1785 | 0.2806 | 0.3301 | 0.1447 | 0.0943 | 0.8118 | 0.2431 |
(1) The lyrics for the Spanish songs were pre-computed by the authors, because inference ran OOM on a V100-32GB.
MUSDB-ALT
| Team | CER | WER | WER' | F-score (P) | F-score (B) | F-score (L) | F-score (S) |
|---|---|---|---|---|---|---|---|
| Frangiadaki et al.(2) | 0.1163 | 0.1617 | 0.1970 | 0.2941 | 0.3190 | 0.8378 | 0.7485 |
| UFAL-PMCG-Whisper | 0.2031 | 0.2740 | 0.3126 | 0.2335 | NA | 0.7748 | 0.2159 |
| UFAL-PMCG-Qwen-ASR | 0.1569 | 0.2181 | 0.2697 | 0.1174 | NA | 0.5615 | NA |
| UFAL-PMCG-Whisper+Qwen | 0.1765 | 0.2362 | 0.2858 | 0.1563 | NA | 0.6896 | 0.2430 |
| UFAL-PMCG-Gemma | 0.2310 | 0.3123 | 0.3520 | 0.0680 | 0.1389 | 0.8138 | 0.2331 |
(2) The lyrics were pre-computed by the authors, because inference ran OOM on a V100-32GB.
JamendoLyrics++
JamendoLyrics++ was used as an additional evaluation set. It contains 80 English songs that do not overlap the previous 2 datasets. The lyrics however do not contain punctuation or capitalization, so only a reduced set of metrics is used.
| Team | CER | WER | F-score (L) | F-score (S) |
|---|---|---|---|---|
| Frangiadiaki et al. | 0.1263 | 0.1768 | 0.8250 | 0.7655 |
| UFAL-PMCG-Whisper | 0.1835 | 0.2625 | 0.7495 | 0.2500 |
| UFAL-PMCG-Qwen-ASR | 0.1391 | 0.2144 | 0.5562 | NA |
| UFAL-PMCG-Whisper+Qwen | 0.1747 | 0.2495 | 0.6698 | 0.2765 |
| UFAL-PMCG-Gemma | 0.2429 | 0.3353 | 0.7667 | 0.2269 |
