2026:Lyrics Transcription - Revision history

Alexandre DHooge: add real example

2026-06-29T03:34:38Z

add real example

Alexandre DHooge: fix list

2026-06-29T03:06:09Z

fix list

Alexandre DHooge: add audio format details

2026-06-29T03:05:21Z

add audio format details

Alexandre DHooge: add missing bib entries

2026-06-13T05:43:25Z

add missing bib entries

Alexandre DHooge: Draft version of the page

2026-06-13T05:33:06Z

Draft version of the page

New page

= Description =

This page describes the '''MIREX2026: Automatic Lyrics Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.
''It is strongly based upon the page of the 2025 ALT challenge.''

The task of Lyrics Transcription aims to identify the words from sung utterances, in the same way as in automatic speech recognition. This can be mathematically expressed as follows:

Prediction('''w''') = argmax P('''w'''|'''X''')

where '''w''' and '''X''' are the word and acoustic features respectively.

Ideally, the lyrics transcriber should return meaningful word sequences:

Prediction('''w''') = [ <w_1>, <w_2>, ..., <w_N> ]

Note that for this year's edition, the input will always be a polyphonic mix (singing voice + musical accompaniment). The submitted algorithms can include a source-separation step if needed.

'''Notice:''' We particularly encourage participations that build on the latest ALT approaches such as using ''pretrained audio foundation models'' or ''LLMs''.

= Evaluation =

This year's edition will include the following metrics for evaluation:

'''Word Error Rate''' (WER): the standard metric used in Automatic Speech Recognition.

WER = (S + I + D) / (C + S + D)

where;
C : correctly predicted words
S : substitution errors
I : insertion errors
D : deletion errors

'''Character Error Rate''' (CER): the above computation can also be done on the character level. This metric penalises the partially correctly predicted / incorrectly spelled words less than WER.

'''Case-Sensitive WER (WER')''': defined in [[#bib-cifka-2024|[1]]], this metric is computed as follows:

<math>
\mathrm{WER'} = \mathrm{WER} + \frac{E_\text{case}}{N}
</math>

where <math>N</math> is the total number of words in a ground-truth lyrics file, and <math>E_\text{case}</math> the number of "casing errors" i.e. words that differ in a ''case-sensitive'' setting like "city" and "City".

'''Punctuation, Parentheses, Line and Section Breaks''': also defined in [[#bib-cifka-2024|[1]]], we include a set of metrics to measure how well the proposed algorithms predict several formatting tokens. The tokens are classified into one of 5 types <math>T\in {W,P,B,L,S}</math> for Word (W), Punctuation (P), Parentheses (B), Line Breaks (L), and Section Breaks (S). Then, for each token type (except Word), we compute Precision, Recall, and F1-Score:

<math>
P_T = \frac{C_T}{C_T + S_T + I_T}
</math>

<math>
R_T = \frac{C_T}{C_T + S_T + D_T}
</math>

<math>
F_T = \frac{2}{P_T^{-1} + R_T^{-1}}
</math>

All these metrics will be based on the [https://github.com/audioshake/alt-eval public code implementation] from [[#bib-cifka-2024|[1]]], simply extended to include CER.

'''Note that WER and CER will be retained as the main ranking criteria, other metrics will be reported for more detailed comparisons.'''

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process a sample.

= Submission Format =

Several submission formats will be accepted to accommodate the different approaches chosen by the participants.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size (hours)
* The number of parameters in the model (if applicable)
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time per hour of audio

=== I / O ===

The submitted algorithm must take as input argument a path to a folder with .wav files and output predicted lyrics to a destination folder, each result file named as the corresponding input, with the extension changed.

==== Input Audio ====

Participating algorithms will have to receive the following input format:

TBD

==== Output File Format ====

A text file (per song) containing the lyrics, organized by lines and paragraphs.

<line1: word1> <line1: word_2> ... <line1: word_N>\n
<line2: word1> <line2: word_2> ... <line2: word_N>\n
\n
<line3: word1> <line3: word_2> ... <line3: word_N>\n
<line4: word1> <line4: word_2> ... <line4: word_N>\n

Words will be identified by splitting the text file at spaces, tabs, and newline characters.
The submission can also simply generate a file of words separated by spaces, but it will then perform (very) poorly on the new metrics taken from [[#bib-cifka-2024|[1]]].

=== API-based submissions ===

For systems that rely on commercial LLM APIs (e.g. systems similar to [[#bib-lyricwhiz|[2]]]), participants must specify the exact model version and, where possible, provide a fallback local model to allow offline evaluation.
The algorithm should either already contain an API key allowing to use the model (we recommend setting an access token with a clear duration or computational limit), or use the fallback option described hereafter.

=== Fallback: Pre-computed lyrics submission ===

In situations where the inference cannot be run directly by the Task Captain, participants may provide the lyrics files produced by their algorithms.
This should only be considered a fallback option in case the algorithm relies on proprietary or commercial resources that cannot be made accessible to the Task Captains.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

In this challenge, the participants are encouraged but '''not obliged''' to use the open source dataset DALI described below.

The DAMP dataset is unfortunately [https://ccrma.stanford.edu/damp/ no longer available].

Participants are free to use other datasets for training, as long as they are described in the submission and don't overlap with the evaluation sets.

=== DALI Dataset ===

DALI (a large '''D'''ataset of synchronised '''A'''udio, '''L'''yr'''I'''cs and notes) [[#bib-syed-2025|[3]]] is the benchmark dataset for building an acoustic model on polyphonic recordings [[#bib-gupta-2020|[4]]], [[#bib-basak-2021|[5]]], [[#bib-demirel-2021|[6]]] and it contains over 7000 songs with semi-automatically aligned lyrics annotations.

The songs are commercial recordings in full-duration, whereas the lyrics are described according to different levels of granularity including words and notes (and syllables underlying a given note).

For each song DALI provides a link to a matched youtube video for the audio retrieval.

* For more details, see its full description [https://github.com/gabolsgabs/DALI here]. Paper [https://arxiv.org/pdf/1906.10606.pdf here].

= Evaluation Datasets =

The datasets listed below are reserved exclusively for evaluation purposes and '''must not''' be used for training models under any circumstances.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and lead to data leakage of some sort.

=== Jam-ALT dataset ===

The Jam-ALT dataset is built upon the Jamendo dataset [[#bib-stoller-2019|[7]]] and contains 79 songs in 4 languages: English, French, German, and Spanish. All tracks include instrumental accompaniment.
The lyrics were cleaned and their format harmonized in [[#bib-cifka-2024|[1]]].

If you’re working with English-only models, only the 20 English songs will be used for evaluation.

=== MUSDB-ALT ===

Introduced in [[#bib-syed-2025|[3]]], this dataset contains 39 English songs and will be used as an additional evaluation set.

=== Undisclosed Evaluation Set ===

In addition to Jam-ALT and MUSDB-ALT, the submissions will be evaluated on an in-house dataset of 10 English songs from the Western Popular Music repertoire, manually assembled and verified by the Task Captain.

= Time and hardware limits =
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.
A hard limit of 24 hours will be imposed on analysis times, using a single GPU with at most 32GB of VRAM (V100 or similar). Submissions that require more resources can be submitted using the fallback method described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Cífka, O., et al. (2024). Lyrics Transcription for Humans: A Readability-Aware Benchmark. Proc. of the 25th ISMIR Conf.

[2] LyricWhiz — TODO.

[3] Syed et al. (2025). TODO.

[4] Gupta, C., Yılmaz, E., & Li, H. (2020). Automatic lyrics alignment and transcription in polyphonic music: Does background music help? In ICASSP 2020, 496-500. IEEE.

[5] Basak, S., Agarwal, S., Ganapathy, S., & Takahashi, N. (2021, June). End-to-End Lyrics Recognition with Voice to Singing Style Transfer. In ICASSP 2021, 266-270. IEEE.

[6] Demirel, E., Ahlbäck, S., & Dixon, S. (2021). MSTRE-Net: Multistreaming Acoustic Modeling for Automatic Lyrics Transcription. Proc. ISMIR 2021.

[7] Stoller, D., Durand, S., & Ewert, S. (2019). End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-Character Recognition Model. In ICASSP 2019, IEEE.

← Older revision		Revision as of 03:34, 29 June 2026
Line 17:		Line 17:

	'''Notice:''' We particularly encourage participations that build on the latest ALT approaches such as using ''pretrained audio foundation models'' or ''LLMs''.		'''Notice:''' We particularly encourage participations that build on the latest ALT approaches such as using ''pretrained audio foundation models'' or ''LLMs''.
		+
		+	== Real Example ==
		+
		+	Given an audio file such as [https://datasets-server.huggingface.co/assets/jamendolyrics/jam-alt/--/28302224954ef050fe752d1628dd9bac4fc8c02b/--/all/test/0/audio/audio.mp3?Expires=1782704825&Signature=HYbFQpoM1OFg9UfbSCHCepQwGBgEfWMhqnX7AV2Wc9KUK8RkeZtcNyHvqMBRX-OzHMDNYF71nA3aoQYJYXXipMc6XDt1mkoM7YwQr~KgY8T4FadL59wEQgNiVIRMbGvAVzmtdJKWxk9kp1uaux2v0zL2KHTWozKXpjsZbU3FazZBmL7l4Qc1ZjiIoWzVyaE2lkhDsHrS8ssLGAEscVaaqZ6jCSlYMRViez2MEUZ8HIrVdawa0rRLcVzfp1hIobFKCCj2jQ~1~d9KEYLvntwz1pqlR5J3QCG0eGf1bReEnxIzTZE7aG4IbfMU8awCDCi3E2fBKUezmB7xPNhH2t9QFg__&Key-Pair-Id=K204OQ5RWQVDLD this one], which comes from the Jam-ALT dataset, the expected lyrics are as follows:
		+
		+	<div class="toccolours mw-collapsible mw-collapsed" style="width:400px; overflow:auto;">
		+	<div style="font-weight:bold;line-height:1.6;">HILA - Give me the same - Lyrics</div>
		+	<div class="mw-collapsible-content">
		+	Lay awake at night<br>
		+	Wondering how could I<br>
		+	Let it get this way<br>
		+	Through all the pain<br>
		+	I took all the blame<br>
		+	While you cursed my last name<br>
		+	I thought we could get better<br>
		+	Stayed committed like a soldier<br>
		+	Believed we're okay<br>
		+	Now I know that you don't care<br>
		+	At all<br>
		+	<br><br>
		+	Time wasted<br>
		+	Gave all of my soul and my heart to someone that would never<br>
		+	(Give me the same)<br>
		+	I was a fool to believe that you would finally be the one to<br>
		+	(Give me the same)<br>
		+	Heartbroken now<br>
		+	How you let me down<br>
		+	But somehow, deep down, I still do<br>
		+	Love you<br>
		+	<br><br>
		+	Now you're out all night<br>
		+	Telling your lies<br>
		+	About the way I hurt you<br>
		+	They couldn't see the way<br>
		+	You made me feel ashamed<br>
		+	For choosing to love you<br>
		+	I began to see clearer<br>
		+	That you were just a liar<br>
		+	Stuck in your ways<br>
		+	So now I choose to walk away<br>
		+	Forever<br>
		+	<br><br>
		+	Time wasted<br>
		+	Gave all of my soul and my heart to someone that would never<br>
		+	(Give me the same)<br>
		+	I was a fool to believe that you would finally be the one to<br>
		+	Give me the same<br>
		+	Heartbroken now<br>
		+	How you let me down<br>
		+	But somehow, deep down, I still do<br>
		+	Love you<br>
		+	<br><br>
		+	How could you just throw it all away, babe<br>
		+	(How could you throw it all away)<br>
		+	I stayed with you through all the lies and pain, oh<br>
		+	(Through all the lies and pain, oh)<br>
		+	Tears roll down my eyes (down my eyes)<br>
		+	As I now realize<br>
		+	Your love was never truly mine<br>
		+	At all<br>
		+	<br><br>
		+	Time wasted<br>
		+	Gave all of my soul and my heart to someone that would never<br>
		+	(Give me the same)<br>
		+	I was a fool to believe that you would finally be the one to<br>
		+	(Give me the same)<br>
		+	Heartbroken now (heartbroken now)<br>
		+	How you let me down (oh)<br>
		+	But somehow, deep down, I still do<br>
		+	Love you<br>
		+	</div></div>
		+
		+

	= Evaluation =		= Evaluation =

@@ Line 91: / Line 91: @@
 Participating algorithms will have to receive the following audio input format:
-- Musical tracks with accompaniment (polyphonic, not vocals only)
+* Musical tracks with accompaniment (polyphonic, not vocals only)
-- 16-bit PCM <code>wav</code> files
+* 16-bit PCM <code>wav</code> files
-- 44.1 kHz sampling rate
+* 44.1 kHz sampling rate
-- stereo (2 channels)
+* stereo (2 channels)
 Be aware that the wav files are obtained through conversion from compressed formats (mp3 or AAC) for the test sets we considered, so the quality is not exactly that of a perfect 44.1 kHz wav file.

@@ Line 171: / Line 171: @@
 <span id="bib-cifka-2024">[1]</span> Cífka, O., et al. (2024). Lyrics Transcription for Humans: A Readability-Aware Benchmark. Proc. of the 25th ISMIR Conf.
-<span id="bib-lyricwhiz">[2]</span> LyricWhiz — TODO.
+<span id="bib-lyricwhiz">[2]</span> Zhuo, L., et al. (2023). LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT. Proc. of the 24th ISMIR Conf.
-<span id="bib-syed-2025">[3]</span> Syed et al. (2025). TODO.
+<span id="bib-syed-2025">[3]</span> Syed, J., et al. (2025). Exploiting Music Source Separation for Automatic Lyrics Transcription with Whisper. ICMEW.
 <span id="bib-gupta-2020">[4]</span> Gupta, C., Yılmaz, E., & Li, H. (2020). Automatic lyrics alignment and transcription in polyphonic music: Does background music help? In ICASSP 2020, 496-500. IEEE.

@@ Line 89: / Line 89: @@
 ==== Input Audio ====
-Participating algorithms will have to receive the following input format:
+Participating algorithms will have to receive the following audio input format:
 ==== Output File Format ====
@@ Line 150: / Line 158: @@
 If you’re working with English-only models, only the 20 English songs will be used for evaluation.
 === MUSDB-ALT ===
 Introduced in [[#bib-syed-2025|[3]]], this dataset contains 39 English songs and will be used as an additional evaluation set.
 === Undisclosed Evaluation Set ===