MIREX Wiki - User contributions [en]

2026:Audio-to-Score Transcription

2026-06-29T08:23:11Z

Alexandre DHooge: /* Undisclosed Evaluation Set */ add more details

= Description =

This page describes the '''MIREX2026: Audio-to-Score Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.

Automatic Music Transcription (AMT) is the task of designing computational algorithms that convert acoustic music signals into a symbolic musical representation [[#bib-benetos-2019|[2]]]. Existing MIREX tasks have evaluated several important AMT components, including melody extraction, drum detection, piano transcription, and multi-instrument transcription. However, these tasks have generally stopped at MIDI or note-event outputs, which are not equivalent to a well-formed musical score.

The Audio-to-Score Transcription (A2S) challenge focuses on this missing step. Participating systems will receive an audio recording of a polyphonic music piece and must output a digital symbolic score that can be read by a musician or notation program. The target output is a kern score encoding standard musical information such as pitches, durations, accidentals, staves, meter, clef, key signature, and bar structure.

The task can be informally expressed as follows:

Prediction('''S''') = argmax P('''S'''|'''X''')

where '''S''' is a symbolic score and '''X''' is the input audio signal.

The challenge will include polyphonic recordings and may include multiple instruments. Pieces may be full recordings or shorter excerpts, such as 3 to 6 bars.

Two tracks will be considered:

* '''Staves-Informed A2S''': staves metadata is provided and may be used to guide voice identification and instrument recognition.
* '''Blind A2S''': only the audio recording is provided.

Participants may enter both tracks. Results will be reported separately.

'''Notice:''' We particularly encourage submissions that move beyond isolated AMT components and produce complete, valid symbolic scores.

== Real Example ==

Given an input audio file such as [https://datasets-server.huggingface.co/assets/PRAIG/quartets-quartets/--/462cb9a3681d5ce06dc4fdb7a13edf1f80794ffe/--/default/train/0/audio/audio.wav?Expires=1782724216&Signature=lzWGXP665ntsvNRqaKyeVOTczqf8PwTWg9NmEzTo2iWkbqwBrvYD6ko6MYhVTZ7Oz6PCFGKnKUEIZyEQgpduxsZItA~GD5hEF1uq7UZuPOvMXBShWDOeqfjLexUQIULXtJ~GtU6HLm2oSRkMxWJYkF0leCDpg7NY0I4Z4X9gfB0j9~27XfgkNu8wWDhak5ryHZaZ1SzXc4qa7rxWRd0FmDtckSQzpzoVnaKQUID6oMQB~VOZ3GVXPKONDAVJI0vc7VYuZ77X2TB8~3F6mtkCgqrtWvYKxEyPV5hZ6ilIQYNr164OCrEuZmD~hycnWw6o4qA-0tLP5oSuQAWe~rIZ4A__&Key-Pair-Id=K204OQ5RWQVDLD this one], the model should generate a <code>**kern</code> file like this:

<div class="toccolours mw-collapsible mw-collapsed" style="width:600px; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Kern score transcription</div>
<div class="mw-collapsible-content">
**kern **dynam **kern **dynam **kern **dynam **kern **dynam
*Icello *Icello *Iviola *Iviola *Ivioln *Ivioln *Iflt *Iflt
*clefF4 * *clefC3 * *clefG2 * *clefG2 *
*k[f#] * *k[f#] * *k[f#] * *k[f#] *
*G: * *G: * *G: * *G: *
*M3/4 * *M3/4 * *M3/4 * *M3/4 *
*MM71 * *MM71 * *MM71 * *MM71 *
! ! ! ! ! ! ! 16aaP/ !
8A . 8c . 2.r . 4gg .
8A . 8c . . . . .
8A . 8c . . . 4.ff# .
8A . 8c . . . . .
8A . 8c . . . . .
8G . 8B . . . 8gg .
=3 =3 =3 =3 =3 =3 =3 =3
8F# . 8A . [2.ddd p 8aa .
8F# . 8A . . . 8ff# .
8F# . 8A . . . 4.dd .
8F# . 8A . . . . .
8F# . 8A . . . . .
8F# . 8A . . . 8cc .
=4 =4 =4 =4 =4 =4 =4 =4
8G . 4A . 4.ddd] . 4cc .
8G . . . . . . .
8G . 4G . . . 4b .
8G . . . 16dd . . .
. . . . 16gg . . .
8G . 4r . 16gg' . 4r .
. . . . 16bb . . .
8G . . . 16ccc . . .
. . . . 16ddd . . .
=5 =5 =5 =5 =5 =5 =5 =5
*- *- *- *- *- *- *- *-
</div></div>

= Evaluation =

Submissions will be evaluated with metrics that assess both the textual quality of the produced kern files and the musical transcription quality of the corresponding scores.

=== Score Quality ===

'''Character Error Rate''' (CER): the Levenshtein distance between the submitted and ground-truth kern files at the character level. This metric penalizes character insertions, deletions, and substitutions, and is sensitive to details such as pitch spelling and note durations.

'''Word Error Rate''' (WER): the Levenshtein distance computed on tab- and newline-separated strings in a kern file. WER is reported in recent A2S papers [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. Compared to CER, it evaluates the output closer to the note-token level, checking whether durations and pitches are jointly correct.

'''Line Error Rate''' (LER): the Levenshtein distance computed on complete lines of the output kern files. LER is stricter than CER and WER, and helps verify whether multiple instruments or voices are correctly transcribed and aligned at the beat and sub-beat level.

=== Musical Transcription Quality ===

In addition to CER, WER, and LER, this challenge will use the '''MV2H''' metric introduced in [[#bib-mcleod-2018|[4]]] and used in recent A2S work [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. MV2H is an F-score obtained as the arithmetic mean of five sub-metrics:

* '''Multi-pitch detection''': pitches must be correct and detected at onsets within a 50 ms error threshold.
* '''Voice separation''': notes belonging to the same voice or instrument should be grouped together.
* '''Metrical alignment''': bars, beats, and sub-beats should be correctly identified.
* '''Note value detection''': note durations should be correctly transcribed.
* '''Harmonic analysis''': average of a key detection score and a chord-symbol recall.

The harmonic analysis component will not be used for the initial edition because no chord information is available in the considered datasets.

Public implementations of MV2H are available:

* [https://github.com/apmcleod/MV2H Official MV2H repository]
* [https://github.com/lucasmpaim/pyMV2H Python implementation]

The evaluation pipeline will adapt these implementations to process kern files, building where appropriate on the public code from [[#bib-alfaro-2024|[1]]]:

* [https://github.com/mariaalfaroc/a2s-transformer A2S Transformer repository]

'''Note that WER and MV2H will be retained as the main ranking criteria.''' CER and LER will be reported for more detailed comparisons.

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process the evaluation set.

= Submission Format =

Several submission formats will be accepted to accommodate different system designs.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size
* The number of parameters in the model, if applicable
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time required for the evaluation set

We strongly recommend sharing the submitted algorithms or checkpoints under permissive open licenses, but any licensing (or even not sharing publicly) is accepted as long as it is clearly reported at the time of submission.

=== I / O ===

The submitted algorithm must take as input an audio file, or a folder containing audio files, and write one predicted <code>**kern</code> file for each input audio file to a specified output directory.

==== Input Audio ====

Participating algorithms will receive audio files in the following format:

* 16 bit FLAC
* 22 050 Hz
* 30s (for Quartets)

If your pipeline expects characteristics different from the ones described above, the conversion should be done automatically in your algorithm.

For the '''Blind A2S''' track, only the audio recording will be provided.

For the '''Staves-Informed A2S''' track, staves metadata will also be provided. This metadata is the <code>**kern</code> header, that looks like this in the Quartets dataset for example:

**kern **dynam **kern **dynam **kern **dynam **kern **dynam
*Icello *Icello *Iviola *Iviola *Ivioln *Ivioln *Iflt *Iflt
*clefF4 * *clefC3 * *clefG2 * *clefG2 *
*k[f#c#] * *k[f#c#] * *k[f#c#] * *k[f#c#] *
*D: * *D: * *D: * *D: *
*M4/4 * *M4/4 * *M4/4 * *M4/4 *
*MM130 * *MM130 * *MM130 * *MM130 *

==== Output File Format ====

Each submitted prediction must be a valid <code>**kern</code> file.

The score must encode, at minimum:

* Staves musical metadata, including meter, clef, and key signature when required
* Pitches, including octaves and accidentals
* Note durations
* Bar lines and metrical structure
* Voice or instrument organization when applicable

For more details, participants may refer to:

* [https://www.humdrum.org/Humdrum/representations/kern.html Humdrum kern representation]

We chose the <code>**kern</code> format for compatibility with existing SOTA work in A2S. Participants may use other formats during training or as intermediate outputs, but the final files should be in <code>**kern</code> format for evaluation.
Please make sure the proposed algorithm include a conversion step if it is required.

=== Code Submissions ===

Participants may provide a Docker container or access to a code repository with clear environment setup instructions and a script that reads the input audio and writes output kern files to the requested destination directory.

The submitted script should be documented in the README and should not require manual intervention during evaluation.

=== Fallback: Pre-computed Kern Submission ===

Participants may alternatively submit the final kern files directly, similarly to the option proposed for MIREX 2024 Polyphonic Transcription.

This fallback is intended to allow participation from systems that cannot be evaluated directly by the Task Captain because of resource, licensing, or infrastructure constraints.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

Participants are free to use the training and validation sets of the datasets listed below. Data augmentation is allowed.

Additional training data may also be used, as long as it does not overlap with the evaluation sets and is clearly documented in the submission.

=== Quartets Dataset ===

The Quartets dataset consists of synthetic audio renderings of Haydn, Mozart, and Beethoven string quartets, together with their full kern scores [[#bib-roman-2019|[6]]].

The scores are taken from the [https://github.com/humdrum-tools/humdrum-data humdrum-data repository]. The kern files were split into 3 to 6 measure excerpts and synthesized into audio from performance MIDI files.

The final dataset contains approximately 20 hours of audio for 38,051 excerpts and 3 composers:

* 18,162 excerpts for Haydn
* 7,435 excerpts for Mozart
* 12,454 excerpts for Beethoven

The train, validation, and test splits used in [[#bib-alfaro-2024|[1]]] are publicly available:

* [https://huggingface.co/datasets/PRAIG/quartets-quartets Quartets dataset on Hugging Face]

= Evaluation Datasets =

The datasets listed below are reserved for evaluation purposes and '''must not''' be used for training models.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and could lead to data leakage.

=== Quartets Test Set ===

Evaluation will include the test split of the Quartets dataset. Results on this dataset will be reported separately.

=== Undisclosed Evaluation Set ===

An additional custom evaluation set will be used to ensure fairness and out-of-distribution data.

It will also include quartets, but with composers other than Haydn, Mozart, and Beethoven.
The data structure will replicate that of the Quartets dataset, and should therefore be usable directly by the submitted algorithms.

= Time and Hardware Limits =

Due to the potentially high number of participants in this and other audio tasks, hard limits on runtime and hardware use will be imposed.

Submissions that require more than 32 GB of VRAM, or more than 24 hours to process the test sets on a single GPU (V100 or similar), cannot be evaluated directly by the Task Captain. Participants in this situation may use the fallback kern submission format described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Alfaro-Contreras, M., et al. (2024). A Transformer Approach for Polyphonic Audio-to-Score Transcription. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2] Benetos, E., et al. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1).

[3] Liu, L., et al. (2021). Joint Multi-Pitch Detection and Score Transcription for Polyphonic Piano Music. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4] Mcleod, A., et al. (2018). Evaluating Automatic Polyphonic Music Transcription. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[5] Roman, M. A., et al. (2018). An End-to-End Framework for Audio-to-Score Music Transcription on Monophonic Excerpts. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[6] Roman, M. A., et al. (2019). A Holistic Approach to Polyphonic Music Transcription with Neural Networks. Proc. of the 20th International Society for Music Information Retrieval Conference (ISMIR).

[7] Smaragdis, P., et al. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.

2026:Audio-to-Score Transcription

2026-06-29T08:15:58Z

Alexandre DHooge: /* Description */ add real example

= Description =

This page describes the '''MIREX2026: Audio-to-Score Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.

Automatic Music Transcription (AMT) is the task of designing computational algorithms that convert acoustic music signals into a symbolic musical representation [[#bib-benetos-2019|[2]]]. Existing MIREX tasks have evaluated several important AMT components, including melody extraction, drum detection, piano transcription, and multi-instrument transcription. However, these tasks have generally stopped at MIDI or note-event outputs, which are not equivalent to a well-formed musical score.

The Audio-to-Score Transcription (A2S) challenge focuses on this missing step. Participating systems will receive an audio recording of a polyphonic music piece and must output a digital symbolic score that can be read by a musician or notation program. The target output is a kern score encoding standard musical information such as pitches, durations, accidentals, staves, meter, clef, key signature, and bar structure.

The task can be informally expressed as follows:

Prediction('''S''') = argmax P('''S'''|'''X''')

where '''S''' is a symbolic score and '''X''' is the input audio signal.

The challenge will include polyphonic recordings and may include multiple instruments. Pieces may be full recordings or shorter excerpts, such as 3 to 6 bars.

Two tracks will be considered:

* '''Staves-Informed A2S''': staves metadata is provided and may be used to guide voice identification and instrument recognition.
* '''Blind A2S''': only the audio recording is provided.

Participants may enter both tracks. Results will be reported separately.

'''Notice:''' We particularly encourage submissions that move beyond isolated AMT components and produce complete, valid symbolic scores.

== Real Example ==

Given an input audio file such as [https://datasets-server.huggingface.co/assets/PRAIG/quartets-quartets/--/462cb9a3681d5ce06dc4fdb7a13edf1f80794ffe/--/default/train/0/audio/audio.wav?Expires=1782724216&Signature=lzWGXP665ntsvNRqaKyeVOTczqf8PwTWg9NmEzTo2iWkbqwBrvYD6ko6MYhVTZ7Oz6PCFGKnKUEIZyEQgpduxsZItA~GD5hEF1uq7UZuPOvMXBShWDOeqfjLexUQIULXtJ~GtU6HLm2oSRkMxWJYkF0leCDpg7NY0I4Z4X9gfB0j9~27XfgkNu8wWDhak5ryHZaZ1SzXc4qa7rxWRd0FmDtckSQzpzoVnaKQUID6oMQB~VOZ3GVXPKONDAVJI0vc7VYuZ77X2TB8~3F6mtkCgqrtWvYKxEyPV5hZ6ilIQYNr164OCrEuZmD~hycnWw6o4qA-0tLP5oSuQAWe~rIZ4A__&Key-Pair-Id=K204OQ5RWQVDLD this one], the model should generate a <code>**kern</code> file like this:

<div class="toccolours mw-collapsible mw-collapsed" style="width:600px; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Kern score transcription</div>
<div class="mw-collapsible-content">
**kern **dynam **kern **dynam **kern **dynam **kern **dynam
*Icello *Icello *Iviola *Iviola *Ivioln *Ivioln *Iflt *Iflt
*clefF4 * *clefC3 * *clefG2 * *clefG2 *
*k[f#] * *k[f#] * *k[f#] * *k[f#] *
*G: * *G: * *G: * *G: *
*M3/4 * *M3/4 * *M3/4 * *M3/4 *
*MM71 * *MM71 * *MM71 * *MM71 *
! ! ! ! ! ! ! 16aaP/ !
8A . 8c . 2.r . 4gg .
8A . 8c . . . . .
8A . 8c . . . 4.ff# .
8A . 8c . . . . .
8A . 8c . . . . .
8G . 8B . . . 8gg .
=3 =3 =3 =3 =3 =3 =3 =3
8F# . 8A . [2.ddd p 8aa .
8F# . 8A . . . 8ff# .
8F# . 8A . . . 4.dd .
8F# . 8A . . . . .
8F# . 8A . . . . .
8F# . 8A . . . 8cc .
=4 =4 =4 =4 =4 =4 =4 =4
8G . 4A . 4.ddd] . 4cc .
8G . . . . . . .
8G . 4G . . . 4b .
8G . . . 16dd . . .
. . . . 16gg . . .
8G . 4r . 16gg' . 4r .
. . . . 16bb . . .
8G . . . 16ccc . . .
. . . . 16ddd . . .
=5 =5 =5 =5 =5 =5 =5 =5
*- *- *- *- *- *- *- *-
</div></div>

= Evaluation =

Submissions will be evaluated with metrics that assess both the textual quality of the produced kern files and the musical transcription quality of the corresponding scores.

=== Score Quality ===

'''Character Error Rate''' (CER): the Levenshtein distance between the submitted and ground-truth kern files at the character level. This metric penalizes character insertions, deletions, and substitutions, and is sensitive to details such as pitch spelling and note durations.

'''Word Error Rate''' (WER): the Levenshtein distance computed on tab- and newline-separated strings in a kern file. WER is reported in recent A2S papers [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. Compared to CER, it evaluates the output closer to the note-token level, checking whether durations and pitches are jointly correct.

'''Line Error Rate''' (LER): the Levenshtein distance computed on complete lines of the output kern files. LER is stricter than CER and WER, and helps verify whether multiple instruments or voices are correctly transcribed and aligned at the beat and sub-beat level.

=== Musical Transcription Quality ===

In addition to CER, WER, and LER, this challenge will use the '''MV2H''' metric introduced in [[#bib-mcleod-2018|[4]]] and used in recent A2S work [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. MV2H is an F-score obtained as the arithmetic mean of five sub-metrics:

* '''Multi-pitch detection''': pitches must be correct and detected at onsets within a 50 ms error threshold.
* '''Voice separation''': notes belonging to the same voice or instrument should be grouped together.
* '''Metrical alignment''': bars, beats, and sub-beats should be correctly identified.
* '''Note value detection''': note durations should be correctly transcribed.
* '''Harmonic analysis''': average of a key detection score and a chord-symbol recall.

The harmonic analysis component will not be used for the initial edition because no chord information is available in the considered datasets.

Public implementations of MV2H are available:

* [https://github.com/apmcleod/MV2H Official MV2H repository]
* [https://github.com/lucasmpaim/pyMV2H Python implementation]

The evaluation pipeline will adapt these implementations to process kern files, building where appropriate on the public code from [[#bib-alfaro-2024|[1]]]:

* [https://github.com/mariaalfaroc/a2s-transformer A2S Transformer repository]

'''Note that WER and MV2H will be retained as the main ranking criteria.''' CER and LER will be reported for more detailed comparisons.

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process the evaluation set.

= Submission Format =

Several submission formats will be accepted to accommodate different system designs.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size
* The number of parameters in the model, if applicable
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time required for the evaluation set

We strongly recommend sharing the submitted algorithms or checkpoints under permissive open licenses, but any licensing (or even not sharing publicly) is accepted as long as it is clearly reported at the time of submission.

=== I / O ===

The submitted algorithm must take as input an audio file, or a folder containing audio files, and write one predicted <code>**kern</code> file for each input audio file to a specified output directory.

==== Input Audio ====

Participating algorithms will receive audio files in the following format:

* 16 bit FLAC
* 22 050 Hz
* 30s (for Quartets)

If your pipeline expects characteristics different from the ones described above, the conversion should be done automatically in your algorithm.

For the '''Blind A2S''' track, only the audio recording will be provided.

For the '''Staves-Informed A2S''' track, staves metadata will also be provided. This metadata is the <code>**kern</code> header, that looks like this in the Quartets dataset for example:

**kern **dynam **kern **dynam **kern **dynam **kern **dynam
*Icello *Icello *Iviola *Iviola *Ivioln *Ivioln *Iflt *Iflt
*clefF4 * *clefC3 * *clefG2 * *clefG2 *
*k[f#c#] * *k[f#c#] * *k[f#c#] * *k[f#c#] *
*D: * *D: * *D: * *D: *
*M4/4 * *M4/4 * *M4/4 * *M4/4 *
*MM130 * *MM130 * *MM130 * *MM130 *

==== Output File Format ====

Each submitted prediction must be a valid <code>**kern</code> file.

The score must encode, at minimum:

* Staves musical metadata, including meter, clef, and key signature when required
* Pitches, including octaves and accidentals
* Note durations
* Bar lines and metrical structure
* Voice or instrument organization when applicable

For more details, participants may refer to:

* [https://www.humdrum.org/Humdrum/representations/kern.html Humdrum kern representation]

We chose the <code>**kern</code> format for compatibility with existing SOTA work in A2S. Participants may use other formats during training or as intermediate outputs, but the final files should be in <code>**kern</code> format for evaluation.
Please make sure the proposed algorithm include a conversion step if it is required.

=== Code Submissions ===

Participants may provide a Docker container or access to a code repository with clear environment setup instructions and a script that reads the input audio and writes output kern files to the requested destination directory.

The submitted script should be documented in the README and should not require manual intervention during evaluation.

=== Fallback: Pre-computed Kern Submission ===

Participants may alternatively submit the final kern files directly, similarly to the option proposed for MIREX 2024 Polyphonic Transcription.

This fallback is intended to allow participation from systems that cannot be evaluated directly by the Task Captain because of resource, licensing, or infrastructure constraints.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

Participants are free to use the training and validation sets of the datasets listed below. Data augmentation is allowed.

Additional training data may also be used, as long as it does not overlap with the evaluation sets and is clearly documented in the submission.

=== Quartets Dataset ===

The Quartets dataset consists of synthetic audio renderings of Haydn, Mozart, and Beethoven string quartets, together with their full kern scores [[#bib-roman-2019|[6]]].

The scores are taken from the [https://github.com/humdrum-tools/humdrum-data humdrum-data repository]. The kern files were split into 3 to 6 measure excerpts and synthesized into audio from performance MIDI files.

The final dataset contains approximately 20 hours of audio for 38,051 excerpts and 3 composers:

* 18,162 excerpts for Haydn
* 7,435 excerpts for Mozart
* 12,454 excerpts for Beethoven

The train, validation, and test splits used in [[#bib-alfaro-2024|[1]]] are publicly available:

* [https://huggingface.co/datasets/PRAIG/quartets-quartets Quartets dataset on Hugging Face]

= Evaluation Datasets =

The datasets listed below are reserved for evaluation purposes and '''must not''' be used for training models.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and could lead to data leakage.

=== Quartets Test Set ===

Evaluation will include the test split of the Quartets dataset. Results on this dataset will be reported separately.

=== Undisclosed Evaluation Set ===

An additional custom evaluation set will be used to ensure fairness and out-of-distribution data.

General details on this dataset are:
TBD

= Time and Hardware Limits =

Due to the potentially high number of participants in this and other audio tasks, hard limits on runtime and hardware use will be imposed.

Submissions that require more than 32 GB of VRAM, or more than 24 hours to process the test sets on a single GPU (V100 or similar), cannot be evaluated directly by the Task Captain. Participants in this situation may use the fallback kern submission format described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Alfaro-Contreras, M., et al. (2024). A Transformer Approach for Polyphonic Audio-to-Score Transcription. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2] Benetos, E., et al. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1).

[3] Liu, L., et al. (2021). Joint Multi-Pitch Detection and Score Transcription for Polyphonic Piano Music. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4] Mcleod, A., et al. (2018). Evaluating Automatic Polyphonic Music Transcription. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[5] Roman, M. A., et al. (2018). An End-to-End Framework for Audio-to-Score Music Transcription on Monophonic Excerpts. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[6] Roman, M. A., et al. (2019). A Holistic Approach to Polyphonic Music Transcription with Neural Networks. Proc. of the 20th International Society for Music Information Retrieval Conference (ISMIR).

[7] Smaragdis, P., et al. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.

2026:Audio-to-Score Transcription

2026-06-29T08:09:50Z

Alexandre DHooge: Remove MuseSyn

= Description =

This page describes the '''MIREX2026: Audio-to-Score Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.

Automatic Music Transcription (AMT) is the task of designing computational algorithms that convert acoustic music signals into a symbolic musical representation [[#bib-benetos-2019|[2]]]. Existing MIREX tasks have evaluated several important AMT components, including melody extraction, drum detection, piano transcription, and multi-instrument transcription. However, these tasks have generally stopped at MIDI or note-event outputs, which are not equivalent to a well-formed musical score.

The Audio-to-Score Transcription (A2S) challenge focuses on this missing step. Participating systems will receive an audio recording of a polyphonic music piece and must output a digital symbolic score that can be read by a musician or notation program. The target output is a kern score encoding standard musical information such as pitches, durations, accidentals, staves, meter, clef, key signature, and bar structure.

The task can be informally expressed as follows:

Prediction('''S''') = argmax P('''S'''|'''X''')

where '''S''' is a symbolic score and '''X''' is the input audio signal.

The challenge will include polyphonic recordings and may include multiple instruments. Pieces may be full recordings or shorter excerpts, such as 3 to 6 bars.

Two tracks will be considered:

* '''Staves-Informed A2S''': staves metadata is provided and may be used to guide voice identification and instrument recognition.
* '''Blind A2S''': only the audio recording is provided.

Participants may enter both tracks. Results will be reported separately.

'''Notice:''' We particularly encourage submissions that move beyond isolated AMT components and produce complete, valid symbolic scores.

= Evaluation =

Submissions will be evaluated with metrics that assess both the textual quality of the produced kern files and the musical transcription quality of the corresponding scores.

=== Score Quality ===

'''Character Error Rate''' (CER): the Levenshtein distance between the submitted and ground-truth kern files at the character level. This metric penalizes character insertions, deletions, and substitutions, and is sensitive to details such as pitch spelling and note durations.

'''Word Error Rate''' (WER): the Levenshtein distance computed on tab- and newline-separated strings in a kern file. WER is reported in recent A2S papers [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. Compared to CER, it evaluates the output closer to the note-token level, checking whether durations and pitches are jointly correct.

'''Line Error Rate''' (LER): the Levenshtein distance computed on complete lines of the output kern files. LER is stricter than CER and WER, and helps verify whether multiple instruments or voices are correctly transcribed and aligned at the beat and sub-beat level.

=== Musical Transcription Quality ===

In addition to CER, WER, and LER, this challenge will use the '''MV2H''' metric introduced in [[#bib-mcleod-2018|[4]]] and used in recent A2S work [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. MV2H is an F-score obtained as the arithmetic mean of five sub-metrics:

* '''Multi-pitch detection''': pitches must be correct and detected at onsets within a 50 ms error threshold.
* '''Voice separation''': notes belonging to the same voice or instrument should be grouped together.
* '''Metrical alignment''': bars, beats, and sub-beats should be correctly identified.
* '''Note value detection''': note durations should be correctly transcribed.
* '''Harmonic analysis''': average of a key detection score and a chord-symbol recall.

The harmonic analysis component will not be used for the initial edition because no chord information is available in the considered datasets.

Public implementations of MV2H are available:

* [https://github.com/apmcleod/MV2H Official MV2H repository]
* [https://github.com/lucasmpaim/pyMV2H Python implementation]

The evaluation pipeline will adapt these implementations to process kern files, building where appropriate on the public code from [[#bib-alfaro-2024|[1]]]:

* [https://github.com/mariaalfaroc/a2s-transformer A2S Transformer repository]

'''Note that WER and MV2H will be retained as the main ranking criteria.''' CER and LER will be reported for more detailed comparisons.

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process the evaluation set.

= Submission Format =

Several submission formats will be accepted to accommodate different system designs.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size
* The number of parameters in the model, if applicable
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time required for the evaluation set

We strongly recommend sharing the submitted algorithms or checkpoints under permissive open licenses, but any licensing (or even not sharing publicly) is accepted as long as it is clearly reported at the time of submission.

=== I / O ===

The submitted algorithm must take as input an audio file, or a folder containing audio files, and write one predicted <code>**kern</code> file for each input audio file to a specified output directory.

==== Input Audio ====

Participating algorithms will receive audio files in the following format:

* 16 bit FLAC
* 22 050 Hz
* 30s (for Quartets)

If your pipeline expects characteristics different from the ones described above, the conversion should be done automatically in your algorithm.

For the '''Blind A2S''' track, only the audio recording will be provided.

For the '''Staves-Informed A2S''' track, staves metadata will also be provided. This metadata is the <code>**kern</code> header, that looks like this in the Quartets dataset for example:

**kern **dynam **kern **dynam **kern **dynam **kern **dynam
*Icello *Icello *Iviola *Iviola *Ivioln *Ivioln *Iflt *Iflt
*clefF4 * *clefC3 * *clefG2 * *clefG2 *
*k[f#c#] * *k[f#c#] * *k[f#c#] * *k[f#c#] *
*D: * *D: * *D: * *D: *
*M4/4 * *M4/4 * *M4/4 * *M4/4 *
*MM130 * *MM130 * *MM130 * *MM130 *

==== Output File Format ====

Each submitted prediction must be a valid <code>**kern</code> file.

The score must encode, at minimum:

* Staves musical metadata, including meter, clef, and key signature when required
* Pitches, including octaves and accidentals
* Note durations
* Bar lines and metrical structure
* Voice or instrument organization when applicable

For more details, participants may refer to:

* [https://www.humdrum.org/Humdrum/representations/kern.html Humdrum kern representation]

We chose the <code>**kern</code> format for compatibility with existing SOTA work in A2S. Participants may use other formats during training or as intermediate outputs, but the final files should be in <code>**kern</code> format for evaluation.
Please make sure the proposed algorithm include a conversion step if it is required.

=== Code Submissions ===

Participants may provide a Docker container or access to a code repository with clear environment setup instructions and a script that reads the input audio and writes output kern files to the requested destination directory.

The submitted script should be documented in the README and should not require manual intervention during evaluation.

=== Fallback: Pre-computed Kern Submission ===

Participants may alternatively submit the final kern files directly, similarly to the option proposed for MIREX 2024 Polyphonic Transcription.

This fallback is intended to allow participation from systems that cannot be evaluated directly by the Task Captain because of resource, licensing, or infrastructure constraints.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

Participants are free to use the training and validation sets of the datasets listed below. Data augmentation is allowed.

Additional training data may also be used, as long as it does not overlap with the evaluation sets and is clearly documented in the submission.

=== Quartets Dataset ===

The Quartets dataset consists of synthetic audio renderings of Haydn, Mozart, and Beethoven string quartets, together with their full kern scores [[#bib-roman-2019|[6]]].

The scores are taken from the [https://github.com/humdrum-tools/humdrum-data humdrum-data repository]. The kern files were split into 3 to 6 measure excerpts and synthesized into audio from performance MIDI files.

The final dataset contains approximately 20 hours of audio for 38,051 excerpts and 3 composers:

* 18,162 excerpts for Haydn
* 7,435 excerpts for Mozart
* 12,454 excerpts for Beethoven

The train, validation, and test splits used in [[#bib-alfaro-2024|[1]]] are publicly available:

* [https://huggingface.co/datasets/PRAIG/quartets-quartets Quartets dataset on Hugging Face]

= Evaluation Datasets =

The datasets listed below are reserved for evaluation purposes and '''must not''' be used for training models.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and could lead to data leakage.

=== Quartets Test Set ===

Evaluation will include the test split of the Quartets dataset. Results on this dataset will be reported separately.

=== Undisclosed Evaluation Set ===

An additional custom evaluation set will be used to ensure fairness and out-of-distribution data.

General details on this dataset are:
TBD

= Time and Hardware Limits =

Due to the potentially high number of participants in this and other audio tasks, hard limits on runtime and hardware use will be imposed.

Submissions that require more than 32 GB of VRAM, or more than 24 hours to process the test sets on a single GPU (V100 or similar), cannot be evaluated directly by the Task Captain. Participants in this situation may use the fallback kern submission format described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Alfaro-Contreras, M., et al. (2024). A Transformer Approach for Polyphonic Audio-to-Score Transcription. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2] Benetos, E., et al. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1).

[3] Liu, L., et al. (2021). Joint Multi-Pitch Detection and Score Transcription for Polyphonic Piano Music. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4] Mcleod, A., et al. (2018). Evaluating Automatic Polyphonic Music Transcription. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[5] Roman, M. A., et al. (2018). An End-to-End Framework for Audio-to-Score Music Transcription on Monophonic Excerpts. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[6] Roman, M. A., et al. (2019). A Holistic Approach to Polyphonic Music Transcription with Neural Networks. Proc. of the 20th International Society for Music Information Retrieval Conference (ISMIR).

[7] Smaragdis, P., et al. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.

2026:Audio-to-Score Transcription

2026-06-29T07:59:30Z

Alexandre DHooge: /* MuseSyn Test Set */

= Description =

This page describes the '''MIREX2026: Audio-to-Score Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.

Automatic Music Transcription (AMT) is the task of designing computational algorithms that convert acoustic music signals into a symbolic musical representation [[#bib-benetos-2019|[2]]]. Existing MIREX tasks have evaluated several important AMT components, including melody extraction, drum detection, piano transcription, and multi-instrument transcription. However, these tasks have generally stopped at MIDI or note-event outputs, which are not equivalent to a well-formed musical score.

The Audio-to-Score Transcription (A2S) challenge focuses on this missing step. Participating systems will receive an audio recording of a polyphonic music piece and must output a digital symbolic score that can be read by a musician or notation program. The target output is a kern score encoding standard musical information such as pitches, durations, accidentals, staves, meter, clef, key signature, and bar structure.

The task can be informally expressed as follows:

Prediction('''S''') = argmax P('''S'''|'''X''')

where '''S''' is a symbolic score and '''X''' is the input audio signal.

The challenge will include polyphonic recordings and may include multiple instruments. Pieces may be full recordings or shorter excerpts, such as 3 to 6 bars.

Two tracks will be considered:

* '''Staves-Informed A2S''': staves metadata is provided and may be used to guide voice identification and instrument recognition.
* '''Blind A2S''': only the audio recording is provided.

Participants may enter both tracks. Results will be reported separately.

'''Notice:''' We particularly encourage submissions that move beyond isolated AMT components and produce complete, valid symbolic scores.

= Evaluation =

Submissions will be evaluated with metrics that assess both the textual quality of the produced kern files and the musical transcription quality of the corresponding scores.

=== Score Quality ===

'''Character Error Rate''' (CER): the Levenshtein distance between the submitted and ground-truth kern files at the character level. This metric penalizes character insertions, deletions, and substitutions, and is sensitive to details such as pitch spelling and note durations.

'''Word Error Rate''' (WER): the Levenshtein distance computed on tab- and newline-separated strings in a kern file. WER is reported in recent A2S papers [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. Compared to CER, it evaluates the output closer to the note-token level, checking whether durations and pitches are jointly correct.

'''Line Error Rate''' (LER): the Levenshtein distance computed on complete lines of the output kern files. LER is stricter than CER and WER, and helps verify whether multiple instruments or voices are correctly transcribed and aligned at the beat and sub-beat level.

=== Musical Transcription Quality ===

In addition to CER, WER, and LER, this challenge will use the '''MV2H''' metric introduced in [[#bib-mcleod-2018|[4]]] and used in recent A2S work [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. MV2H is an F-score obtained as the arithmetic mean of five sub-metrics:

* '''Multi-pitch detection''': pitches must be correct and detected at onsets within a 50 ms error threshold.
* '''Voice separation''': notes belonging to the same voice or instrument should be grouped together.
* '''Metrical alignment''': bars, beats, and sub-beats should be correctly identified.
* '''Note value detection''': note durations should be correctly transcribed.
* '''Harmonic analysis''': average of a key detection score and a chord-symbol recall.

The harmonic analysis component will not be used for the initial edition because no chord information is available in the considered datasets.

Public implementations of MV2H are available:

* [https://github.com/apmcleod/MV2H Official MV2H repository]
* [https://github.com/lucasmpaim/pyMV2H Python implementation]

The evaluation pipeline will adapt these implementations to process kern files, building where appropriate on the public code from [[#bib-alfaro-2024|[1]]]:

* [https://github.com/mariaalfaroc/a2s-transformer A2S Transformer repository]

'''Note that WER and MV2H will be retained as the main ranking criteria.''' CER and LER will be reported for more detailed comparisons.

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process the evaluation set.

= Submission Format =

Several submission formats will be accepted to accommodate different system designs.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size
* The number of parameters in the model, if applicable
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time required for the evaluation set

We strongly recommend sharing the submitted algorithms or checkpoints under permissive open licenses, but any licensing (or even not sharing publicly) is accepted as long as it is clearly reported at the time of submission.

=== I / O ===

The submitted algorithm must take as input an audio file, or a folder containing audio files, and write one predicted <code>**kern</code> file for each input audio file to a specified output directory.

==== Input Audio ====

Participating algorithms will receive audio files in the following format:

* 16 bit FLAC
* 44.1 kHz for MuseSyn, 22 050 Hz for Quartets
* 30s (for Quartets) to over 7 minutes (for MuseSyn)

If your pipeline expects characteristics different from the ones described above, the conversion should be done automatically in your algorithm.

For the '''Blind A2S''' track, only the audio recording will be provided.

For the '''Staves-Informed A2S''' track, staves metadata will also be provided. This metadata is the <code>**kern</code> header, that looks like this in the Quartets dataset for example:

**kern **dynam **kern **dynam **kern **dynam **kern **dynam
*Icello *Icello *Iviola *Iviola *Ivioln *Ivioln *Iflt *Iflt
*clefF4 * *clefC3 * *clefG2 * *clefG2 *
*k[f#c#] * *k[f#c#] * *k[f#c#] * *k[f#c#] *
*D: * *D: * *D: * *D: *
*M4/4 * *M4/4 * *M4/4 * *M4/4 *
*MM130 * *MM130 * *MM130 * *MM130 *

==== Output File Format ====

Each submitted prediction must be a valid <code>**kern</code> file.

The score must encode, at minimum:

* Staves musical metadata, including meter, clef, and key signature when required
* Pitches, including octaves and accidentals
* Note durations
* Bar lines and metrical structure
* Voice or instrument organization when applicable

For more details, participants may refer to:

* [https://www.humdrum.org/Humdrum/representations/kern.html Humdrum kern representation]

We chose the <code>**kern</code> format for compatibility with existing SOTA work in A2S. Participants may use other formats during training or as intermediate outputs, but the final files should be in <code>**kern</code> format for evaluation.
Please make sure the proposed algorithm include a conversion step if it is required.

=== Code Submissions ===

Participants may provide a Docker container or access to a code repository with clear environment setup instructions and a script that reads the input audio and writes output kern files to the requested destination directory.

The submitted script should be documented in the README and should not require manual intervention during evaluation.

=== Fallback: Pre-computed Kern Submission ===

Participants may alternatively submit the final kern files directly, similarly to the option proposed for MIREX 2024 Polyphonic Transcription.

This fallback is intended to allow participation from systems that cannot be evaluated directly by the Task Captain because of resource, licensing, or infrastructure constraints.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

Participants are free to use the training and validation sets of the datasets listed below. Data augmentation is allowed.

Additional training data may also be used, as long as it does not overlap with the evaluation sets and is clearly documented in the submission.

=== Quartets Dataset ===

The Quartets dataset consists of synthetic audio renderings of Haydn, Mozart, and Beethoven string quartets, together with their full kern scores [[#bib-roman-2019|[6]]].

The scores are taken from the [https://github.com/humdrum-tools/humdrum-data humdrum-data repository]. The kern files were split into 3 to 6 measure excerpts and synthesized into audio from performance MIDI files.

The final dataset contains approximately 20 hours of audio for 38,051 excerpts and 3 composers:

* 18,162 excerpts for Haydn
* 7,435 excerpts for Mozart
* 12,454 excerpts for Beethoven

The train, validation, and test splits used in [[#bib-alfaro-2024|[1]]] are publicly available:

* [https://huggingface.co/datasets/PRAIG/quartets-quartets Quartets dataset on Hugging Face]

=== MuseSyn Dataset ===

The MuseSyn dataset contains 210 piano pieces with scores in MusicXML format and audio synthesized through four different piano models [[#bib-liu-2021|[3]]]. It amounts to almost 10 hours of audio recordings for each piano timbre.

The pieces cover a wide range of key signatures, time signatures, tempos, and polyphony levels. The dataset is available upon request for non-commercial research use:

* [https://zenodo.org/records/4527460 MuseSyn on Zenodo]

Converting the MusicXML files to <code>**kern</code> can be done using [https://github.com/adhooge/humlib/tree/gcc15-cstdint-fix this code]. Contact the task captain if you encounter any issues.

'''Please follow the splits indicated [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata here] and only use the train and validation splits during training.'''

= Evaluation Datasets =

The datasets listed below are reserved for evaluation purposes and '''must not''' be used for training models.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and could lead to data leakage.

=== Quartets Test Set ===

Evaluation will include the test split of the Quartets dataset. Results on this dataset will be reported separately.

=== MuseSyn Test Set ===

Evaluation will be conducted on the test split of the MuseSyn dataset conducted on the scores converted to kern using [https://github.com/adhooge/humlib/tree/gcc15-cstdint-fix this code]. The converted scores were verified manually and did not show any noticeable errors. Results on this dataset will be reported separately.

We use the test split as [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata defined in the official repository].

''Note:'' Some songs have lyrics written down as kern comments. These will be ignored during the evaluation process as they are not present in the audio.

=== Undisclosed Evaluation Set ===

An additional custom evaluation set will be used to ensure fairness and out-of-distribution data.

General details on this dataset are:
TBD

= Time and Hardware Limits =

Due to the potentially high number of participants in this and other audio tasks, hard limits on runtime and hardware use will be imposed.

Submissions that require more than 32 GB of VRAM, or more than 24 hours to process the test sets on a single GPU (V100 or similar), cannot be evaluated directly by the Task Captain. Participants in this situation may use the fallback kern submission format described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Alfaro-Contreras, M., et al. (2024). A Transformer Approach for Polyphonic Audio-to-Score Transcription. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2] Benetos, E., et al. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1).

[3] Liu, L., et al. (2021). Joint Multi-Pitch Detection and Score Transcription for Polyphonic Piano Music. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4] Mcleod, A., et al. (2018). Evaluating Automatic Polyphonic Music Transcription. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[5] Roman, M. A., et al. (2018). An End-to-End Framework for Audio-to-Score Music Transcription on Monophonic Excerpts. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[6] Roman, M. A., et al. (2019). A Holistic Approach to Polyphonic Music Transcription with Neural Networks. Proc. of the 20th International Society for Music Information Retrieval Conference (ISMIR).

[7] Smaragdis, P., et al. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.

2026:Audio-to-Score Transcription

2026-06-29T07:51:08Z

Alexandre DHooge: /* MuseSyn Test Set */

= Description =

This page describes the '''MIREX2026: Audio-to-Score Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.

Automatic Music Transcription (AMT) is the task of designing computational algorithms that convert acoustic music signals into a symbolic musical representation [[#bib-benetos-2019|[2]]]. Existing MIREX tasks have evaluated several important AMT components, including melody extraction, drum detection, piano transcription, and multi-instrument transcription. However, these tasks have generally stopped at MIDI or note-event outputs, which are not equivalent to a well-formed musical score.

The Audio-to-Score Transcription (A2S) challenge focuses on this missing step. Participating systems will receive an audio recording of a polyphonic music piece and must output a digital symbolic score that can be read by a musician or notation program. The target output is a kern score encoding standard musical information such as pitches, durations, accidentals, staves, meter, clef, key signature, and bar structure.

The task can be informally expressed as follows:

Prediction('''S''') = argmax P('''S'''|'''X''')

where '''S''' is a symbolic score and '''X''' is the input audio signal.

The challenge will include polyphonic recordings and may include multiple instruments. Pieces may be full recordings or shorter excerpts, such as 3 to 6 bars.

Two tracks will be considered:

* '''Staves-Informed A2S''': staves metadata is provided and may be used to guide voice identification and instrument recognition.
* '''Blind A2S''': only the audio recording is provided.

Participants may enter both tracks. Results will be reported separately.

'''Notice:''' We particularly encourage submissions that move beyond isolated AMT components and produce complete, valid symbolic scores.

= Evaluation =

Submissions will be evaluated with metrics that assess both the textual quality of the produced kern files and the musical transcription quality of the corresponding scores.

=== Score Quality ===

'''Character Error Rate''' (CER): the Levenshtein distance between the submitted and ground-truth kern files at the character level. This metric penalizes character insertions, deletions, and substitutions, and is sensitive to details such as pitch spelling and note durations.

'''Word Error Rate''' (WER): the Levenshtein distance computed on tab- and newline-separated strings in a kern file. WER is reported in recent A2S papers [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. Compared to CER, it evaluates the output closer to the note-token level, checking whether durations and pitches are jointly correct.

'''Line Error Rate''' (LER): the Levenshtein distance computed on complete lines of the output kern files. LER is stricter than CER and WER, and helps verify whether multiple instruments or voices are correctly transcribed and aligned at the beat and sub-beat level.

=== Musical Transcription Quality ===

In addition to CER, WER, and LER, this challenge will use the '''MV2H''' metric introduced in [[#bib-mcleod-2018|[4]]] and used in recent A2S work [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. MV2H is an F-score obtained as the arithmetic mean of five sub-metrics:

* '''Multi-pitch detection''': pitches must be correct and detected at onsets within a 50 ms error threshold.
* '''Voice separation''': notes belonging to the same voice or instrument should be grouped together.
* '''Metrical alignment''': bars, beats, and sub-beats should be correctly identified.
* '''Note value detection''': note durations should be correctly transcribed.
* '''Harmonic analysis''': average of a key detection score and a chord-symbol recall.

The harmonic analysis component will not be used for the initial edition because no chord information is available in the considered datasets.

Public implementations of MV2H are available:

* [https://github.com/apmcleod/MV2H Official MV2H repository]
* [https://github.com/lucasmpaim/pyMV2H Python implementation]

The evaluation pipeline will adapt these implementations to process kern files, building where appropriate on the public code from [[#bib-alfaro-2024|[1]]]:

* [https://github.com/mariaalfaroc/a2s-transformer A2S Transformer repository]

'''Note that WER and MV2H will be retained as the main ranking criteria.''' CER and LER will be reported for more detailed comparisons.

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process the evaluation set.

= Submission Format =

Several submission formats will be accepted to accommodate different system designs.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size
* The number of parameters in the model, if applicable
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time required for the evaluation set

We strongly recommend sharing the submitted algorithms or checkpoints under permissive open licenses, but any licensing (or even not sharing publicly) is accepted as long as it is clearly reported at the time of submission.

=== I / O ===

The submitted algorithm must take as input an audio file, or a folder containing audio files, and write one predicted <code>**kern</code> file for each input audio file to a specified output directory.

==== Input Audio ====

Participating algorithms will receive audio files in the following format:

* 16 bit FLAC
* 44.1 kHz for MuseSyn, 22 050 Hz for Quartets
* 30s (for Quartets) to over 7 minutes (for MuseSyn)

If your pipeline expects characteristics different from the ones described above, the conversion should be done automatically in your algorithm.

For the '''Blind A2S''' track, only the audio recording will be provided.

For the '''Staves-Informed A2S''' track, staves metadata will also be provided. This metadata is the <code>**kern</code> header, that looks like this in the Quartets dataset for example:

**kern **dynam **kern **dynam **kern **dynam **kern **dynam
*Icello *Icello *Iviola *Iviola *Ivioln *Ivioln *Iflt *Iflt
*clefF4 * *clefC3 * *clefG2 * *clefG2 *
*k[f#c#] * *k[f#c#] * *k[f#c#] * *k[f#c#] *
*D: * *D: * *D: * *D: *
*M4/4 * *M4/4 * *M4/4 * *M4/4 *
*MM130 * *MM130 * *MM130 * *MM130 *

==== Output File Format ====

Each submitted prediction must be a valid <code>**kern</code> file.

The score must encode, at minimum:

* Staves musical metadata, including meter, clef, and key signature when required
* Pitches, including octaves and accidentals
* Note durations
* Bar lines and metrical structure
* Voice or instrument organization when applicable

For more details, participants may refer to:

* [https://www.humdrum.org/Humdrum/representations/kern.html Humdrum kern representation]

We chose the <code>**kern</code> format for compatibility with existing SOTA work in A2S. Participants may use other formats during training or as intermediate outputs, but the final files should be in <code>**kern</code> format for evaluation.
Please make sure the proposed algorithm include a conversion step if it is required.

=== Code Submissions ===

Participants may provide a Docker container or access to a code repository with clear environment setup instructions and a script that reads the input audio and writes output kern files to the requested destination directory.

The submitted script should be documented in the README and should not require manual intervention during evaluation.

=== Fallback: Pre-computed Kern Submission ===

Participants may alternatively submit the final kern files directly, similarly to the option proposed for MIREX 2024 Polyphonic Transcription.

This fallback is intended to allow participation from systems that cannot be evaluated directly by the Task Captain because of resource, licensing, or infrastructure constraints.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

Participants are free to use the training and validation sets of the datasets listed below. Data augmentation is allowed.

Additional training data may also be used, as long as it does not overlap with the evaluation sets and is clearly documented in the submission.

=== Quartets Dataset ===

The Quartets dataset consists of synthetic audio renderings of Haydn, Mozart, and Beethoven string quartets, together with their full kern scores [[#bib-roman-2019|[6]]].

The scores are taken from the [https://github.com/humdrum-tools/humdrum-data humdrum-data repository]. The kern files were split into 3 to 6 measure excerpts and synthesized into audio from performance MIDI files.

The final dataset contains approximately 20 hours of audio for 38,051 excerpts and 3 composers:

* 18,162 excerpts for Haydn
* 7,435 excerpts for Mozart
* 12,454 excerpts for Beethoven

The train, validation, and test splits used in [[#bib-alfaro-2024|[1]]] are publicly available:

* [https://huggingface.co/datasets/PRAIG/quartets-quartets Quartets dataset on Hugging Face]

=== MuseSyn Dataset ===

The MuseSyn dataset contains 210 piano pieces with scores in MusicXML format and audio synthesized through four different piano models [[#bib-liu-2021|[3]]]. It amounts to almost 10 hours of audio recordings for each piano timbre.

The pieces cover a wide range of key signatures, time signatures, tempos, and polyphony levels. The dataset is available upon request for non-commercial research use:

* [https://zenodo.org/records/4527460 MuseSyn on Zenodo]

Converting the MusicXML files to <code>**kern</code> can be done using [https://github.com/adhooge/humlib/tree/gcc15-cstdint-fix this code]. Contact the task captain if you encounter any issues.

'''Please follow the splits indicated [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata here] and only use the train and validation splits during training.'''

= Evaluation Datasets =

The datasets listed below are reserved for evaluation purposes and '''must not''' be used for training models.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and could lead to data leakage.

=== Quartets Test Set ===

Evaluation will include the test split of the Quartets dataset. Results on this dataset will be reported separately.

=== MuseSyn Test Set ===

Evaluation will include the test split of the MuseSyn dataset after conversion of the reference scores to kern format. Results on this dataset will be reported separately.

We use the test split as [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata defined in the official repository].

''Note:'' Some songs have lyrics written down as kern comments. These will be ignored during the evaluation process as they are not present in the audio.

=== Undisclosed Evaluation Set ===

An additional custom evaluation set will be used to ensure fairness and out-of-distribution data.

General details on this dataset are:
TBD

= Time and Hardware Limits =

Due to the potentially high number of participants in this and other audio tasks, hard limits on runtime and hardware use will be imposed.

Submissions that require more than 32 GB of VRAM, or more than 24 hours to process the test sets on a single GPU (V100 or similar), cannot be evaluated directly by the Task Captain. Participants in this situation may use the fallback kern submission format described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Alfaro-Contreras, M., et al. (2024). A Transformer Approach for Polyphonic Audio-to-Score Transcription. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2] Benetos, E., et al. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1).

[3] Liu, L., et al. (2021). Joint Multi-Pitch Detection and Score Transcription for Polyphonic Piano Music. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4] Mcleod, A., et al. (2018). Evaluating Automatic Polyphonic Music Transcription. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[5] Roman, M. A., et al. (2018). An End-to-End Framework for Audio-to-Score Music Transcription on Monophonic Excerpts. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[6] Roman, M. A., et al. (2019). A Holistic Approach to Polyphonic Music Transcription with Neural Networks. Proc. of the 20th International Society for Music Information Retrieval Conference (ISMIR).

[7] Smaragdis, P., et al. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.

2026:Audio-to-Score Transcription

2026-06-29T07:35:34Z

Alexandre DHooge: /* Input Audio */ add kern header example

= Description =

This page describes the '''MIREX2026: Audio-to-Score Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.

Automatic Music Transcription (AMT) is the task of designing computational algorithms that convert acoustic music signals into a symbolic musical representation [[#bib-benetos-2019|[2]]]. Existing MIREX tasks have evaluated several important AMT components, including melody extraction, drum detection, piano transcription, and multi-instrument transcription. However, these tasks have generally stopped at MIDI or note-event outputs, which are not equivalent to a well-formed musical score.

The Audio-to-Score Transcription (A2S) challenge focuses on this missing step. Participating systems will receive an audio recording of a polyphonic music piece and must output a digital symbolic score that can be read by a musician or notation program. The target output is a kern score encoding standard musical information such as pitches, durations, accidentals, staves, meter, clef, key signature, and bar structure.

The task can be informally expressed as follows:

Prediction('''S''') = argmax P('''S'''|'''X''')

where '''S''' is a symbolic score and '''X''' is the input audio signal.

The challenge will include polyphonic recordings and may include multiple instruments. Pieces may be full recordings or shorter excerpts, such as 3 to 6 bars.

Two tracks will be considered:

* '''Staves-Informed A2S''': staves metadata is provided and may be used to guide voice identification and instrument recognition.
* '''Blind A2S''': only the audio recording is provided.

Participants may enter both tracks. Results will be reported separately.

'''Notice:''' We particularly encourage submissions that move beyond isolated AMT components and produce complete, valid symbolic scores.

= Evaluation =

Submissions will be evaluated with metrics that assess both the textual quality of the produced kern files and the musical transcription quality of the corresponding scores.

=== Score Quality ===

'''Character Error Rate''' (CER): the Levenshtein distance between the submitted and ground-truth kern files at the character level. This metric penalizes character insertions, deletions, and substitutions, and is sensitive to details such as pitch spelling and note durations.

'''Word Error Rate''' (WER): the Levenshtein distance computed on tab- and newline-separated strings in a kern file. WER is reported in recent A2S papers [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. Compared to CER, it evaluates the output closer to the note-token level, checking whether durations and pitches are jointly correct.

'''Line Error Rate''' (LER): the Levenshtein distance computed on complete lines of the output kern files. LER is stricter than CER and WER, and helps verify whether multiple instruments or voices are correctly transcribed and aligned at the beat and sub-beat level.

=== Musical Transcription Quality ===

In addition to CER, WER, and LER, this challenge will use the '''MV2H''' metric introduced in [[#bib-mcleod-2018|[4]]] and used in recent A2S work [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. MV2H is an F-score obtained as the arithmetic mean of five sub-metrics:

* '''Multi-pitch detection''': pitches must be correct and detected at onsets within a 50 ms error threshold.
* '''Voice separation''': notes belonging to the same voice or instrument should be grouped together.
* '''Metrical alignment''': bars, beats, and sub-beats should be correctly identified.
* '''Note value detection''': note durations should be correctly transcribed.
* '''Harmonic analysis''': average of a key detection score and a chord-symbol recall.

The harmonic analysis component will not be used for the initial edition because no chord information is available in the considered datasets.

Public implementations of MV2H are available:

* [https://github.com/apmcleod/MV2H Official MV2H repository]
* [https://github.com/lucasmpaim/pyMV2H Python implementation]

The evaluation pipeline will adapt these implementations to process kern files, building where appropriate on the public code from [[#bib-alfaro-2024|[1]]]:

* [https://github.com/mariaalfaroc/a2s-transformer A2S Transformer repository]

'''Note that WER and MV2H will be retained as the main ranking criteria.''' CER and LER will be reported for more detailed comparisons.

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process the evaluation set.

= Submission Format =

Several submission formats will be accepted to accommodate different system designs.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size
* The number of parameters in the model, if applicable
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time required for the evaluation set

We strongly recommend sharing the submitted algorithms or checkpoints under permissive open licenses, but any licensing (or even not sharing publicly) is accepted as long as it is clearly reported at the time of submission.

=== I / O ===

The submitted algorithm must take as input an audio file, or a folder containing audio files, and write one predicted <code>**kern</code> file for each input audio file to a specified output directory.

==== Input Audio ====

Participating algorithms will receive audio files in the following format:

* 16 bit FLAC
* 44.1 kHz for MuseSyn, 22 050 Hz for Quartets
* 30s (for Quartets) to over 7 minutes (for MuseSyn)

If your pipeline expects characteristics different from the ones described above, the conversion should be done automatically in your algorithm.

For the '''Blind A2S''' track, only the audio recording will be provided.

For the '''Staves-Informed A2S''' track, staves metadata will also be provided. This metadata is the <code>**kern</code> header, that looks like this in the Quartets dataset for example:

**kern **dynam **kern **dynam **kern **dynam **kern **dynam
*Icello *Icello *Iviola *Iviola *Ivioln *Ivioln *Iflt *Iflt
*clefF4 * *clefC3 * *clefG2 * *clefG2 *
*k[f#c#] * *k[f#c#] * *k[f#c#] * *k[f#c#] *
*D: * *D: * *D: * *D: *
*M4/4 * *M4/4 * *M4/4 * *M4/4 *
*MM130 * *MM130 * *MM130 * *MM130 *

==== Output File Format ====

Each submitted prediction must be a valid <code>**kern</code> file.

The score must encode, at minimum:

* Staves musical metadata, including meter, clef, and key signature when required
* Pitches, including octaves and accidentals
* Note durations
* Bar lines and metrical structure
* Voice or instrument organization when applicable

For more details, participants may refer to:

* [https://www.humdrum.org/Humdrum/representations/kern.html Humdrum kern representation]

We chose the <code>**kern</code> format for compatibility with existing SOTA work in A2S. Participants may use other formats during training or as intermediate outputs, but the final files should be in <code>**kern</code> format for evaluation.
Please make sure the proposed algorithm include a conversion step if it is required.

=== Code Submissions ===

Participants may provide a Docker container or access to a code repository with clear environment setup instructions and a script that reads the input audio and writes output kern files to the requested destination directory.

The submitted script should be documented in the README and should not require manual intervention during evaluation.

=== Fallback: Pre-computed Kern Submission ===

Participants may alternatively submit the final kern files directly, similarly to the option proposed for MIREX 2024 Polyphonic Transcription.

This fallback is intended to allow participation from systems that cannot be evaluated directly by the Task Captain because of resource, licensing, or infrastructure constraints.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

Participants are free to use the training and validation sets of the datasets listed below. Data augmentation is allowed.

Additional training data may also be used, as long as it does not overlap with the evaluation sets and is clearly documented in the submission.

=== Quartets Dataset ===

The Quartets dataset consists of synthetic audio renderings of Haydn, Mozart, and Beethoven string quartets, together with their full kern scores [[#bib-roman-2019|[6]]].

The scores are taken from the [https://github.com/humdrum-tools/humdrum-data humdrum-data repository]. The kern files were split into 3 to 6 measure excerpts and synthesized into audio from performance MIDI files.

The final dataset contains approximately 20 hours of audio for 38,051 excerpts and 3 composers:

* 18,162 excerpts for Haydn
* 7,435 excerpts for Mozart
* 12,454 excerpts for Beethoven

The train, validation, and test splits used in [[#bib-alfaro-2024|[1]]] are publicly available:

* [https://huggingface.co/datasets/PRAIG/quartets-quartets Quartets dataset on Hugging Face]

=== MuseSyn Dataset ===

The MuseSyn dataset contains 210 piano pieces with scores in MusicXML format and audio synthesized through four different piano models [[#bib-liu-2021|[3]]]. It amounts to almost 10 hours of audio recordings for each piano timbre.

The pieces cover a wide range of key signatures, time signatures, tempos, and polyphony levels. The dataset is available upon request for non-commercial research use:

* [https://zenodo.org/records/4527460 MuseSyn on Zenodo]

Converting the MusicXML files to <code>**kern</code> can be done using [https://github.com/adhooge/humlib/tree/gcc15-cstdint-fix this code]. Contact the task captain if you encounter any issues.

'''Please follow the splits indicated [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata here] and only use the train and validation splits during training.'''

= Evaluation Datasets =

The datasets listed below are reserved for evaluation purposes and '''must not''' be used for training models.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and could lead to data leakage.

=== Quartets Test Set ===

Evaluation will include the test split of the Quartets dataset. Results on this dataset will be reported separately.

=== MuseSyn Test Set ===

Evaluation will include the test split of the MuseSyn dataset after conversion of the reference scores to kern format. Results on this dataset will be reported separately.

We use the test split as [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata defined in the official repository].

=== Undisclosed Evaluation Set ===

An additional custom evaluation set will be used to ensure fairness and out-of-distribution data.

General details on this dataset are:
TBD

= Time and Hardware Limits =

Due to the potentially high number of participants in this and other audio tasks, hard limits on runtime and hardware use will be imposed.

Submissions that require more than 32 GB of VRAM, or more than 24 hours to process the test sets on a single GPU (V100 or similar), cannot be evaluated directly by the Task Captain. Participants in this situation may use the fallback kern submission format described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Alfaro-Contreras, M., et al. (2024). A Transformer Approach for Polyphonic Audio-to-Score Transcription. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2] Benetos, E., et al. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1).

[3] Liu, L., et al. (2021). Joint Multi-Pitch Detection and Score Transcription for Polyphonic Piano Music. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4] Mcleod, A., et al. (2018). Evaluating Automatic Polyphonic Music Transcription. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[5] Roman, M. A., et al. (2018). An End-to-End Framework for Audio-to-Score Music Transcription on Monophonic Excerpts. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[6] Roman, M. A., et al. (2019). A Holistic Approach to Polyphonic Music Transcription with Neural Networks. Proc. of the 20th International Society for Music Information Retrieval Conference (ISMIR).

[7] Smaragdis, P., et al. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.

2026:Audio-to-Score Transcription

2026-06-29T07:30:47Z

Alexandre DHooge: /* MuseSyn Dataset */ refer to correct conversion repo

= Description =

This page describes the '''MIREX2026: Audio-to-Score Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.

Automatic Music Transcription (AMT) is the task of designing computational algorithms that convert acoustic music signals into a symbolic musical representation [[#bib-benetos-2019|[2]]]. Existing MIREX tasks have evaluated several important AMT components, including melody extraction, drum detection, piano transcription, and multi-instrument transcription. However, these tasks have generally stopped at MIDI or note-event outputs, which are not equivalent to a well-formed musical score.

The Audio-to-Score Transcription (A2S) challenge focuses on this missing step. Participating systems will receive an audio recording of a polyphonic music piece and must output a digital symbolic score that can be read by a musician or notation program. The target output is a kern score encoding standard musical information such as pitches, durations, accidentals, staves, meter, clef, key signature, and bar structure.

The task can be informally expressed as follows:

Prediction('''S''') = argmax P('''S'''|'''X''')

where '''S''' is a symbolic score and '''X''' is the input audio signal.

The challenge will include polyphonic recordings and may include multiple instruments. Pieces may be full recordings or shorter excerpts, such as 3 to 6 bars.

Two tracks will be considered:

* '''Staves-Informed A2S''': staves metadata is provided and may be used to guide voice identification and instrument recognition.
* '''Blind A2S''': only the audio recording is provided.

Participants may enter both tracks. Results will be reported separately.

'''Notice:''' We particularly encourage submissions that move beyond isolated AMT components and produce complete, valid symbolic scores.

= Evaluation =

Submissions will be evaluated with metrics that assess both the textual quality of the produced kern files and the musical transcription quality of the corresponding scores.

=== Score Quality ===

'''Character Error Rate''' (CER): the Levenshtein distance between the submitted and ground-truth kern files at the character level. This metric penalizes character insertions, deletions, and substitutions, and is sensitive to details such as pitch spelling and note durations.

'''Word Error Rate''' (WER): the Levenshtein distance computed on tab- and newline-separated strings in a kern file. WER is reported in recent A2S papers [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. Compared to CER, it evaluates the output closer to the note-token level, checking whether durations and pitches are jointly correct.

'''Line Error Rate''' (LER): the Levenshtein distance computed on complete lines of the output kern files. LER is stricter than CER and WER, and helps verify whether multiple instruments or voices are correctly transcribed and aligned at the beat and sub-beat level.

=== Musical Transcription Quality ===

In addition to CER, WER, and LER, this challenge will use the '''MV2H''' metric introduced in [[#bib-mcleod-2018|[4]]] and used in recent A2S work [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. MV2H is an F-score obtained as the arithmetic mean of five sub-metrics:

* '''Multi-pitch detection''': pitches must be correct and detected at onsets within a 50 ms error threshold.
* '''Voice separation''': notes belonging to the same voice or instrument should be grouped together.
* '''Metrical alignment''': bars, beats, and sub-beats should be correctly identified.
* '''Note value detection''': note durations should be correctly transcribed.
* '''Harmonic analysis''': average of a key detection score and a chord-symbol recall.

The harmonic analysis component will not be used for the initial edition because no chord information is available in the considered datasets.

Public implementations of MV2H are available:

* [https://github.com/apmcleod/MV2H Official MV2H repository]
* [https://github.com/lucasmpaim/pyMV2H Python implementation]

The evaluation pipeline will adapt these implementations to process kern files, building where appropriate on the public code from [[#bib-alfaro-2024|[1]]]:

* [https://github.com/mariaalfaroc/a2s-transformer A2S Transformer repository]

'''Note that WER and MV2H will be retained as the main ranking criteria.''' CER and LER will be reported for more detailed comparisons.

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process the evaluation set.

= Submission Format =

Several submission formats will be accepted to accommodate different system designs.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size
* The number of parameters in the model, if applicable
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time required for the evaluation set

We strongly recommend sharing the submitted algorithms or checkpoints under permissive open licenses, but any licensing (or even not sharing publicly) is accepted as long as it is clearly reported at the time of submission.

=== I / O ===

The submitted algorithm must take as input an audio file, or a folder containing audio files, and write one predicted <code>**kern</code> file for each input audio file to a specified output directory.

==== Input Audio ====

Participating algorithms will receive audio files in the following format:

* 16 bit FLAC
* 44.1 kHz for MuseSyn, 22 050 Hz for Quartets
* 30s (for Quartets) to over 7 minutes (for MuseSyn)

If your pipeline expects characteristics different from the ones described above, the conversion should be done automatically in your algorithm.

For the '''Blind A2S''' track, only the audio recording will be provided.

For the '''Staves-Informed A2S''' track, staves metadata will also be provided. This metadata may include meter, clef, key signature, and other score-structure information needed to guide transcription. The exact metadata packaging will be specified before submissions open.

==== Output File Format ====

Each submitted prediction must be a valid <code>**kern</code> file.

The score must encode, at minimum:

* Staves musical metadata, including meter, clef, and key signature when required
* Pitches, including octaves and accidentals
* Note durations
* Bar lines and metrical structure
* Voice or instrument organization when applicable

For more details, participants may refer to:

* [https://www.humdrum.org/Humdrum/representations/kern.html Humdrum kern representation]

We chose the <code>**kern</code> format for compatibility with existing SOTA work in A2S. Participants may use other formats during training or as intermediate outputs, but the final files should be in <code>**kern</code> format for evaluation.
Please make sure the proposed algorithm include a conversion step if it is required.

=== Code Submissions ===

Participants may provide a Docker container or access to a code repository with clear environment setup instructions and a script that reads the input audio and writes output kern files to the requested destination directory.

The submitted script should be documented in the README and should not require manual intervention during evaluation.

=== Fallback: Pre-computed Kern Submission ===

Participants may alternatively submit the final kern files directly, similarly to the option proposed for MIREX 2024 Polyphonic Transcription.

This fallback is intended to allow participation from systems that cannot be evaluated directly by the Task Captain because of resource, licensing, or infrastructure constraints.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

Participants are free to use the training and validation sets of the datasets listed below. Data augmentation is allowed.

Additional training data may also be used, as long as it does not overlap with the evaluation sets and is clearly documented in the submission.

=== Quartets Dataset ===

The Quartets dataset consists of synthetic audio renderings of Haydn, Mozart, and Beethoven string quartets, together with their full kern scores [[#bib-roman-2019|[6]]].

The scores are taken from the [https://github.com/humdrum-tools/humdrum-data humdrum-data repository]. The kern files were split into 3 to 6 measure excerpts and synthesized into audio from performance MIDI files.

The final dataset contains approximately 20 hours of audio for 38,051 excerpts and 3 composers:

* 18,162 excerpts for Haydn
* 7,435 excerpts for Mozart
* 12,454 excerpts for Beethoven

The train, validation, and test splits used in [[#bib-alfaro-2024|[1]]] are publicly available:

* [https://huggingface.co/datasets/PRAIG/quartets-quartets Quartets dataset on Hugging Face]

=== MuseSyn Dataset ===

The MuseSyn dataset contains 210 piano pieces with scores in MusicXML format and audio synthesized through four different piano models [[#bib-liu-2021|[3]]]. It amounts to almost 10 hours of audio recordings for each piano timbre.

The pieces cover a wide range of key signatures, time signatures, tempos, and polyphony levels. The dataset is available upon request for non-commercial research use:

* [https://zenodo.org/records/4527460 MuseSyn on Zenodo]

Converting the MusicXML files to <code>**kern</code> can be done using [https://github.com/adhooge/humlib/tree/gcc15-cstdint-fix this code]. Contact the task captain if you encounter any issues.

'''Please follow the splits indicated [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata here] and only use the train and validation splits during training.'''

= Evaluation Datasets =

The datasets listed below are reserved for evaluation purposes and '''must not''' be used for training models.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and could lead to data leakage.

=== Quartets Test Set ===

Evaluation will include the test split of the Quartets dataset. Results on this dataset will be reported separately.

=== MuseSyn Test Set ===

Evaluation will include the test split of the MuseSyn dataset after conversion of the reference scores to kern format. Results on this dataset will be reported separately.

We use the test split as [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata defined in the official repository].

=== Undisclosed Evaluation Set ===

An additional custom evaluation set will be used to ensure fairness and out-of-distribution data.

General details on this dataset are:
TBD

= Time and Hardware Limits =

Due to the potentially high number of participants in this and other audio tasks, hard limits on runtime and hardware use will be imposed.

Submissions that require more than 32 GB of VRAM, or more than 24 hours to process the test sets on a single GPU (V100 or similar), cannot be evaluated directly by the Task Captain. Participants in this situation may use the fallback kern submission format described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Alfaro-Contreras, M., et al. (2024). A Transformer Approach for Polyphonic Audio-to-Score Transcription. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2] Benetos, E., et al. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1).

[3] Liu, L., et al. (2021). Joint Multi-Pitch Detection and Score Transcription for Polyphonic Piano Music. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4] Mcleod, A., et al. (2018). Evaluating Automatic Polyphonic Music Transcription. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[5] Roman, M. A., et al. (2018). An End-to-End Framework for Audio-to-Score Music Transcription on Monophonic Excerpts. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[6] Roman, M. A., et al. (2019). A Holistic Approach to Polyphonic Music Transcription with Neural Networks. Proc. of the 20th International Society for Music Information Retrieval Conference (ISMIR).

[7] Smaragdis, P., et al. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.

2026:Audio-to-Score Transcription

2026-06-29T07:29:32Z

Alexandre DHooge: /* Input Audio */ add audio format information

= Description =

This page describes the '''MIREX2026: Audio-to-Score Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.

Automatic Music Transcription (AMT) is the task of designing computational algorithms that convert acoustic music signals into a symbolic musical representation [[#bib-benetos-2019|[2]]]. Existing MIREX tasks have evaluated several important AMT components, including melody extraction, drum detection, piano transcription, and multi-instrument transcription. However, these tasks have generally stopped at MIDI or note-event outputs, which are not equivalent to a well-formed musical score.

The Audio-to-Score Transcription (A2S) challenge focuses on this missing step. Participating systems will receive an audio recording of a polyphonic music piece and must output a digital symbolic score that can be read by a musician or notation program. The target output is a kern score encoding standard musical information such as pitches, durations, accidentals, staves, meter, clef, key signature, and bar structure.

The task can be informally expressed as follows:

Prediction('''S''') = argmax P('''S'''|'''X''')

where '''S''' is a symbolic score and '''X''' is the input audio signal.

The challenge will include polyphonic recordings and may include multiple instruments. Pieces may be full recordings or shorter excerpts, such as 3 to 6 bars.

Two tracks will be considered:

* '''Staves-Informed A2S''': staves metadata is provided and may be used to guide voice identification and instrument recognition.
* '''Blind A2S''': only the audio recording is provided.

Participants may enter both tracks. Results will be reported separately.

'''Notice:''' We particularly encourage submissions that move beyond isolated AMT components and produce complete, valid symbolic scores.

= Evaluation =

Submissions will be evaluated with metrics that assess both the textual quality of the produced kern files and the musical transcription quality of the corresponding scores.

=== Score Quality ===

'''Character Error Rate''' (CER): the Levenshtein distance between the submitted and ground-truth kern files at the character level. This metric penalizes character insertions, deletions, and substitutions, and is sensitive to details such as pitch spelling and note durations.

'''Word Error Rate''' (WER): the Levenshtein distance computed on tab- and newline-separated strings in a kern file. WER is reported in recent A2S papers [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. Compared to CER, it evaluates the output closer to the note-token level, checking whether durations and pitches are jointly correct.

'''Line Error Rate''' (LER): the Levenshtein distance computed on complete lines of the output kern files. LER is stricter than CER and WER, and helps verify whether multiple instruments or voices are correctly transcribed and aligned at the beat and sub-beat level.

=== Musical Transcription Quality ===

In addition to CER, WER, and LER, this challenge will use the '''MV2H''' metric introduced in [[#bib-mcleod-2018|[4]]] and used in recent A2S work [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. MV2H is an F-score obtained as the arithmetic mean of five sub-metrics:

* '''Multi-pitch detection''': pitches must be correct and detected at onsets within a 50 ms error threshold.
* '''Voice separation''': notes belonging to the same voice or instrument should be grouped together.
* '''Metrical alignment''': bars, beats, and sub-beats should be correctly identified.
* '''Note value detection''': note durations should be correctly transcribed.
* '''Harmonic analysis''': average of a key detection score and a chord-symbol recall.

The harmonic analysis component will not be used for the initial edition because no chord information is available in the considered datasets.

Public implementations of MV2H are available:

* [https://github.com/apmcleod/MV2H Official MV2H repository]
* [https://github.com/lucasmpaim/pyMV2H Python implementation]

The evaluation pipeline will adapt these implementations to process kern files, building where appropriate on the public code from [[#bib-alfaro-2024|[1]]]:

* [https://github.com/mariaalfaroc/a2s-transformer A2S Transformer repository]

'''Note that WER and MV2H will be retained as the main ranking criteria.''' CER and LER will be reported for more detailed comparisons.

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process the evaluation set.

= Submission Format =

Several submission formats will be accepted to accommodate different system designs.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size
* The number of parameters in the model, if applicable
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time required for the evaluation set

We strongly recommend sharing the submitted algorithms or checkpoints under permissive open licenses, but any licensing (or even not sharing publicly) is accepted as long as it is clearly reported at the time of submission.

=== I / O ===

The submitted algorithm must take as input an audio file, or a folder containing audio files, and write one predicted <code>**kern</code> file for each input audio file to a specified output directory.

==== Input Audio ====

Participating algorithms will receive audio files in the following format:

* 16 bit FLAC
* 44.1 kHz for MuseSyn, 22 050 Hz for Quartets
* 30s (for Quartets) to over 7 minutes (for MuseSyn)

If your pipeline expects characteristics different from the ones described above, the conversion should be done automatically in your algorithm.

For the '''Blind A2S''' track, only the audio recording will be provided.

For the '''Staves-Informed A2S''' track, staves metadata will also be provided. This metadata may include meter, clef, key signature, and other score-structure information needed to guide transcription. The exact metadata packaging will be specified before submissions open.

==== Output File Format ====

Each submitted prediction must be a valid <code>**kern</code> file.

The score must encode, at minimum:

* Staves musical metadata, including meter, clef, and key signature when required
* Pitches, including octaves and accidentals
* Note durations
* Bar lines and metrical structure
* Voice or instrument organization when applicable

For more details, participants may refer to:

* [https://www.humdrum.org/Humdrum/representations/kern.html Humdrum kern representation]

We chose the <code>**kern</code> format for compatibility with existing SOTA work in A2S. Participants may use other formats during training or as intermediate outputs, but the final files should be in <code>**kern</code> format for evaluation.
Please make sure the proposed algorithm include a conversion step if it is required.

=== Code Submissions ===

Participants may provide a Docker container or access to a code repository with clear environment setup instructions and a script that reads the input audio and writes output kern files to the requested destination directory.

The submitted script should be documented in the README and should not require manual intervention during evaluation.

=== Fallback: Pre-computed Kern Submission ===

Participants may alternatively submit the final kern files directly, similarly to the option proposed for MIREX 2024 Polyphonic Transcription.

This fallback is intended to allow participation from systems that cannot be evaluated directly by the Task Captain because of resource, licensing, or infrastructure constraints.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

Participants are free to use the training and validation sets of the datasets listed below. Data augmentation is allowed.

Additional training data may also be used, as long as it does not overlap with the evaluation sets and is clearly documented in the submission.

=== Quartets Dataset ===

The Quartets dataset consists of synthetic audio renderings of Haydn, Mozart, and Beethoven string quartets, together with their full kern scores [[#bib-roman-2019|[6]]].

The scores are taken from the [https://github.com/humdrum-tools/humdrum-data humdrum-data repository]. The kern files were split into 3 to 6 measure excerpts and synthesized into audio from performance MIDI files.

The final dataset contains approximately 20 hours of audio for 38,051 excerpts and 3 composers:

* 18,162 excerpts for Haydn
* 7,435 excerpts for Mozart
* 12,454 excerpts for Beethoven

The train, validation, and test splits used in [[#bib-alfaro-2024|[1]]] are publicly available:

* [https://huggingface.co/datasets/PRAIG/quartets-quartets Quartets dataset on Hugging Face]

=== MuseSyn Dataset ===

The MuseSyn dataset contains 210 piano pieces with scores in MusicXML format and audio synthesized through four different piano models [[#bib-liu-2021|[3]]]. It amounts to almost 10 hours of audio recordings for each piano timbre.

The pieces cover a wide range of key signatures, time signatures, tempos, and polyphony levels. The dataset is available upon request for non-commercial research use:

* [https://zenodo.org/records/4527460 MuseSyn on Zenodo]

Converting the MusicXML files to <code>**kern</code> can be done using [https://extra.humdrum.org/man/xml2hum/ xml2hum]. Contact the task captain if you encounter any issues.

'''Please follow the splits indicated [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata here] and only use the train and validation splits during training.'''

= Evaluation Datasets =

The datasets listed below are reserved for evaluation purposes and '''must not''' be used for training models.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and could lead to data leakage.

=== Quartets Test Set ===

Evaluation will include the test split of the Quartets dataset. Results on this dataset will be reported separately.

=== MuseSyn Test Set ===

Evaluation will include the test split of the MuseSyn dataset after conversion of the reference scores to kern format. Results on this dataset will be reported separately.

We use the test split as [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata defined in the official repository].

=== Undisclosed Evaluation Set ===

An additional custom evaluation set will be used to ensure fairness and out-of-distribution data.

General details on this dataset are:
TBD

= Time and Hardware Limits =

Due to the potentially high number of participants in this and other audio tasks, hard limits on runtime and hardware use will be imposed.

Submissions that require more than 32 GB of VRAM, or more than 24 hours to process the test sets on a single GPU (V100 or similar), cannot be evaluated directly by the Task Captain. Participants in this situation may use the fallback kern submission format described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Alfaro-Contreras, M., et al. (2024). A Transformer Approach for Polyphonic Audio-to-Score Transcription. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2] Benetos, E., et al. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1).

[3] Liu, L., et al. (2021). Joint Multi-Pitch Detection and Score Transcription for Polyphonic Piano Music. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4] Mcleod, A., et al. (2018). Evaluating Automatic Polyphonic Music Transcription. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[5] Roman, M. A., et al. (2018). An End-to-End Framework for Audio-to-Score Music Transcription on Monophonic Excerpts. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[6] Roman, M. A., et al. (2019). A Holistic Approach to Polyphonic Music Transcription with Neural Networks. Proc. of the 20th International Society for Music Information Retrieval Conference (ISMIR).

[7] Smaragdis, P., et al. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.

2026:Audio-to-Score Transcription

2026-06-29T05:22:36Z

Alexandre DHooge: /* MuseSyn Dataset */ explain conversion to kern

= Description =

This page describes the '''MIREX2026: Audio-to-Score Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.

Automatic Music Transcription (AMT) is the task of designing computational algorithms that convert acoustic music signals into a symbolic musical representation [[#bib-benetos-2019|[2]]]. Existing MIREX tasks have evaluated several important AMT components, including melody extraction, drum detection, piano transcription, and multi-instrument transcription. However, these tasks have generally stopped at MIDI or note-event outputs, which are not equivalent to a well-formed musical score.

The Audio-to-Score Transcription (A2S) challenge focuses on this missing step. Participating systems will receive an audio recording of a polyphonic music piece and must output a digital symbolic score that can be read by a musician or notation program. The target output is a kern score encoding standard musical information such as pitches, durations, accidentals, staves, meter, clef, key signature, and bar structure.

The task can be informally expressed as follows:

Prediction('''S''') = argmax P('''S'''|'''X''')

where '''S''' is a symbolic score and '''X''' is the input audio signal.

The challenge will include polyphonic recordings and may include multiple instruments. Pieces may be full recordings or shorter excerpts, such as 3 to 6 bars.

Two tracks will be considered:

* '''Staves-Informed A2S''': staves metadata is provided and may be used to guide voice identification and instrument recognition.
* '''Blind A2S''': only the audio recording is provided.

Participants may enter both tracks. Results will be reported separately.

'''Notice:''' We particularly encourage submissions that move beyond isolated AMT components and produce complete, valid symbolic scores.

= Evaluation =

Submissions will be evaluated with metrics that assess both the textual quality of the produced kern files and the musical transcription quality of the corresponding scores.

=== Score Quality ===

'''Character Error Rate''' (CER): the Levenshtein distance between the submitted and ground-truth kern files at the character level. This metric penalizes character insertions, deletions, and substitutions, and is sensitive to details such as pitch spelling and note durations.

'''Word Error Rate''' (WER): the Levenshtein distance computed on tab- and newline-separated strings in a kern file. WER is reported in recent A2S papers [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. Compared to CER, it evaluates the output closer to the note-token level, checking whether durations and pitches are jointly correct.

'''Line Error Rate''' (LER): the Levenshtein distance computed on complete lines of the output kern files. LER is stricter than CER and WER, and helps verify whether multiple instruments or voices are correctly transcribed and aligned at the beat and sub-beat level.

=== Musical Transcription Quality ===

In addition to CER, WER, and LER, this challenge will use the '''MV2H''' metric introduced in [[#bib-mcleod-2018|[4]]] and used in recent A2S work [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. MV2H is an F-score obtained as the arithmetic mean of five sub-metrics:

* '''Multi-pitch detection''': pitches must be correct and detected at onsets within a 50 ms error threshold.
* '''Voice separation''': notes belonging to the same voice or instrument should be grouped together.
* '''Metrical alignment''': bars, beats, and sub-beats should be correctly identified.
* '''Note value detection''': note durations should be correctly transcribed.
* '''Harmonic analysis''': average of a key detection score and a chord-symbol recall.

The harmonic analysis component will not be used for the initial edition because no chord information is available in the considered datasets.

Public implementations of MV2H are available:

* [https://github.com/apmcleod/MV2H Official MV2H repository]
* [https://github.com/lucasmpaim/pyMV2H Python implementation]

The evaluation pipeline will adapt these implementations to process kern files, building where appropriate on the public code from [[#bib-alfaro-2024|[1]]]:

* [https://github.com/mariaalfaroc/a2s-transformer A2S Transformer repository]

'''Note that WER and MV2H will be retained as the main ranking criteria.''' CER and LER will be reported for more detailed comparisons.

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process the evaluation set.

= Submission Format =

Several submission formats will be accepted to accommodate different system designs.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size
* The number of parameters in the model, if applicable
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time required for the evaluation set

We strongly recommend sharing the submitted algorithms or checkpoints under permissive open licenses, but any licensing (or even not sharing publicly) is accepted as long as it is clearly reported at the time of submission.

=== I / O ===

The submitted algorithm must take as input an audio file, or a folder containing audio files, and write one predicted <code>**kern</code> file for each input audio file to a specified output directory.

==== Input Audio ====

Participating algorithms will receive audio files in the following format:

TBD

For the '''Blind A2S''' track, only the audio recording will be provided.

For the '''Staves-Informed A2S''' track, staves metadata will also be provided. This metadata may include meter, clef, key signature, and other score-structure information needed to guide transcription. The exact metadata packaging will be specified before submissions open.

==== Output File Format ====

Each submitted prediction must be a valid <code>**kern</code> file.

The score must encode, at minimum:

* Staves musical metadata, including meter, clef, and key signature when required
* Pitches, including octaves and accidentals
* Note durations
* Bar lines and metrical structure
* Voice or instrument organization when applicable

For more details, participants may refer to:

* [https://www.humdrum.org/Humdrum/representations/kern.html Humdrum kern representation]

We chose the <code>**kern</code> format for compatibility with existing SOTA work in A2S. Participants may use other formats during training or as intermediate outputs, but the final files should be in <code>**kern</code> format for evaluation.
Please make sure the proposed algorithm include a conversion step if it is required.

=== Code Submissions ===

Participants may provide a Docker container or access to a code repository with clear environment setup instructions and a script that reads the input audio and writes output kern files to the requested destination directory.

The submitted script should be documented in the README and should not require manual intervention during evaluation.

=== Fallback: Pre-computed Kern Submission ===

Participants may alternatively submit the final kern files directly, similarly to the option proposed for MIREX 2024 Polyphonic Transcription.

This fallback is intended to allow participation from systems that cannot be evaluated directly by the Task Captain because of resource, licensing, or infrastructure constraints.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

Participants are free to use the training and validation sets of the datasets listed below. Data augmentation is allowed.

Additional training data may also be used, as long as it does not overlap with the evaluation sets and is clearly documented in the submission.

=== Quartets Dataset ===

The Quartets dataset consists of synthetic audio renderings of Haydn, Mozart, and Beethoven string quartets, together with their full kern scores [[#bib-roman-2019|[6]]].

The scores are taken from the [https://github.com/humdrum-tools/humdrum-data humdrum-data repository]. The kern files were split into 3 to 6 measure excerpts and synthesized into audio from performance MIDI files.

The final dataset contains approximately 20 hours of audio for 38,051 excerpts and 3 composers:

* 18,162 excerpts for Haydn
* 7,435 excerpts for Mozart
* 12,454 excerpts for Beethoven

The train, validation, and test splits used in [[#bib-alfaro-2024|[1]]] are publicly available:

* [https://huggingface.co/datasets/PRAIG/quartets-quartets Quartets dataset on Hugging Face]

=== MuseSyn Dataset ===

The MuseSyn dataset contains 210 piano pieces with scores in MusicXML format and audio synthesized through four different piano models [[#bib-liu-2021|[3]]]. It amounts to almost 10 hours of audio recordings for each piano timbre.

The pieces cover a wide range of key signatures, time signatures, tempos, and polyphony levels. The dataset is available upon request for non-commercial research use:

* [https://zenodo.org/records/4527460 MuseSyn on Zenodo]

Converting the MusicXML files to <code>**kern</code> can be done using [https://extra.humdrum.org/man/xml2hum/ xml2hum]. Contact the task captain if you encounter any issues.

'''Please follow the splits indicated [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata here] and only use the train and validation splits during training.'''

= Evaluation Datasets =

The datasets listed below are reserved for evaluation purposes and '''must not''' be used for training models.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and could lead to data leakage.

=== Quartets Test Set ===

Evaluation will include the test split of the Quartets dataset. Results on this dataset will be reported separately.

=== MuseSyn Test Set ===

Evaluation will include the test split of the MuseSyn dataset after conversion of the reference scores to kern format. Results on this dataset will be reported separately.

We use the test split as [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata defined in the official repository].

=== Undisclosed Evaluation Set ===

An additional custom evaluation set will be used to ensure fairness and out-of-distribution data.

General details on this dataset are:
TBD

= Time and Hardware Limits =

Due to the potentially high number of participants in this and other audio tasks, hard limits on runtime and hardware use will be imposed.

Submissions that require more than 32 GB of VRAM, or more than 24 hours to process the test sets on a single GPU (V100 or similar), cannot be evaluated directly by the Task Captain. Participants in this situation may use the fallback kern submission format described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Alfaro-Contreras, M., et al. (2024). A Transformer Approach for Polyphonic Audio-to-Score Transcription. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2] Benetos, E., et al. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1).

[3] Liu, L., et al. (2021). Joint Multi-Pitch Detection and Score Transcription for Polyphonic Piano Music. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4] Mcleod, A., et al. (2018). Evaluating Automatic Polyphonic Music Transcription. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[5] Roman, M. A., et al. (2018). An End-to-End Framework for Audio-to-Score Music Transcription on Monophonic Excerpts. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[6] Roman, M. A., et al. (2019). A Holistic Approach to Polyphonic Music Transcription with Neural Networks. Proc. of the 20th International Society for Music Information Retrieval Conference (ISMIR).

[7] Smaragdis, P., et al. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.

2026:Audio-to-Score Transcription

2026-06-29T05:13:18Z

Alexandre DHooge: /* MuseSyn Test Set */

= Description =

This page describes the '''MIREX2026: Audio-to-Score Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.

Automatic Music Transcription (AMT) is the task of designing computational algorithms that convert acoustic music signals into a symbolic musical representation [[#bib-benetos-2019|[2]]]. Existing MIREX tasks have evaluated several important AMT components, including melody extraction, drum detection, piano transcription, and multi-instrument transcription. However, these tasks have generally stopped at MIDI or note-event outputs, which are not equivalent to a well-formed musical score.

The Audio-to-Score Transcription (A2S) challenge focuses on this missing step. Participating systems will receive an audio recording of a polyphonic music piece and must output a digital symbolic score that can be read by a musician or notation program. The target output is a kern score encoding standard musical information such as pitches, durations, accidentals, staves, meter, clef, key signature, and bar structure.

The task can be informally expressed as follows:

Prediction('''S''') = argmax P('''S'''|'''X''')

where '''S''' is a symbolic score and '''X''' is the input audio signal.

The challenge will include polyphonic recordings and may include multiple instruments. Pieces may be full recordings or shorter excerpts, such as 3 to 6 bars.

Two tracks will be considered:

* '''Staves-Informed A2S''': staves metadata is provided and may be used to guide voice identification and instrument recognition.
* '''Blind A2S''': only the audio recording is provided.

Participants may enter both tracks. Results will be reported separately.

'''Notice:''' We particularly encourage submissions that move beyond isolated AMT components and produce complete, valid symbolic scores.

= Evaluation =

Submissions will be evaluated with metrics that assess both the textual quality of the produced kern files and the musical transcription quality of the corresponding scores.

=== Score Quality ===

'''Character Error Rate''' (CER): the Levenshtein distance between the submitted and ground-truth kern files at the character level. This metric penalizes character insertions, deletions, and substitutions, and is sensitive to details such as pitch spelling and note durations.

'''Word Error Rate''' (WER): the Levenshtein distance computed on tab- and newline-separated strings in a kern file. WER is reported in recent A2S papers [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. Compared to CER, it evaluates the output closer to the note-token level, checking whether durations and pitches are jointly correct.

'''Line Error Rate''' (LER): the Levenshtein distance computed on complete lines of the output kern files. LER is stricter than CER and WER, and helps verify whether multiple instruments or voices are correctly transcribed and aligned at the beat and sub-beat level.

=== Musical Transcription Quality ===

In addition to CER, WER, and LER, this challenge will use the '''MV2H''' metric introduced in [[#bib-mcleod-2018|[4]]] and used in recent A2S work [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. MV2H is an F-score obtained as the arithmetic mean of five sub-metrics:

* '''Multi-pitch detection''': pitches must be correct and detected at onsets within a 50 ms error threshold.
* '''Voice separation''': notes belonging to the same voice or instrument should be grouped together.
* '''Metrical alignment''': bars, beats, and sub-beats should be correctly identified.
* '''Note value detection''': note durations should be correctly transcribed.
* '''Harmonic analysis''': average of a key detection score and a chord-symbol recall.

The harmonic analysis component will not be used for the initial edition because no chord information is available in the considered datasets.

Public implementations of MV2H are available:

* [https://github.com/apmcleod/MV2H Official MV2H repository]
* [https://github.com/lucasmpaim/pyMV2H Python implementation]

The evaluation pipeline will adapt these implementations to process kern files, building where appropriate on the public code from [[#bib-alfaro-2024|[1]]]:

* [https://github.com/mariaalfaroc/a2s-transformer A2S Transformer repository]

'''Note that WER and MV2H will be retained as the main ranking criteria.''' CER and LER will be reported for more detailed comparisons.

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process the evaluation set.

= Submission Format =

Several submission formats will be accepted to accommodate different system designs.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size
* The number of parameters in the model, if applicable
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time required for the evaluation set

We strongly recommend sharing the submitted algorithms or checkpoints under permissive open licenses, but any licensing (or even not sharing publicly) is accepted as long as it is clearly reported at the time of submission.

=== I / O ===

The submitted algorithm must take as input an audio file, or a folder containing audio files, and write one predicted <code>**kern</code> file for each input audio file to a specified output directory.

==== Input Audio ====

Participating algorithms will receive audio files in the following format:

TBD

For the '''Blind A2S''' track, only the audio recording will be provided.

For the '''Staves-Informed A2S''' track, staves metadata will also be provided. This metadata may include meter, clef, key signature, and other score-structure information needed to guide transcription. The exact metadata packaging will be specified before submissions open.

==== Output File Format ====

Each submitted prediction must be a valid <code>**kern</code> file.

The score must encode, at minimum:

* Staves musical metadata, including meter, clef, and key signature when required
* Pitches, including octaves and accidentals
* Note durations
* Bar lines and metrical structure
* Voice or instrument organization when applicable

For more details, participants may refer to:

* [https://www.humdrum.org/Humdrum/representations/kern.html Humdrum kern representation]

We chose the <code>**kern</code> format for compatibility with existing SOTA work in A2S. Participants may use other formats during training or as intermediate outputs, but the final files should be in <code>**kern</code> format for evaluation.
Please make sure the proposed algorithm include a conversion step if it is required.

=== Code Submissions ===

Participants may provide a Docker container or access to a code repository with clear environment setup instructions and a script that reads the input audio and writes output kern files to the requested destination directory.

The submitted script should be documented in the README and should not require manual intervention during evaluation.

=== Fallback: Pre-computed Kern Submission ===

Participants may alternatively submit the final kern files directly, similarly to the option proposed for MIREX 2024 Polyphonic Transcription.

This fallback is intended to allow participation from systems that cannot be evaluated directly by the Task Captain because of resource, licensing, or infrastructure constraints.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

Participants are free to use the training and validation sets of the datasets listed below. Data augmentation is allowed.

Additional training data may also be used, as long as it does not overlap with the evaluation sets and is clearly documented in the submission.

=== Quartets Dataset ===

The Quartets dataset consists of synthetic audio renderings of Haydn, Mozart, and Beethoven string quartets, together with their full kern scores [[#bib-roman-2019|[6]]].

The scores are taken from the [https://github.com/humdrum-tools/humdrum-data humdrum-data repository]. The kern files were split into 3 to 6 measure excerpts and synthesized into audio from performance MIDI files.

The final dataset contains approximately 20 hours of audio for 38,051 excerpts and 3 composers:

* 18,162 excerpts for Haydn
* 7,435 excerpts for Mozart
* 12,454 excerpts for Beethoven

The train, validation, and test splits used in [[#bib-alfaro-2024|[1]]] are publicly available:

* [https://huggingface.co/datasets/PRAIG/quartets-quartets Quartets dataset on Hugging Face]

=== MuseSyn Dataset ===

The MuseSyn dataset contains 210 piano pieces with scores in MusicXML format and audio synthesized through four different piano models [[#bib-liu-2021|[3]]]. It amounts to almost 10 hours of audio recordings for each piano timbre.

The pieces cover a wide range of key signatures, time signatures, tempos, and polyphony levels. The dataset is available upon request for non-commercial research use:

* [https://zenodo.org/records/4527460 MuseSyn on Zenodo]

For this challenge, the MusicXML files will be converted to kern format, with manual verification where needed, to unify the evaluation pipeline.

'''Please follow the splits indicated [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata here] and only use the train and validation splits during training.'''

= Evaluation Datasets =

The datasets listed below are reserved for evaluation purposes and '''must not''' be used for training models.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and could lead to data leakage.

=== Quartets Test Set ===

Evaluation will include the test split of the Quartets dataset. Results on this dataset will be reported separately.

=== MuseSyn Test Set ===

Evaluation will include the test split of the MuseSyn dataset after conversion of the reference scores to kern format. Results on this dataset will be reported separately.

We use the test split as [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata defined in the official repository].

=== Undisclosed Evaluation Set ===

An additional custom evaluation set will be used to ensure fairness and out-of-distribution data.

General details on this dataset are:
TBD

= Time and Hardware Limits =

Due to the potentially high number of participants in this and other audio tasks, hard limits on runtime and hardware use will be imposed.

Submissions that require more than 32 GB of VRAM, or more than 24 hours to process the test sets on a single GPU (V100 or similar), cannot be evaluated directly by the Task Captain. Participants in this situation may use the fallback kern submission format described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Alfaro-Contreras, M., et al. (2024). A Transformer Approach for Polyphonic Audio-to-Score Transcription. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2] Benetos, E., et al. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1).

[3] Liu, L., et al. (2021). Joint Multi-Pitch Detection and Score Transcription for Polyphonic Piano Music. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4] Mcleod, A., et al. (2018). Evaluating Automatic Polyphonic Music Transcription. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[5] Roman, M. A., et al. (2018). An End-to-End Framework for Audio-to-Score Music Transcription on Monophonic Excerpts. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[6] Roman, M. A., et al. (2019). A Holistic Approach to Polyphonic Music Transcription with Neural Networks. Proc. of the 20th International Society for Music Information Retrieval Conference (ISMIR).

[7] Smaragdis, P., et al. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.

2026:Audio-to-Score Transcription

2026-06-29T05:12:16Z

Alexandre DHooge: /* MuseSyn Dataset */ add splits details

= Description =

This page describes the '''MIREX2026: Audio-to-Score Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.

Automatic Music Transcription (AMT) is the task of designing computational algorithms that convert acoustic music signals into a symbolic musical representation [[#bib-benetos-2019|[2]]]. Existing MIREX tasks have evaluated several important AMT components, including melody extraction, drum detection, piano transcription, and multi-instrument transcription. However, these tasks have generally stopped at MIDI or note-event outputs, which are not equivalent to a well-formed musical score.

The Audio-to-Score Transcription (A2S) challenge focuses on this missing step. Participating systems will receive an audio recording of a polyphonic music piece and must output a digital symbolic score that can be read by a musician or notation program. The target output is a kern score encoding standard musical information such as pitches, durations, accidentals, staves, meter, clef, key signature, and bar structure.

The task can be informally expressed as follows:

Prediction('''S''') = argmax P('''S'''|'''X''')

where '''S''' is a symbolic score and '''X''' is the input audio signal.

The challenge will include polyphonic recordings and may include multiple instruments. Pieces may be full recordings or shorter excerpts, such as 3 to 6 bars.

Two tracks will be considered:

* '''Staves-Informed A2S''': staves metadata is provided and may be used to guide voice identification and instrument recognition.
* '''Blind A2S''': only the audio recording is provided.

Participants may enter both tracks. Results will be reported separately.

'''Notice:''' We particularly encourage submissions that move beyond isolated AMT components and produce complete, valid symbolic scores.

= Evaluation =

Submissions will be evaluated with metrics that assess both the textual quality of the produced kern files and the musical transcription quality of the corresponding scores.

=== Score Quality ===

'''Character Error Rate''' (CER): the Levenshtein distance between the submitted and ground-truth kern files at the character level. This metric penalizes character insertions, deletions, and substitutions, and is sensitive to details such as pitch spelling and note durations.

'''Word Error Rate''' (WER): the Levenshtein distance computed on tab- and newline-separated strings in a kern file. WER is reported in recent A2S papers [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. Compared to CER, it evaluates the output closer to the note-token level, checking whether durations and pitches are jointly correct.

'''Line Error Rate''' (LER): the Levenshtein distance computed on complete lines of the output kern files. LER is stricter than CER and WER, and helps verify whether multiple instruments or voices are correctly transcribed and aligned at the beat and sub-beat level.

=== Musical Transcription Quality ===

In addition to CER, WER, and LER, this challenge will use the '''MV2H''' metric introduced in [[#bib-mcleod-2018|[4]]] and used in recent A2S work [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. MV2H is an F-score obtained as the arithmetic mean of five sub-metrics:

* '''Multi-pitch detection''': pitches must be correct and detected at onsets within a 50 ms error threshold.
* '''Voice separation''': notes belonging to the same voice or instrument should be grouped together.
* '''Metrical alignment''': bars, beats, and sub-beats should be correctly identified.
* '''Note value detection''': note durations should be correctly transcribed.
* '''Harmonic analysis''': average of a key detection score and a chord-symbol recall.

The harmonic analysis component will not be used for the initial edition because no chord information is available in the considered datasets.

Public implementations of MV2H are available:

* [https://github.com/apmcleod/MV2H Official MV2H repository]
* [https://github.com/lucasmpaim/pyMV2H Python implementation]

The evaluation pipeline will adapt these implementations to process kern files, building where appropriate on the public code from [[#bib-alfaro-2024|[1]]]:

* [https://github.com/mariaalfaroc/a2s-transformer A2S Transformer repository]

'''Note that WER and MV2H will be retained as the main ranking criteria.''' CER and LER will be reported for more detailed comparisons.

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process the evaluation set.

= Submission Format =

Several submission formats will be accepted to accommodate different system designs.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size
* The number of parameters in the model, if applicable
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time required for the evaluation set

We strongly recommend sharing the submitted algorithms or checkpoints under permissive open licenses, but any licensing (or even not sharing publicly) is accepted as long as it is clearly reported at the time of submission.

=== I / O ===

The submitted algorithm must take as input an audio file, or a folder containing audio files, and write one predicted <code>**kern</code> file for each input audio file to a specified output directory.

==== Input Audio ====

Participating algorithms will receive audio files in the following format:

TBD

For the '''Blind A2S''' track, only the audio recording will be provided.

For the '''Staves-Informed A2S''' track, staves metadata will also be provided. This metadata may include meter, clef, key signature, and other score-structure information needed to guide transcription. The exact metadata packaging will be specified before submissions open.

==== Output File Format ====

Each submitted prediction must be a valid <code>**kern</code> file.

The score must encode, at minimum:

* Staves musical metadata, including meter, clef, and key signature when required
* Pitches, including octaves and accidentals
* Note durations
* Bar lines and metrical structure
* Voice or instrument organization when applicable

For more details, participants may refer to:

* [https://www.humdrum.org/Humdrum/representations/kern.html Humdrum kern representation]

We chose the <code>**kern</code> format for compatibility with existing SOTA work in A2S. Participants may use other formats during training or as intermediate outputs, but the final files should be in <code>**kern</code> format for evaluation.
Please make sure the proposed algorithm include a conversion step if it is required.

=== Code Submissions ===

Participants may provide a Docker container or access to a code repository with clear environment setup instructions and a script that reads the input audio and writes output kern files to the requested destination directory.

The submitted script should be documented in the README and should not require manual intervention during evaluation.

=== Fallback: Pre-computed Kern Submission ===

Participants may alternatively submit the final kern files directly, similarly to the option proposed for MIREX 2024 Polyphonic Transcription.

This fallback is intended to allow participation from systems that cannot be evaluated directly by the Task Captain because of resource, licensing, or infrastructure constraints.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

Participants are free to use the training and validation sets of the datasets listed below. Data augmentation is allowed.

Additional training data may also be used, as long as it does not overlap with the evaluation sets and is clearly documented in the submission.

=== Quartets Dataset ===

The Quartets dataset consists of synthetic audio renderings of Haydn, Mozart, and Beethoven string quartets, together with their full kern scores [[#bib-roman-2019|[6]]].

The scores are taken from the [https://github.com/humdrum-tools/humdrum-data humdrum-data repository]. The kern files were split into 3 to 6 measure excerpts and synthesized into audio from performance MIDI files.

The final dataset contains approximately 20 hours of audio for 38,051 excerpts and 3 composers:

* 18,162 excerpts for Haydn
* 7,435 excerpts for Mozart
* 12,454 excerpts for Beethoven

The train, validation, and test splits used in [[#bib-alfaro-2024|[1]]] are publicly available:

* [https://huggingface.co/datasets/PRAIG/quartets-quartets Quartets dataset on Hugging Face]

=== MuseSyn Dataset ===

The MuseSyn dataset contains 210 piano pieces with scores in MusicXML format and audio synthesized through four different piano models [[#bib-liu-2021|[3]]]. It amounts to almost 10 hours of audio recordings for each piano timbre.

The pieces cover a wide range of key signatures, time signatures, tempos, and polyphony levels. The dataset is available upon request for non-commercial research use:

* [https://zenodo.org/records/4527460 MuseSyn on Zenodo]

For this challenge, the MusicXML files will be converted to kern format, with manual verification where needed, to unify the evaluation pipeline.

'''Please follow the splits indicated [https://github.com/cheriell/ICASSP2021-A2S/tree/main/metadata here] and only use the train and validation splits during training.'''

= Evaluation Datasets =

The datasets listed below are reserved for evaluation purposes and '''must not''' be used for training models.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and could lead to data leakage.

=== Quartets Test Set ===

Evaluation will include the test split of the Quartets dataset. Results on this dataset will be reported separately.

=== MuseSyn Test Set ===

Evaluation will include the test split of the MuseSyn dataset after conversion of the reference scores to kern format. Results on this dataset will be reported separately.

=== Undisclosed Evaluation Set ===

An additional custom evaluation set will be used to ensure fairness and out-of-distribution data.

General details on this dataset are:
TBD

= Time and Hardware Limits =

Due to the potentially high number of participants in this and other audio tasks, hard limits on runtime and hardware use will be imposed.

Submissions that require more than 32 GB of VRAM, or more than 24 hours to process the test sets on a single GPU (V100 or similar), cannot be evaluated directly by the Task Captain. Participants in this situation may use the fallback kern submission format described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Alfaro-Contreras, M., et al. (2024). A Transformer Approach for Polyphonic Audio-to-Score Transcription. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2] Benetos, E., et al. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1).

[3] Liu, L., et al. (2021). Joint Multi-Pitch Detection and Score Transcription for Polyphonic Piano Music. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4] Mcleod, A., et al. (2018). Evaluating Automatic Polyphonic Music Transcription. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[5] Roman, M. A., et al. (2018). An End-to-End Framework for Audio-to-Score Music Transcription on Monophonic Excerpts. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[6] Roman, M. A., et al. (2019). A Holistic Approach to Polyphonic Music Transcription with Neural Networks. Proc. of the 20th International Society for Music Information Retrieval Conference (ISMIR).

[7] Smaragdis, P., et al. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.

2026:Lyrics Transcription

2026-06-29T03:34:38Z

Alexandre DHooge: add real example

= Description =

This page describes the '''MIREX2026: Automatic Lyrics Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.
''It is strongly based upon the page of the 2025 ALT challenge.''

The task of Lyrics Transcription aims to identify the words from sung utterances, in the same way as in automatic speech recognition. This can be mathematically expressed as follows:

Prediction('''w''') = argmax P('''w'''|'''X''')

where '''w''' and '''X''' are the word and acoustic features respectively.

Ideally, the lyrics transcriber should return meaningful word sequences:

Prediction('''w''') = [ <w_1>, <w_2>, ..., <w_N> ]

Note that for this year's edition, the input will always be a polyphonic mix (singing voice + musical accompaniment). The submitted algorithms can include a source-separation step if needed.

'''Notice:''' We particularly encourage participations that build on the latest ALT approaches such as using ''pretrained audio foundation models'' or ''LLMs''.

== Real Example ==

Given an audio file such as [https://datasets-server.huggingface.co/assets/jamendolyrics/jam-alt/--/28302224954ef050fe752d1628dd9bac4fc8c02b/--/all/test/0/audio/audio.mp3?Expires=1782704825&Signature=HYbFQpoM1OFg9UfbSCHCepQwGBgEfWMhqnX7AV2Wc9KUK8RkeZtcNyHvqMBRX-OzHMDNYF71nA3aoQYJYXXipMc6XDt1mkoM7YwQr~KgY8T4FadL59wEQgNiVIRMbGvAVzmtdJKWxk9kp1uaux2v0zL2KHTWozKXpjsZbU3FazZBmL7l4Qc1ZjiIoWzVyaE2lkhDsHrS8ssLGAEscVaaqZ6jCSlYMRViez2MEUZ8HIrVdawa0rRLcVzfp1hIobFKCCj2jQ~1~d9KEYLvntwz1pqlR5J3QCG0eGf1bReEnxIzTZE7aG4IbfMU8awCDCi3E2fBKUezmB7xPNhH2t9QFg__&Key-Pair-Id=K204OQ5RWQVDLD this one], which comes from the Jam-ALT dataset, the expected lyrics are as follows:

<div class="toccolours mw-collapsible mw-collapsed" style="width:400px; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">HILA - Give me the same - Lyrics</div>
<div class="mw-collapsible-content">
Lay awake at night 
Wondering how could I 
Let it get this way 
Through all the pain 
I took all the blame 
While you cursed my last name 
I thought we could get better 
Stayed committed like a soldier 
Believed we're okay 
Now I know that you don't care 
At all 
 
Time wasted 
Gave all of my soul and my heart to someone that would never 
(Give me the same) 
I was a fool to believe that you would finally be the one to 
(Give me the same) 
Heartbroken now 
How you let me down 
But somehow, deep down, I still do 
Love you 
 
Now you're out all night 
Telling your lies 
About the way I hurt you 
They couldn't see the way 
You made me feel ashamed 
For choosing to love you 
I began to see clearer 
That you were just a liar 
Stuck in your ways 
So now I choose to walk away 
Forever 
 
Time wasted 
Gave all of my soul and my heart to someone that would never 
(Give me the same) 
I was a fool to believe that you would finally be the one to 
Give me the same 
Heartbroken now 
How you let me down 
But somehow, deep down, I still do 
Love you 
 
How could you just throw it all away, babe 
(How could you throw it all away) 
I stayed with you through all the lies and pain, oh 
(Through all the lies and pain, oh) 
Tears roll down my eyes (down my eyes) 
As I now realize 
Your love was never truly mine 
At all 
 
Time wasted 
Gave all of my soul and my heart to someone that would never 
(Give me the same) 
I was a fool to believe that you would finally be the one to 
(Give me the same) 
Heartbroken now (heartbroken now) 
How you let me down (oh) 
But somehow, deep down, I still do 
Love you 
</div></div>

= Evaluation =

This year's edition will include the following metrics for evaluation:

'''Word Error Rate''' (WER): the standard metric used in Automatic Speech Recognition.

WER = (S + I + D) / (C + S + D)

where;
C : correctly predicted words
S : substitution errors
I : insertion errors
D : deletion errors

'''Character Error Rate''' (CER): the above computation can also be done on the character level. This metric penalises the partially correctly predicted / incorrectly spelled words less than WER.

'''Case-Sensitive WER (WER')''': defined in [[#bib-cifka-2024|[1]]], this metric is computed as follows:

<math>
\mathrm{WER'} = \mathrm{WER} + \frac{E_\text{case}}{N}
</math>

where <math>N</math> is the total number of words in a ground-truth lyrics file, and <math>E_\text{case}</math> the number of "casing errors" i.e. words that differ in a ''case-sensitive'' setting like "city" and "City".

'''Punctuation, Parentheses, Line and Section Breaks''': also defined in [[#bib-cifka-2024|[1]]], we include a set of metrics to measure how well the proposed algorithms predict several formatting tokens. The tokens are classified into one of 5 types <math>T\in {W,P,B,L,S}</math> for Word (W), Punctuation (P), Parentheses (B), Line Breaks (L), and Section Breaks (S). Then, for each token type (except Word), we compute Precision, Recall, and F1-Score:

<math>
P_T = \frac{C_T}{C_T + S_T + I_T}
</math>

<math>
R_T = \frac{C_T}{C_T + S_T + D_T}
</math>

<math>
F_T = \frac{2}{P_T^{-1} + R_T^{-1}}
</math>

All these metrics will be based on the [https://github.com/audioshake/alt-eval public code implementation] from [[#bib-cifka-2024|[1]]], simply extended to include CER.

'''Note that WER and CER will be retained as the main ranking criteria, other metrics will be reported for more detailed comparisons.'''

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process a sample.

= Submission Format =

Several submission formats will be accepted to accommodate the different approaches chosen by the participants.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size (hours)
* The number of parameters in the model (if applicable)
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time per hour of audio

=== I / O ===

The submitted algorithm must take as input argument a path to a folder with .wav files and output predicted lyrics to a destination folder, each result file named as the corresponding input, with the extension changed.

==== Input Audio ====

Participating algorithms will have to receive the following audio input format:

* Musical tracks with accompaniment (polyphonic, not vocals only)
* 16-bit PCM <code>wav</code> files
* 44.1 kHz sampling rate
* stereo (2 channels)

Be aware that the wav files are obtained through conversion from compressed formats (mp3 or AAC) for the test sets we considered, so the quality is not exactly that of a perfect 44.1 kHz wav file.

This format is chosen as a standard input. If your pipeline expect different characteristics, the conversion process should be part of the submitted algorithm.

==== Output File Format ====

A text file (per song) containing the lyrics, organized by lines and paragraphs.

<line1: word1> <line1: word_2> ... <line1: word_N>\n
<line2: word1> <line2: word_2> ... <line2: word_N>\n
\n
<line3: word1> <line3: word_2> ... <line3: word_N>\n
<line4: word1> <line4: word_2> ... <line4: word_N>\n

Words will be identified by splitting the text file at spaces, tabs, and newline characters.
The submission can also simply generate a file of words separated by spaces, but it will then perform (very) poorly on the new metrics taken from [[#bib-cifka-2024|[1]]].

=== API-based submissions ===

For systems that rely on commercial LLM APIs (e.g. systems similar to [[#bib-lyricwhiz|[2]]]), participants must specify the exact model version and, where possible, provide a fallback local model to allow offline evaluation.
The algorithm should either already contain an API key allowing to use the model (we recommend setting an access token with a clear duration or computational limit), or use the fallback option described hereafter.

=== Fallback: Pre-computed lyrics submission ===

In situations where the inference cannot be run directly by the Task Captain, participants may provide the lyrics files produced by their algorithms.
This should only be considered a fallback option in case the algorithm relies on proprietary or commercial resources that cannot be made accessible to the Task Captains.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

In this challenge, the participants are encouraged but '''not obliged''' to use the open source dataset DALI described below.

The DAMP dataset is unfortunately [https://ccrma.stanford.edu/damp/ no longer available].

Participants are free to use other datasets for training, as long as they are described in the submission and don't overlap with the evaluation sets.

=== DALI Dataset ===

DALI (a large '''D'''ataset of synchronised '''A'''udio, '''L'''yr'''I'''cs and notes) [[#bib-syed-2025|[3]]] is the benchmark dataset for building an acoustic model on polyphonic recordings [[#bib-gupta-2020|[4]]], [[#bib-basak-2021|[5]]], [[#bib-demirel-2021|[6]]] and it contains over 7000 songs with semi-automatically aligned lyrics annotations.

The songs are commercial recordings in full-duration, whereas the lyrics are described according to different levels of granularity including words and notes (and syllables underlying a given note).

For each song DALI provides a link to a matched youtube video for the audio retrieval.

* For more details, see its full description [https://github.com/gabolsgabs/DALI here]. Paper [https://arxiv.org/pdf/1906.10606.pdf here].

= Evaluation Datasets =

The datasets listed below are reserved exclusively for evaluation purposes and '''must not''' be used for training models under any circumstances.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and lead to data leakage of some sort.

=== Jam-ALT dataset ===

The Jam-ALT dataset is built upon the Jamendo dataset [[#bib-stoller-2019|[7]]] and contains 79 songs in 4 languages: English, French, German, and Spanish. All tracks include instrumental accompaniment.
The lyrics were cleaned and their format harmonized in [[#bib-cifka-2024|[1]]].

If you’re working with English-only models, only the 20 English songs will be used for evaluation.

The <code>mp3</code> tracks from the original dataset were converted to 16-bit PCM <code>wav</code> at 44.1 kHz sample rate for this MIREX challenge.

=== MUSDB-ALT ===

Introduced in [[#bib-syed-2025|[3]]], this dataset contains 39 English songs and will be used as an additional evaluation set.

We use the dataset in its [https://sigsep.github.io/datasets/musdb.html#musdb18-compressed-stems compressed stems] version, which means that the spectral energy is null above 16kHz.
We extracted the full mixture from the Native Instruments stems files and exported them as 16-bit PCM <code>wav</code> files with 44.1 kHz sample rate.

=== Undisclosed Evaluation Set ===

In addition to Jam-ALT and MUSDB-ALT, the submissions will be evaluated on an in-house dataset of 10 English songs from the Western Popular Music repertoire, manually assembled and verified by the Task Captain.

= Time and hardware limits =
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.
A hard limit of 24 hours will be imposed on analysis times, using a single GPU with at most 32GB of VRAM (V100 or similar). Submissions that require more resources can be submitted using the fallback method described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Cífka, O., et al. (2024). Lyrics Transcription for Humans: A Readability-Aware Benchmark. Proc. of the 25th ISMIR Conf.

[2] Zhuo, L., et al. (2023). LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT. Proc. of the 24th ISMIR Conf.

[3] Syed, J., et al. (2025). Exploiting Music Source Separation for Automatic Lyrics Transcription with Whisper. ICMEW.

[4] Gupta, C., Yılmaz, E., & Li, H. (2020). Automatic lyrics alignment and transcription in polyphonic music: Does background music help? In ICASSP 2020, 496-500. IEEE.

[5] Basak, S., Agarwal, S., Ganapathy, S., & Takahashi, N. (2021, June). End-to-End Lyrics Recognition with Voice to Singing Style Transfer. In ICASSP 2021, 266-270. IEEE.

[6] Demirel, E., Ahlbäck, S., & Dixon, S. (2021). MSTRE-Net: Multistreaming Acoustic Modeling for Automatic Lyrics Transcription. Proc. ISMIR 2021.

[7] Stoller, D., Durand, S., & Ewert, S. (2019). End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-Character Recognition Model. In ICASSP 2019, IEEE.

2026:Lyrics Transcription

2026-06-29T03:06:09Z

Alexandre DHooge: fix list

= Description =

This page describes the '''MIREX2026: Automatic Lyrics Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.
''It is strongly based upon the page of the 2025 ALT challenge.''

The task of Lyrics Transcription aims to identify the words from sung utterances, in the same way as in automatic speech recognition. This can be mathematically expressed as follows:

Prediction('''w''') = argmax P('''w'''|'''X''')

where '''w''' and '''X''' are the word and acoustic features respectively.

Ideally, the lyrics transcriber should return meaningful word sequences:

Prediction('''w''') = [ <w_1>, <w_2>, ..., <w_N> ]

Note that for this year's edition, the input will always be a polyphonic mix (singing voice + musical accompaniment). The submitted algorithms can include a source-separation step if needed.

'''Notice:''' We particularly encourage participations that build on the latest ALT approaches such as using ''pretrained audio foundation models'' or ''LLMs''.

= Evaluation =

This year's edition will include the following metrics for evaluation:

'''Word Error Rate''' (WER): the standard metric used in Automatic Speech Recognition.

WER = (S + I + D) / (C + S + D)

where;
C : correctly predicted words
S : substitution errors
I : insertion errors
D : deletion errors

'''Character Error Rate''' (CER): the above computation can also be done on the character level. This metric penalises the partially correctly predicted / incorrectly spelled words less than WER.

'''Case-Sensitive WER (WER')''': defined in [[#bib-cifka-2024|[1]]], this metric is computed as follows:

<math>
\mathrm{WER'} = \mathrm{WER} + \frac{E_\text{case}}{N}
</math>

where <math>N</math> is the total number of words in a ground-truth lyrics file, and <math>E_\text{case}</math> the number of "casing errors" i.e. words that differ in a ''case-sensitive'' setting like "city" and "City".

'''Punctuation, Parentheses, Line and Section Breaks''': also defined in [[#bib-cifka-2024|[1]]], we include a set of metrics to measure how well the proposed algorithms predict several formatting tokens. The tokens are classified into one of 5 types <math>T\in {W,P,B,L,S}</math> for Word (W), Punctuation (P), Parentheses (B), Line Breaks (L), and Section Breaks (S). Then, for each token type (except Word), we compute Precision, Recall, and F1-Score:

<math>
P_T = \frac{C_T}{C_T + S_T + I_T}
</math>

<math>
R_T = \frac{C_T}{C_T + S_T + D_T}
</math>

<math>
F_T = \frac{2}{P_T^{-1} + R_T^{-1}}
</math>

All these metrics will be based on the [https://github.com/audioshake/alt-eval public code implementation] from [[#bib-cifka-2024|[1]]], simply extended to include CER.

'''Note that WER and CER will be retained as the main ranking criteria, other metrics will be reported for more detailed comparisons.'''

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process a sample.

= Submission Format =

Several submission formats will be accepted to accommodate the different approaches chosen by the participants.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size (hours)
* The number of parameters in the model (if applicable)
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time per hour of audio

=== I / O ===

The submitted algorithm must take as input argument a path to a folder with .wav files and output predicted lyrics to a destination folder, each result file named as the corresponding input, with the extension changed.

==== Input Audio ====

Participating algorithms will have to receive the following audio input format:

* Musical tracks with accompaniment (polyphonic, not vocals only)
* 16-bit PCM <code>wav</code> files
* 44.1 kHz sampling rate
* stereo (2 channels)

Be aware that the wav files are obtained through conversion from compressed formats (mp3 or AAC) for the test sets we considered, so the quality is not exactly that of a perfect 44.1 kHz wav file.

This format is chosen as a standard input. If your pipeline expect different characteristics, the conversion process should be part of the submitted algorithm.

==== Output File Format ====

A text file (per song) containing the lyrics, organized by lines and paragraphs.

<line1: word1> <line1: word_2> ... <line1: word_N>\n
<line2: word1> <line2: word_2> ... <line2: word_N>\n
\n
<line3: word1> <line3: word_2> ... <line3: word_N>\n
<line4: word1> <line4: word_2> ... <line4: word_N>\n

Words will be identified by splitting the text file at spaces, tabs, and newline characters.
The submission can also simply generate a file of words separated by spaces, but it will then perform (very) poorly on the new metrics taken from [[#bib-cifka-2024|[1]]].

=== API-based submissions ===

For systems that rely on commercial LLM APIs (e.g. systems similar to [[#bib-lyricwhiz|[2]]]), participants must specify the exact model version and, where possible, provide a fallback local model to allow offline evaluation.
The algorithm should either already contain an API key allowing to use the model (we recommend setting an access token with a clear duration or computational limit), or use the fallback option described hereafter.

=== Fallback: Pre-computed lyrics submission ===

In situations where the inference cannot be run directly by the Task Captain, participants may provide the lyrics files produced by their algorithms.
This should only be considered a fallback option in case the algorithm relies on proprietary or commercial resources that cannot be made accessible to the Task Captains.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

In this challenge, the participants are encouraged but '''not obliged''' to use the open source dataset DALI described below.

The DAMP dataset is unfortunately [https://ccrma.stanford.edu/damp/ no longer available].

Participants are free to use other datasets for training, as long as they are described in the submission and don't overlap with the evaluation sets.

=== DALI Dataset ===

DALI (a large '''D'''ataset of synchronised '''A'''udio, '''L'''yr'''I'''cs and notes) [[#bib-syed-2025|[3]]] is the benchmark dataset for building an acoustic model on polyphonic recordings [[#bib-gupta-2020|[4]]], [[#bib-basak-2021|[5]]], [[#bib-demirel-2021|[6]]] and it contains over 7000 songs with semi-automatically aligned lyrics annotations.

The songs are commercial recordings in full-duration, whereas the lyrics are described according to different levels of granularity including words and notes (and syllables underlying a given note).

For each song DALI provides a link to a matched youtube video for the audio retrieval.

* For more details, see its full description [https://github.com/gabolsgabs/DALI here]. Paper [https://arxiv.org/pdf/1906.10606.pdf here].

= Evaluation Datasets =

The datasets listed below are reserved exclusively for evaluation purposes and '''must not''' be used for training models under any circumstances.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and lead to data leakage of some sort.

=== Jam-ALT dataset ===

The Jam-ALT dataset is built upon the Jamendo dataset [[#bib-stoller-2019|[7]]] and contains 79 songs in 4 languages: English, French, German, and Spanish. All tracks include instrumental accompaniment.
The lyrics were cleaned and their format harmonized in [[#bib-cifka-2024|[1]]].

If you’re working with English-only models, only the 20 English songs will be used for evaluation.

The <code>mp3</code> tracks from the original dataset were converted to 16-bit PCM <code>wav</code> at 44.1 kHz sample rate for this MIREX challenge.

=== MUSDB-ALT ===

Introduced in [[#bib-syed-2025|[3]]], this dataset contains 39 English songs and will be used as an additional evaluation set.

We use the dataset in its [https://sigsep.github.io/datasets/musdb.html#musdb18-compressed-stems compressed stems] version, which means that the spectral energy is null above 16kHz.
We extracted the full mixture from the Native Instruments stems files and exported them as 16-bit PCM <code>wav</code> files with 44.1 kHz sample rate.

=== Undisclosed Evaluation Set ===

In addition to Jam-ALT and MUSDB-ALT, the submissions will be evaluated on an in-house dataset of 10 English songs from the Western Popular Music repertoire, manually assembled and verified by the Task Captain.

= Time and hardware limits =
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.
A hard limit of 24 hours will be imposed on analysis times, using a single GPU with at most 32GB of VRAM (V100 or similar). Submissions that require more resources can be submitted using the fallback method described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Cífka, O., et al. (2024). Lyrics Transcription for Humans: A Readability-Aware Benchmark. Proc. of the 25th ISMIR Conf.

[2] Zhuo, L., et al. (2023). LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT. Proc. of the 24th ISMIR Conf.

[3] Syed, J., et al. (2025). Exploiting Music Source Separation for Automatic Lyrics Transcription with Whisper. ICMEW.

[4] Gupta, C., Yılmaz, E., & Li, H. (2020). Automatic lyrics alignment and transcription in polyphonic music: Does background music help? In ICASSP 2020, 496-500. IEEE.

[5] Basak, S., Agarwal, S., Ganapathy, S., & Takahashi, N. (2021, June). End-to-End Lyrics Recognition with Voice to Singing Style Transfer. In ICASSP 2021, 266-270. IEEE.

[6] Demirel, E., Ahlbäck, S., & Dixon, S. (2021). MSTRE-Net: Multistreaming Acoustic Modeling for Automatic Lyrics Transcription. Proc. ISMIR 2021.

[7] Stoller, D., Durand, S., & Ewert, S. (2019). End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-Character Recognition Model. In ICASSP 2019, IEEE.

2026:Lyrics Transcription

2026-06-29T03:05:21Z

Alexandre DHooge: add audio format details

= Description =

This page describes the '''MIREX2026: Automatic Lyrics Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.
''It is strongly based upon the page of the 2025 ALT challenge.''

The task of Lyrics Transcription aims to identify the words from sung utterances, in the same way as in automatic speech recognition. This can be mathematically expressed as follows:

Prediction('''w''') = argmax P('''w'''|'''X''')

where '''w''' and '''X''' are the word and acoustic features respectively.

Ideally, the lyrics transcriber should return meaningful word sequences:

Prediction('''w''') = [ <w_1>, <w_2>, ..., <w_N> ]

Note that for this year's edition, the input will always be a polyphonic mix (singing voice + musical accompaniment). The submitted algorithms can include a source-separation step if needed.

'''Notice:''' We particularly encourage participations that build on the latest ALT approaches such as using ''pretrained audio foundation models'' or ''LLMs''.

= Evaluation =

This year's edition will include the following metrics for evaluation:

'''Word Error Rate''' (WER): the standard metric used in Automatic Speech Recognition.

WER = (S + I + D) / (C + S + D)

where;
C : correctly predicted words
S : substitution errors
I : insertion errors
D : deletion errors

'''Character Error Rate''' (CER): the above computation can also be done on the character level. This metric penalises the partially correctly predicted / incorrectly spelled words less than WER.

'''Case-Sensitive WER (WER')''': defined in [[#bib-cifka-2024|[1]]], this metric is computed as follows:

<math>
\mathrm{WER'} = \mathrm{WER} + \frac{E_\text{case}}{N}
</math>

where <math>N</math> is the total number of words in a ground-truth lyrics file, and <math>E_\text{case}</math> the number of "casing errors" i.e. words that differ in a ''case-sensitive'' setting like "city" and "City".

'''Punctuation, Parentheses, Line and Section Breaks''': also defined in [[#bib-cifka-2024|[1]]], we include a set of metrics to measure how well the proposed algorithms predict several formatting tokens. The tokens are classified into one of 5 types <math>T\in {W,P,B,L,S}</math> for Word (W), Punctuation (P), Parentheses (B), Line Breaks (L), and Section Breaks (S). Then, for each token type (except Word), we compute Precision, Recall, and F1-Score:

<math>
P_T = \frac{C_T}{C_T + S_T + I_T}
</math>

<math>
R_T = \frac{C_T}{C_T + S_T + D_T}
</math>

<math>
F_T = \frac{2}{P_T^{-1} + R_T^{-1}}
</math>

All these metrics will be based on the [https://github.com/audioshake/alt-eval public code implementation] from [[#bib-cifka-2024|[1]]], simply extended to include CER.

'''Note that WER and CER will be retained as the main ranking criteria, other metrics will be reported for more detailed comparisons.'''

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process a sample.

= Submission Format =

Several submission formats will be accepted to accommodate the different approaches chosen by the participants.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size (hours)
* The number of parameters in the model (if applicable)
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time per hour of audio

=== I / O ===

The submitted algorithm must take as input argument a path to a folder with .wav files and output predicted lyrics to a destination folder, each result file named as the corresponding input, with the extension changed.

==== Input Audio ====

Participating algorithms will have to receive the following audio input format:

- Musical tracks with accompaniment (polyphonic, not vocals only)
- 16-bit PCM <code>wav</code> files
- 44.1 kHz sampling rate
- stereo (2 channels)

Be aware that the wav files are obtained through conversion from compressed formats (mp3 or AAC) for the test sets we considered, so the quality is not exactly that of a perfect 44.1 kHz wav file.

This format is chosen as a standard input. If your pipeline expect different characteristics, the conversion process should be part of the submitted algorithm.

==== Output File Format ====

A text file (per song) containing the lyrics, organized by lines and paragraphs.

<line1: word1> <line1: word_2> ... <line1: word_N>\n
<line2: word1> <line2: word_2> ... <line2: word_N>\n
\n
<line3: word1> <line3: word_2> ... <line3: word_N>\n
<line4: word1> <line4: word_2> ... <line4: word_N>\n

Words will be identified by splitting the text file at spaces, tabs, and newline characters.
The submission can also simply generate a file of words separated by spaces, but it will then perform (very) poorly on the new metrics taken from [[#bib-cifka-2024|[1]]].

=== API-based submissions ===

For systems that rely on commercial LLM APIs (e.g. systems similar to [[#bib-lyricwhiz|[2]]]), participants must specify the exact model version and, where possible, provide a fallback local model to allow offline evaluation.
The algorithm should either already contain an API key allowing to use the model (we recommend setting an access token with a clear duration or computational limit), or use the fallback option described hereafter.

=== Fallback: Pre-computed lyrics submission ===

In situations where the inference cannot be run directly by the Task Captain, participants may provide the lyrics files produced by their algorithms.
This should only be considered a fallback option in case the algorithm relies on proprietary or commercial resources that cannot be made accessible to the Task Captains.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

In this challenge, the participants are encouraged but '''not obliged''' to use the open source dataset DALI described below.

The DAMP dataset is unfortunately [https://ccrma.stanford.edu/damp/ no longer available].

Participants are free to use other datasets for training, as long as they are described in the submission and don't overlap with the evaluation sets.

=== DALI Dataset ===

DALI (a large '''D'''ataset of synchronised '''A'''udio, '''L'''yr'''I'''cs and notes) [[#bib-syed-2025|[3]]] is the benchmark dataset for building an acoustic model on polyphonic recordings [[#bib-gupta-2020|[4]]], [[#bib-basak-2021|[5]]], [[#bib-demirel-2021|[6]]] and it contains over 7000 songs with semi-automatically aligned lyrics annotations.

The songs are commercial recordings in full-duration, whereas the lyrics are described according to different levels of granularity including words and notes (and syllables underlying a given note).

For each song DALI provides a link to a matched youtube video for the audio retrieval.

* For more details, see its full description [https://github.com/gabolsgabs/DALI here]. Paper [https://arxiv.org/pdf/1906.10606.pdf here].

= Evaluation Datasets =

The datasets listed below are reserved exclusively for evaluation purposes and '''must not''' be used for training models under any circumstances.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and lead to data leakage of some sort.

=== Jam-ALT dataset ===

The Jam-ALT dataset is built upon the Jamendo dataset [[#bib-stoller-2019|[7]]] and contains 79 songs in 4 languages: English, French, German, and Spanish. All tracks include instrumental accompaniment.
The lyrics were cleaned and their format harmonized in [[#bib-cifka-2024|[1]]].

If you’re working with English-only models, only the 20 English songs will be used for evaluation.

The <code>mp3</code> tracks from the original dataset were converted to 16-bit PCM <code>wav</code> at 44.1 kHz sample rate for this MIREX challenge.

=== MUSDB-ALT ===

Introduced in [[#bib-syed-2025|[3]]], this dataset contains 39 English songs and will be used as an additional evaluation set.

We use the dataset in its [https://sigsep.github.io/datasets/musdb.html#musdb18-compressed-stems compressed stems] version, which means that the spectral energy is null above 16kHz.
We extracted the full mixture from the Native Instruments stems files and exported them as 16-bit PCM <code>wav</code> files with 44.1 kHz sample rate.

=== Undisclosed Evaluation Set ===

In addition to Jam-ALT and MUSDB-ALT, the submissions will be evaluated on an in-house dataset of 10 English songs from the Western Popular Music repertoire, manually assembled and verified by the Task Captain.

= Time and hardware limits =
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.
A hard limit of 24 hours will be imposed on analysis times, using a single GPU with at most 32GB of VRAM (V100 or similar). Submissions that require more resources can be submitted using the fallback method described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Cífka, O., et al. (2024). Lyrics Transcription for Humans: A Readability-Aware Benchmark. Proc. of the 25th ISMIR Conf.

[2] Zhuo, L., et al. (2023). LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT. Proc. of the 24th ISMIR Conf.

[3] Syed, J., et al. (2025). Exploiting Music Source Separation for Automatic Lyrics Transcription with Whisper. ICMEW.

[4] Gupta, C., Yılmaz, E., & Li, H. (2020). Automatic lyrics alignment and transcription in polyphonic music: Does background music help? In ICASSP 2020, 496-500. IEEE.

[5] Basak, S., Agarwal, S., Ganapathy, S., & Takahashi, N. (2021, June). End-to-End Lyrics Recognition with Voice to Singing Style Transfer. In ICASSP 2021, 266-270. IEEE.

[6] Demirel, E., Ahlbäck, S., & Dixon, S. (2021). MSTRE-Net: Multistreaming Acoustic Modeling for Automatic Lyrics Transcription. Proc. ISMIR 2021.

[7] Stoller, D., Durand, S., & Ewert, S. (2019). End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-Character Recognition Model. In ICASSP 2019, IEEE.

2026:Audio-to-Score Transcription

2026-06-13T06:14:18Z

Alexandre DHooge: Create draft wiki page

= Description =

This page describes the '''MIREX2026: Audio-to-Score Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.

Automatic Music Transcription (AMT) is the task of designing computational algorithms that convert acoustic music signals into a symbolic musical representation [[#bib-benetos-2019|[2]]]. Existing MIREX tasks have evaluated several important AMT components, including melody extraction, drum detection, piano transcription, and multi-instrument transcription. However, these tasks have generally stopped at MIDI or note-event outputs, which are not equivalent to a well-formed musical score.

The Audio-to-Score Transcription (A2S) challenge focuses on this missing step. Participating systems will receive an audio recording of a polyphonic music piece and must output a digital symbolic score that can be read by a musician or notation program. The target output is a kern score encoding standard musical information such as pitches, durations, accidentals, staves, meter, clef, key signature, and bar structure.

The task can be informally expressed as follows:

Prediction('''S''') = argmax P('''S'''|'''X''')

where '''S''' is a symbolic score and '''X''' is the input audio signal.

The challenge will include polyphonic recordings and may include multiple instruments. Pieces may be full recordings or shorter excerpts, such as 3 to 6 bars.

Two tracks will be considered:

* '''Staves-Informed A2S''': staves metadata is provided and may be used to guide voice identification and instrument recognition.
* '''Blind A2S''': only the audio recording is provided.

Participants may enter both tracks. Results will be reported separately.

'''Notice:''' We particularly encourage submissions that move beyond isolated AMT components and produce complete, valid symbolic scores.

= Evaluation =

Submissions will be evaluated with metrics that assess both the textual quality of the produced kern files and the musical transcription quality of the corresponding scores.

=== Score Quality ===

'''Character Error Rate''' (CER): the Levenshtein distance between the submitted and ground-truth kern files at the character level. This metric penalizes character insertions, deletions, and substitutions, and is sensitive to details such as pitch spelling and note durations.

'''Word Error Rate''' (WER): the Levenshtein distance computed on tab- and newline-separated strings in a kern file. WER is reported in recent A2S papers [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. Compared to CER, it evaluates the output closer to the note-token level, checking whether durations and pitches are jointly correct.

'''Line Error Rate''' (LER): the Levenshtein distance computed on complete lines of the output kern files. LER is stricter than CER and WER, and helps verify whether multiple instruments or voices are correctly transcribed and aligned at the beat and sub-beat level.

=== Musical Transcription Quality ===

In addition to CER, WER, and LER, this challenge will use the '''MV2H''' metric introduced in [[#bib-mcleod-2018|[4]]] and used in recent A2S work [[#bib-alfaro-2024|[1]]], [[#bib-liu-2021|[3]]]. MV2H is an F-score obtained as the arithmetic mean of five sub-metrics:

* '''Multi-pitch detection''': pitches must be correct and detected at onsets within a 50 ms error threshold.
* '''Voice separation''': notes belonging to the same voice or instrument should be grouped together.
* '''Metrical alignment''': bars, beats, and sub-beats should be correctly identified.
* '''Note value detection''': note durations should be correctly transcribed.
* '''Harmonic analysis''': average of a key detection score and a chord-symbol recall.

The harmonic analysis component will not be used for the initial edition because no chord information is available in the considered datasets.

Public implementations of MV2H are available:

* [https://github.com/apmcleod/MV2H Official MV2H repository]
* [https://github.com/lucasmpaim/pyMV2H Python implementation]

The evaluation pipeline will adapt these implementations to process kern files, building where appropriate on the public code from [[#bib-alfaro-2024|[1]]]:

* [https://github.com/mariaalfaroc/a2s-transformer A2S Transformer repository]

'''Note that WER and MV2H will be retained as the main ranking criteria.''' CER and LER will be reported for more detailed comparisons.

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process the evaluation set.

= Submission Format =

Several submission formats will be accepted to accommodate different system designs.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size
* The number of parameters in the model, if applicable
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time required for the evaluation set

We strongly recommend sharing the submitted algorithms or checkpoints under permissive open licenses, but any licensing (or even not sharing publicly) is accepted as long as it is clearly reported at the time of submission.

=== I / O ===

The submitted algorithm must take as input an audio file, or a folder containing audio files, and write one predicted <code>**kern</code> file for each input audio file to a specified output directory.

==== Input Audio ====

Participating algorithms will receive audio files in the following format:

TBD

For the '''Blind A2S''' track, only the audio recording will be provided.

For the '''Staves-Informed A2S''' track, staves metadata will also be provided. This metadata may include meter, clef, key signature, and other score-structure information needed to guide transcription. The exact metadata packaging will be specified before submissions open.

==== Output File Format ====

Each submitted prediction must be a valid <code>**kern</code> file.

The score must encode, at minimum:

* Staves musical metadata, including meter, clef, and key signature when required
* Pitches, including octaves and accidentals
* Note durations
* Bar lines and metrical structure
* Voice or instrument organization when applicable

For more details, participants may refer to:

* [https://www.humdrum.org/Humdrum/representations/kern.html Humdrum kern representation]

We chose the <code>**kern</code> format for compatibility with existing SOTA work in A2S. Participants may use other formats during training or as intermediate outputs, but the final files should be in <code>**kern</code> format for evaluation.
Please make sure the proposed algorithm include a conversion step if it is required.

=== Code Submissions ===

Participants may provide a Docker container or access to a code repository with clear environment setup instructions and a script that reads the input audio and writes output kern files to the requested destination directory.

The submitted script should be documented in the README and should not require manual intervention during evaluation.

=== Fallback: Pre-computed Kern Submission ===

Participants may alternatively submit the final kern files directly, similarly to the option proposed for MIREX 2024 Polyphonic Transcription.

This fallback is intended to allow participation from systems that cannot be evaluated directly by the Task Captain because of resource, licensing, or infrastructure constraints.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

Participants are free to use the training and validation sets of the datasets listed below. Data augmentation is allowed.

Additional training data may also be used, as long as it does not overlap with the evaluation sets and is clearly documented in the submission.

=== Quartets Dataset ===

The Quartets dataset consists of synthetic audio renderings of Haydn, Mozart, and Beethoven string quartets, together with their full kern scores [[#bib-roman-2019|[6]]].

The scores are taken from the [https://github.com/humdrum-tools/humdrum-data humdrum-data repository]. The kern files were split into 3 to 6 measure excerpts and synthesized into audio from performance MIDI files.

The final dataset contains approximately 20 hours of audio for 38,051 excerpts and 3 composers:

* 18,162 excerpts for Haydn
* 7,435 excerpts for Mozart
* 12,454 excerpts for Beethoven

The train, validation, and test splits used in [[#bib-alfaro-2024|[1]]] are publicly available:

* [https://huggingface.co/datasets/PRAIG/quartets-quartets Quartets dataset on Hugging Face]

=== MuseSyn Dataset ===

The MuseSyn dataset contains 210 piano pieces with scores in MusicXML format and audio synthesized through four different piano models [[#bib-liu-2021|[3]]]. It amounts to almost 10 hours of audio recordings for each piano timbre.

The pieces cover a wide range of key signatures, time signatures, tempos, and polyphony levels. The dataset is available upon request for non-commercial research use:

* [https://zenodo.org/records/4527460 MuseSyn on Zenodo]

For this challenge, the MusicXML files will be converted to kern format, with manual verification where needed, to unify the evaluation pipeline.

= Evaluation Datasets =

The datasets listed below are reserved for evaluation purposes and '''must not''' be used for training models.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and could lead to data leakage.

=== Quartets Test Set ===

Evaluation will include the test split of the Quartets dataset. Results on this dataset will be reported separately.

=== MuseSyn Test Set ===

Evaluation will include the test split of the MuseSyn dataset after conversion of the reference scores to kern format. Results on this dataset will be reported separately.

=== Undisclosed Evaluation Set ===

An additional custom evaluation set will be used to ensure fairness and out-of-distribution data.

General details on this dataset are:
TBD

= Time and Hardware Limits =

Due to the potentially high number of participants in this and other audio tasks, hard limits on runtime and hardware use will be imposed.

Submissions that require more than 32 GB of VRAM, or more than 24 hours to process the test sets on a single GPU (V100 or similar), cannot be evaluated directly by the Task Captain. Participants in this situation may use the fallback kern submission format described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Alfaro-Contreras, M., et al. (2024). A Transformer Approach for Polyphonic Audio-to-Score Transcription. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2] Benetos, E., et al. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1).

[3] Liu, L., et al. (2021). Joint Multi-Pitch Detection and Score Transcription for Polyphonic Piano Music. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4] Mcleod, A., et al. (2018). Evaluating Automatic Polyphonic Music Transcription. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[5] Roman, M. A., et al. (2018). An End-to-End Framework for Audio-to-Score Music Transcription on Monophonic Excerpts. Proc. of the 19th International Society for Music Information Retrieval Conference (ISMIR).

[6] Roman, M. A., et al. (2019). A Holistic Approach to Polyphonic Music Transcription with Neural Networks. Proc. of the 20th International Society for Music Information Retrieval Conference (ISMIR).

[7] Smaragdis, P., et al. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.

2026:Lyrics Transcription

2026-06-13T05:43:25Z

Alexandre DHooge: add missing bib entries

= Description =

This page describes the '''MIREX2026: Automatic Lyrics Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.
''It is strongly based upon the page of the 2025 ALT challenge.''

The task of Lyrics Transcription aims to identify the words from sung utterances, in the same way as in automatic speech recognition. This can be mathematically expressed as follows:

Prediction('''w''') = argmax P('''w'''|'''X''')

where '''w''' and '''X''' are the word and acoustic features respectively.

Ideally, the lyrics transcriber should return meaningful word sequences:

Prediction('''w''') = [ <w_1>, <w_2>, ..., <w_N> ]

Note that for this year's edition, the input will always be a polyphonic mix (singing voice + musical accompaniment). The submitted algorithms can include a source-separation step if needed.

'''Notice:''' We particularly encourage participations that build on the latest ALT approaches such as using ''pretrained audio foundation models'' or ''LLMs''.

= Evaluation =

This year's edition will include the following metrics for evaluation:

'''Word Error Rate''' (WER): the standard metric used in Automatic Speech Recognition.

WER = (S + I + D) / (C + S + D)

where;
C : correctly predicted words
S : substitution errors
I : insertion errors
D : deletion errors

'''Character Error Rate''' (CER): the above computation can also be done on the character level. This metric penalises the partially correctly predicted / incorrectly spelled words less than WER.

'''Case-Sensitive WER (WER')''': defined in [[#bib-cifka-2024|[1]]], this metric is computed as follows:

<math>
\mathrm{WER'} = \mathrm{WER} + \frac{E_\text{case}}{N}
</math>

where <math>N</math> is the total number of words in a ground-truth lyrics file, and <math>E_\text{case}</math> the number of "casing errors" i.e. words that differ in a ''case-sensitive'' setting like "city" and "City".

'''Punctuation, Parentheses, Line and Section Breaks''': also defined in [[#bib-cifka-2024|[1]]], we include a set of metrics to measure how well the proposed algorithms predict several formatting tokens. The tokens are classified into one of 5 types <math>T\in {W,P,B,L,S}</math> for Word (W), Punctuation (P), Parentheses (B), Line Breaks (L), and Section Breaks (S). Then, for each token type (except Word), we compute Precision, Recall, and F1-Score:

<math>
P_T = \frac{C_T}{C_T + S_T + I_T}
</math>

<math>
R_T = \frac{C_T}{C_T + S_T + D_T}
</math>

<math>
F_T = \frac{2}{P_T^{-1} + R_T^{-1}}
</math>

All these metrics will be based on the [https://github.com/audioshake/alt-eval public code implementation] from [[#bib-cifka-2024|[1]]], simply extended to include CER.

'''Note that WER and CER will be retained as the main ranking criteria, other metrics will be reported for more detailed comparisons.'''

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process a sample.

= Submission Format =

Several submission formats will be accepted to accommodate the different approaches chosen by the participants.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size (hours)
* The number of parameters in the model (if applicable)
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time per hour of audio

=== I / O ===

The submitted algorithm must take as input argument a path to a folder with .wav files and output predicted lyrics to a destination folder, each result file named as the corresponding input, with the extension changed.

==== Input Audio ====

Participating algorithms will have to receive the following input format:

TBD

==== Output File Format ====

A text file (per song) containing the lyrics, organized by lines and paragraphs.

<line1: word1> <line1: word_2> ... <line1: word_N>\n
<line2: word1> <line2: word_2> ... <line2: word_N>\n
\n
<line3: word1> <line3: word_2> ... <line3: word_N>\n
<line4: word1> <line4: word_2> ... <line4: word_N>\n

Words will be identified by splitting the text file at spaces, tabs, and newline characters.
The submission can also simply generate a file of words separated by spaces, but it will then perform (very) poorly on the new metrics taken from [[#bib-cifka-2024|[1]]].

=== API-based submissions ===

For systems that rely on commercial LLM APIs (e.g. systems similar to [[#bib-lyricwhiz|[2]]]), participants must specify the exact model version and, where possible, provide a fallback local model to allow offline evaluation.
The algorithm should either already contain an API key allowing to use the model (we recommend setting an access token with a clear duration or computational limit), or use the fallback option described hereafter.

=== Fallback: Pre-computed lyrics submission ===

In situations where the inference cannot be run directly by the Task Captain, participants may provide the lyrics files produced by their algorithms.
This should only be considered a fallback option in case the algorithm relies on proprietary or commercial resources that cannot be made accessible to the Task Captains.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

In this challenge, the participants are encouraged but '''not obliged''' to use the open source dataset DALI described below.

The DAMP dataset is unfortunately [https://ccrma.stanford.edu/damp/ no longer available].

Participants are free to use other datasets for training, as long as they are described in the submission and don't overlap with the evaluation sets.

=== DALI Dataset ===

DALI (a large '''D'''ataset of synchronised '''A'''udio, '''L'''yr'''I'''cs and notes) [[#bib-syed-2025|[3]]] is the benchmark dataset for building an acoustic model on polyphonic recordings [[#bib-gupta-2020|[4]]], [[#bib-basak-2021|[5]]], [[#bib-demirel-2021|[6]]] and it contains over 7000 songs with semi-automatically aligned lyrics annotations.

The songs are commercial recordings in full-duration, whereas the lyrics are described according to different levels of granularity including words and notes (and syllables underlying a given note).

For each song DALI provides a link to a matched youtube video for the audio retrieval.

* For more details, see its full description [https://github.com/gabolsgabs/DALI here]. Paper [https://arxiv.org/pdf/1906.10606.pdf here].

= Evaluation Datasets =

The datasets listed below are reserved exclusively for evaluation purposes and '''must not''' be used for training models under any circumstances.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and lead to data leakage of some sort.

=== Jam-ALT dataset ===

The Jam-ALT dataset is built upon the Jamendo dataset [[#bib-stoller-2019|[7]]] and contains 79 songs in 4 languages: English, French, German, and Spanish. All tracks include instrumental accompaniment.
The lyrics were cleaned and their format harmonized in [[#bib-cifka-2024|[1]]].

If you’re working with English-only models, only the 20 English songs will be used for evaluation.

=== MUSDB-ALT ===

Introduced in [[#bib-syed-2025|[3]]], this dataset contains 39 English songs and will be used as an additional evaluation set.

=== Undisclosed Evaluation Set ===

In addition to Jam-ALT and MUSDB-ALT, the submissions will be evaluated on an in-house dataset of 10 English songs from the Western Popular Music repertoire, manually assembled and verified by the Task Captain.

= Time and hardware limits =
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.
A hard limit of 24 hours will be imposed on analysis times, using a single GPU with at most 32GB of VRAM (V100 or similar). Submissions that require more resources can be submitted using the fallback method described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Cífka, O., et al. (2024). Lyrics Transcription for Humans: A Readability-Aware Benchmark. Proc. of the 25th ISMIR Conf.

[2] Zhuo, L., et al. (2023). LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT. Proc. of the 24th ISMIR Conf.

[3] Syed, J., et al. (2025). Exploiting Music Source Separation for Automatic Lyrics Transcription with Whisper. ICMEW.

[4] Gupta, C., Yılmaz, E., & Li, H. (2020). Automatic lyrics alignment and transcription in polyphonic music: Does background music help? In ICASSP 2020, 496-500. IEEE.

[5] Basak, S., Agarwal, S., Ganapathy, S., & Takahashi, N. (2021, June). End-to-End Lyrics Recognition with Voice to Singing Style Transfer. In ICASSP 2021, 266-270. IEEE.

[6] Demirel, E., Ahlbäck, S., & Dixon, S. (2021). MSTRE-Net: Multistreaming Acoustic Modeling for Automatic Lyrics Transcription. Proc. ISMIR 2021.

[7] Stoller, D., Durand, S., & Ewert, S. (2019). End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-Character Recognition Model. In ICASSP 2019, IEEE.

2026:Lyrics Transcription

2026-06-13T05:33:06Z

Alexandre DHooge: Draft version of the page

= Description =

This page describes the '''MIREX2026: Automatic Lyrics Transcription''' challenge. For evaluation procedure and the submission format please scroll down the page.
''It is strongly based upon the page of the 2025 ALT challenge.''

The task of Lyrics Transcription aims to identify the words from sung utterances, in the same way as in automatic speech recognition. This can be mathematically expressed as follows:

Prediction('''w''') = argmax P('''w'''|'''X''')

where '''w''' and '''X''' are the word and acoustic features respectively.

Ideally, the lyrics transcriber should return meaningful word sequences:

Prediction('''w''') = [ <w_1>, <w_2>, ..., <w_N> ]

Note that for this year's edition, the input will always be a polyphonic mix (singing voice + musical accompaniment). The submitted algorithms can include a source-separation step if needed.

'''Notice:''' We particularly encourage participations that build on the latest ALT approaches such as using ''pretrained audio foundation models'' or ''LLMs''.

= Evaluation =

This year's edition will include the following metrics for evaluation:

'''Word Error Rate''' (WER): the standard metric used in Automatic Speech Recognition.

WER = (S + I + D) / (C + S + D)

where;
C : correctly predicted words
S : substitution errors
I : insertion errors
D : deletion errors

'''Character Error Rate''' (CER): the above computation can also be done on the character level. This metric penalises the partially correctly predicted / incorrectly spelled words less than WER.

'''Case-Sensitive WER (WER')''': defined in [[#bib-cifka-2024|[1]]], this metric is computed as follows:

<math>
\mathrm{WER'} = \mathrm{WER} + \frac{E_\text{case}}{N}
</math>

where <math>N</math> is the total number of words in a ground-truth lyrics file, and <math>E_\text{case}</math> the number of "casing errors" i.e. words that differ in a ''case-sensitive'' setting like "city" and "City".

'''Punctuation, Parentheses, Line and Section Breaks''': also defined in [[#bib-cifka-2024|[1]]], we include a set of metrics to measure how well the proposed algorithms predict several formatting tokens. The tokens are classified into one of 5 types <math>T\in {W,P,B,L,S}</math> for Word (W), Punctuation (P), Parentheses (B), Line Breaks (L), and Section Breaks (S). Then, for each token type (except Word), we compute Precision, Recall, and F1-Score:

<math>
P_T = \frac{C_T}{C_T + S_T + I_T}
</math>

<math>
R_T = \frac{C_T}{C_T + S_T + D_T}
</math>

<math>
F_T = \frac{2}{P_T^{-1} + R_T^{-1}}
</math>

All these metrics will be based on the [https://github.com/audioshake/alt-eval public code implementation] from [[#bib-cifka-2024|[1]]], simply extended to include CER.

'''Note that WER and CER will be retained as the main ranking criteria, other metrics will be reported for more detailed comparisons.'''

----

In addition to these performance metrics, each submission will be evaluated in terms of memory use, number of operations, and computational time required to process a sample.

= Submission Format =

Several submission formats will be accepted to accommodate the different approaches chosen by the participants.
If you encounter any issue with the submission process, please contact the Task Captain.

=== General Guidelines ===

All submissions should be "plug-and-play", with a clear README detailing usage steps.

The recommended submission format is a Docker image or a code repository with a main bash or Python script to run.

'''Resources Declaration''': All submissions must state:
* The training data size (hours)
* The number of parameters in the model (if applicable)
* The amount of GPU/CPU hours used for training, with device information (model and VRAM)
* The inference time per hour of audio

=== I / O ===

The submitted algorithm must take as input argument a path to a folder with .wav files and output predicted lyrics to a destination folder, each result file named as the corresponding input, with the extension changed.

==== Input Audio ====

Participating algorithms will have to receive the following input format:

TBD

==== Output File Format ====

A text file (per song) containing the lyrics, organized by lines and paragraphs.

<line1: word1> <line1: word_2> ... <line1: word_N>\n
<line2: word1> <line2: word_2> ... <line2: word_N>\n
\n
<line3: word1> <line3: word_2> ... <line3: word_N>\n
<line4: word1> <line4: word_2> ... <line4: word_N>\n

Words will be identified by splitting the text file at spaces, tabs, and newline characters.
The submission can also simply generate a file of words separated by spaces, but it will then perform (very) poorly on the new metrics taken from [[#bib-cifka-2024|[1]]].

=== API-based submissions ===

For systems that rely on commercial LLM APIs (e.g. systems similar to [[#bib-lyricwhiz|[2]]]), participants must specify the exact model version and, where possible, provide a fallback local model to allow offline evaluation.
The algorithm should either already contain an API key allowing to use the model (we recommend setting an access token with a clear duration or computational limit), or use the fallback option described hereafter.

=== Fallback: Pre-computed lyrics submission ===

In situations where the inference cannot be run directly by the Task Captain, participants may provide the lyrics files produced by their algorithms.
This should only be considered a fallback option in case the algorithm relies on proprietary or commercial resources that cannot be made accessible to the Task Captains.
''Such submissions will be clearly flagged in the results page.''

= Training Datasets =

In this challenge, the participants are encouraged but '''not obliged''' to use the open source dataset DALI described below.

The DAMP dataset is unfortunately [https://ccrma.stanford.edu/damp/ no longer available].

Participants are free to use other datasets for training, as long as they are described in the submission and don't overlap with the evaluation sets.

=== DALI Dataset ===

DALI (a large '''D'''ataset of synchronised '''A'''udio, '''L'''yr'''I'''cs and notes) [[#bib-syed-2025|[3]]] is the benchmark dataset for building an acoustic model on polyphonic recordings [[#bib-gupta-2020|[4]]], [[#bib-basak-2021|[5]]], [[#bib-demirel-2021|[6]]] and it contains over 7000 songs with semi-automatically aligned lyrics annotations.

The songs are commercial recordings in full-duration, whereas the lyrics are described according to different levels of granularity including words and notes (and syllables underlying a given note).

For each song DALI provides a link to a matched youtube video for the audio retrieval.

* For more details, see its full description [https://github.com/gabolsgabs/DALI here]. Paper [https://arxiv.org/pdf/1906.10606.pdf here].

= Evaluation Datasets =

The datasets listed below are reserved exclusively for evaluation purposes and '''must not''' be used for training models under any circumstances.

We also request participants to refrain from checking the performance of their algorithms on these sets before submission, as it would make them equivalent to validation sets and lead to data leakage of some sort.

=== Jam-ALT dataset ===

The Jam-ALT dataset is built upon the Jamendo dataset [[#bib-stoller-2019|[7]]] and contains 79 songs in 4 languages: English, French, German, and Spanish. All tracks include instrumental accompaniment.
The lyrics were cleaned and their format harmonized in [[#bib-cifka-2024|[1]]].

If you’re working with English-only models, only the 20 English songs will be used for evaluation.

=== MUSDB-ALT ===

Introduced in [[#bib-syed-2025|[3]]], this dataset contains 39 English songs and will be used as an additional evaluation set.

=== Undisclosed Evaluation Set ===

In addition to Jam-ALT and MUSDB-ALT, the submissions will be evaluated on an in-house dataset of 10 English songs from the Western Popular Music repertoire, manually assembled and verified by the Task Captain.

= Time and hardware limits =
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.
A hard limit of 24 hours will be imposed on analysis times, using a single GPU with at most 32GB of VRAM (V100 or similar). Submissions that require more resources can be submitted using the fallback method described above.

= Questions? =

* Contact Alexandre D'Hooge (Alex, he/him): dhooge[at]gbu[dot]edu[dot]cn

= Bibliography =

[1] Cífka, O., et al. (2024). Lyrics Transcription for Humans: A Readability-Aware Benchmark. Proc. of the 25th ISMIR Conf.

[2] LyricWhiz — TODO.

[3] Syed et al. (2025). TODO.

[4] Gupta, C., Yılmaz, E., & Li, H. (2020). Automatic lyrics alignment and transcription in polyphonic music: Does background music help? In ICASSP 2020, 496-500. IEEE.

[5] Basak, S., Agarwal, S., Ganapathy, S., & Takahashi, N. (2021, June). End-to-End Lyrics Recognition with Voice to Singing Style Transfer. In ICASSP 2021, 266-270. IEEE.

[6] Demirel, E., Ahlbäck, S., & Dixon, S. (2021). MSTRE-Net: Multistreaming Acoustic Modeling for Automatic Lyrics Transcription. Proc. ISMIR 2021.

[7] Stoller, D., Durand, S., & Ewert, S. (2019). End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-Character Recognition Model. In ICASSP 2019, IEEE.