MIREX Wiki - Recent changes [en]

MIREX HOME

Junyan — Fri, 03 Jul 2026 12:28:57 GMT

‎Task Descriptions

2026:Music Structure Analysis

Junyan — Thu, 02 Jul 2026 16:49:15 GMT

‎Baseline

2026:AI-Generated Music Detection

Ldzhangyx — Thu, 02 Jul 2026 14:10:08 GMT

‎Candidate Dataset Size Summary

2025:Symbolic Music Generation Results

Xinyue2896 — Wed, 01 Jul 2026 10:12:17 GMT

2026:Music Performance Difficulty Prediction

Huanz — Wed, 01 Jul 2026 09:36:49 GMT

‎Training Datasets

2026:Audio Instrument Recognition

Chestnut — Wed, 01 Jul 2026 04:57:25 GMT

MIREX HOME

Junyan — Wed, 01 Jul 2026 00:54:10 GMT

‎Task Descriptions

2026:Lyrics-to-Audio Alignment

Junyan — Mon, 29 Jun 2026 19:31:13 GMT

Created page with "==Description== Standardized Evaluation Suites: [https://github.com/ismir-mirex/mirex-evaluation ismir-mirex/mirex-evaluation]. The task of automatic lyrics-to-audio alignm..."

New page

==Description==

Standardized Evaluation Suites: [https://github.com/ismir-mirex/mirex-evaluation ismir-mirex/mirex-evaluation].

The task of automatic lyrics-to-audio alignment has as an end goal the synchronization between an audio recording of singing and its corresponding written lyrics. The beginning timestamps of lyrics units can be estimated on different granularity: phonemes, words, lyrics lines, phrases. For this task word-level alignment is required.

----------------------- ---------------------------------------------------
| Mixed singing audio | | Lyrics at word-level: no more carefree ... ... |
----------------------- ---------------------------------------------------
| |
--------------------------------------------
|
--------------------
| Alignment system |
--------------------
|
|
--------------------------
| 0.123 0.798 no |
| 0.798 1.123 more |
| 1.345 2.176 carefree|
| ... ... |
--------------------------
The algorithm receives two inputs - mixed singing audio (singing voice + musical accompaniment) and its corresponding lyrics at word-level, outputs the onset and offset timestamps (second) of each word.

== What's New ==

Compared to previous years:

* Submission format: docker image is required. See the submission format section.

== Data ==

=== Test Data ===

Participants are not allowed to use any split of the following datasets for training, validation, model selection, parameter tuning, or any other development purpose:

'''Jamendo''': This dataset contains 20 full-duration music pieces with 10 different Western music genres, annotated with start-of-word timestamps. All songs have instrumental accompaniment. It is available online on [https://github.com/f90/jamendolyrics Github]. Because Jamendo is held out for this task, it must not be used for training, validation, model selection, or parameter tuning. For more information also refer to [https://arxiv.org/abs/1902.06797 this paper].

* file duration up to 4:43 (total time: 1h 12m)
* 5677 words annotated in total

=== Other Collections ===

Other collections may be used, but participants must clearly state their dataset usage and, when applicable, the data collection process in the extended abstract.

==== DAMP dataset ====
The DAMP dataset contains a large number (34 000+) of a cappella recordings from a wide variety of amateur singers, collected with the Sing! Karaoke mobile app in different recording conditions, but generally with good audio quality. A carefully curated subset DAMPB of 20 performances of each of the 300 songs has been created by (Kruspe, 2016). Here is the [https://docs.google.com/spreadsheets/d/1YwhPhXU6t-BMZfdEODS_pNW_umFIsciYL62kh-fiBWI/edit?usp=sharing list of recordings].

* The audio can be downloaded from the [https://ccrma.stanford.edu/damp/ Smule web site]
* No lyrics boundary annotations are available, still the textual lyrics are on the [https://www.smule.com/songs Smule Sing! Karaoke website]

==== DALI Dataset ====

The DALI dataset (a large '''D'''ataset of synchronised '''A'''udio, '''L'''yr'''I'''cs and notes) contains over 5000 songs with semi-automatically aligned lyrics annotations. The songs are commercial recordings in full-duration, whereas the lyrics are described according to different levels of granularity including words and notes (and syllables underlying a given note). For each song DALI provides a link to a matched youtube video, from which the audio could be retrieved.
For more details how, see its full description [https://github.com/gabolsgabs/DALI here].

==== Hansen's Dataset ====
The dataset contains 9 pop music songs in English with annotations of both beginnings- and ending-timestamps of each word. The ending timestamps are for convenience (copies of next word's beginning timestamp) and are not used in the evaluation. Sentence-level annotations are also provided.
The audio has two versions: the original mix with instrumental accompaniment and a cappella singing voice only one. An example song can be seen [https://www.dropbox.com/sh/wm6k4dqrww0fket/AAC1o1uRFxBPg9iAeSAd1Wxta?dl=0 here]

You can read in detail about how the dataset was made here: [http://smcnetwork.org/system/files/smc2012-198.pdf Recognition of Phonemes in A-cappella Recordings using Temporal Patterns and Mel Frequency Cepstral Coefficients]. The dataset has been kindly provided by Jens Kofod Hansen.

* file duration up to 4:40 minutes (total time: 35:33 minutes)
* 3590 words annotated in total

==== Mauch's Dataset ====
The dataset contains 20 pop music songs in English with annotations of beginning-timestamps of each word. Non-vocal sections are not explicitly annotated (but remain included in the last preceding word). We prefer to leave it this way, to enable comparison to previous work, evaluated on this dataset.
The audio has instrumental accompaniment. An example song can be seen [https://www.dropbox.com/sh/8pp4u2xg93z36d4/AAAsCE2eYW68gxRhKiPH_VvFa?dl=0 here].

You can read in detail about how the dataset was used for the first time here: [https://pdfs.semanticscholar.org/547d/7a5d105380562ca3543bf05b4d5f7a8bee66.pdf Integrating Additional Chord Information Into HMM-Based Lyrics-to-Audio Alignment]. The dataset has been kindly provided by Sungkyun Chang.

* file duration up to 5:40 minutes (total time: 1h 19m)
* 5050 words annotated in total

==== Gracenote Dataset ====
The dataset contains 8 pop music song excerpts with instrumental accompaniment, with annotations of beginning-timestamps of each word. The dataset has been used in the recent [https://ieeexplore.ieee.org/abstract/document/7952235/references paper].

* file duration up to 1:11 (total time: 11m)
* 1181 words annotated in total

=== Phonetization ===
A popular choice for phonetization of the words is the [http://www.speech.cs.cmu.edu/cgi-bin/cmudict CMU pronunciation dictionary]. One can phonetize them with the [http://www.speech.cs.cmu.edu/tools/lextool.html online tool]. A list of all words of both datasets, which are outside of the [https://github.com/georgid/AlignmentDuration/blob/noteOnsets/src/for_english/cmudict.0.6d.syll list of CMU words] is given [https://www.dropbox.com/s/flu4cpqff916bas/words_not_in_dict?dl=0 here].

=== Audio Format ===

The data are sound wav/mp3 files, plus the associated word boundaries (in csv-like .txt/.tsv files)

* CD-quality (PCM, 16-bit, 44100 Hz)
* single channel (mono) for a cappella and two channels for original

==Evaluation==

The submitted algorithms will be evaluated at the word boundaries for the originally mixed songs (a cappella singing + instrumental accompaniment). Evaluation metrics on the a cappella singing can be reported as well on request, for the sake of getting insights on the impact of instrumental accompaniment on the algorithm, but will not be considered for the ranking.

'''Average absolute error/deviation''' Initially utilized in [http://www.cs.tut.fi/~mesaros/pubs/autalign_cr.pdf Mesaros and Virtanen (2008)], the absolute error measures the time displacement between the actual timestamp and its estimate at the beginning and the end of each lyrical unit. The error is then averaged over all individual errors. An error in absolute terms has the drawback that the perception of an error with the same duration can be different depending on the tempo of the song.
Here is a [https://github.com/georgid/AlignmentEvaluation/blob/126c3fa5fa1994acdcfbe3ea1344acfe71ae2b8e/test/EvalMetricsTest.py#L117 test] of using this metric.

'''Percentage of correct segments''' The perceptual dependence on tempo is mitigated by measuring the percentage of the total length of the segments, labeled correctly to the total duration of the song. This metric is suggested by [https://www.researchgate.net/publication/224241940_LyricSynchronizer_Automatic_Synchronization_System_Between_Musical_Audio_Signals_and_Lyrics Fujihara et al. (2011), Figure 9].
Here is a [https://github.com/georgid/AlignmentEvaluation/blob/126c3fa5fa1994acdcfbe3ea1344acfe71ae2b8e/test/EvalMetricsTest.py#L76 test] of using this metric.

'''Percentage of correct estimates according to a tolerance window''' A metric that takes into consideration that the onset displacements from ground truth below a certain threshold could be tolerated by human listeners. We use 0.3 seconds as the tolerance window. This metric is suggested in [https://pdfs.semanticscholar.org/547d/7a5d105380562ca3543bf05b4d5f7a8bee66.pdf Integrating Additional Chord Information Into HMM-Based Lyrics-to-Audio Alignment].
Here is a [https://github.com/georgid/AlignmentEvaluation/blob/126c3fa5fa1994acdcfbe3ea1344acfe71ae2b8e/test/EvalMetricsTest.py#L151 test] of using this metric.

For more detailed definition and formulas about the metrics, please check the section 2.2.1 of [https://doi.org/10.5281/zenodo.841979 this thesis].

'''To obtain all three metrics for one detected output:'''

<code> python [https://github.com/georgid/AlignmentEvaluation/blob/master/align_eval/eval.py eval.py] <file path of the reference word boundaries> <file path of the detected word boundaries> </code>

Note that evaluation scripts depend on [https://github.com/craffel/mir_eval/ mir_eval].

== Submission Format ==

Every submission must be packed into a docker image containing the bash file <code>main.sh</code> in the root folder.

=== Input Data ===
Participating algorithms will have to receive the following input format:

* Audio in wav, 44.1kHz, stereo.
* Lyrics in .txt file where each word is separated by a space, each lyrics phrase is separated by a line break mark (\n).

=== Output File Format ===

The alignment output file format is a tab-delimited ASCII text format.

Three column text file of the format

<onset_time(sec)>\t<offset_time(sec)>\t<label>\n
<onset_time(sec)>\t<offset_time(sec)>\t<label>\n
...

where \t denotes a tab, \n denotes the end of the line. The < and > characters are not included. An example output file would look something like:

0.000 5.223 word1
5.223 15.101 word2
15.101 20.334 word3

'''NOTE:''' the offset timestamps column is utilized only by the percentage of correct segments metric. Therefore skipping the second column is acceptable, and could result in degraded performance of this respective metric only.

=== Command line calling format ===

The submitted algorithm must take as arguments .wav file, .txt file as well as the full output path and filename of the output file in the following format:

<pre>your_program %input_audio %input_txt %output_txt</pre>

=== Packaging submissions ===

Please see [[Submission Guidelines]] first if you are not familiar with docker.

* Every submission must be packed into a docker image
* Every submission will be deployed and evaluated automatically with <code>docker run</code>

Accepted submission form:

* Docker Hub: You can create a free account at [https://hub.docker.com/ Docker Hub] and upload your docker image there.
* Github Container Registry: If you are using Github, you can use the Github Container Registry to upload your docker image.
* Google Drive: You can upload your docker image to Google Drive and share the link with the evaluation organizers.

Notice that MIREX server currently does not support docker image upload. This might be supported in future years.

Here are the docker commands that will be used to run all systems:

<pre>
docker run -v dataset_folder:/app/data %input_audio %input_txt %output_txt
</pre>

== Time and hardware limits ==

A Linux server with one Nvidia GeForce RTX 3090 is used for evaluation. CPU, OS, and memory specifications will be announced later.

Time limit: within 5 times the total duration of the test set.

== Bibliography ==

Stoller, D. and Durand, S. and Ewert, S. (2019) End-to-end Lyrics Alignment for Polyphonic Music Using An Audio-to-Character Recognition Model. ICASSP 2019.

Sharma B, Gupta C. (2019) Automatic Lyrics-to-audio Alignment on Polyphonic Music Using Singing-adapted Acoustic Models. ICASSP 2019

Lee S. W., Scott, J. (2017) Word-level lyrics-audio synchronization using separated vocals", Acoustics Speech and Signal Processing, ICASSP IEEE International Conference on, pp. 646-650

Chang, S., & Lee, K. (2017). Lyrics-to-Audio Alignment by Unsupervised Discovery of Repetitive Patterns in Vowel Acoustics. arXiv preprint arXiv:1701.06078.

Pons, J. Gong, R. and Serra, X. (2017). Score-informed syllable segmentation for a cappella singing voice with convolutional neural networks. ISMIR 2017

Kruspe, A. (2016). Bootstrapping a System for Phoneme Recognition and Keyword Spotting in Unaccompanied Singing, ISMIR 2016

Dzhambazov, G. and Serra, X. (2015) Modeling of phoneme durations for alignment between polyphonic audio and lyrics, in 12th Sound and Music Computing Conference

Fujihara, H., & Goto, M. (2012). Lyrics-to-audio alignment and its application. In Dagstuhl Follow-Ups (Vol. 3). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.

Mauch, M., Fujihara, H., & Goto, M. (2012). Integrating additional chord information into HMM-based lyrics-to-audio alignment. IEEE Transactions on Audio, Speech, and Language Processing, 20(1), 200-210.

Fujihara, H. Goto, M. Ogata, J. and Okuno, H. G. (2011) Lyricsynchronizer: Automatic synchronization system between musical audio signals and lyrics, IEEE Journal of Selected Topics in Signal Processing

Mesaros, A. and Virtanen, T. (2008), Automatic alignment of music audio and lyrics, in Proceedings of the 11th Int. Conference on Digital Audio Effects (DAFx-08), Espoo, Finland, 2008.

2026:Audio Downbeat Estimation

Junyan — Mon, 29 Jun 2026 19:29:54 GMT

‎Data

2026:Music Structure Analysis

Junyan — Mon, 29 Jun 2026 19:28:43 GMT

Created page with "== Music Structure Analysis (MIREX 2026) == Standardized Evaluation Suites: [https://github.com/ismir-mirex/mirex-evaluation ismir-mirex/mirex-evaluation]. === Description..."

New page

== Music Structure Analysis (MIREX 2026) ==

Standardized Evaluation Suites: [https://github.com/ismir-mirex/mirex-evaluation ismir-mirex/mirex-evaluation].

=== Description ===

The aim of the MIREX Music Structure Analysis task is to identify and label key structural sections in musical audio. Understanding the musical form (e.g., intro, verse, chorus) is fundamental to music understanding and a crucial component in many music information retrieval applications. While traditional approaches focused on segmenting music into internally consistent, but arbitrarily labeled, sections (e.g., A, B, C), this task has evolved.

Since 2020, a new paradigm has emerged, focusing on '''functional structure analysis'''. The goal is to segment the audio and assign a specific functional label to each segment from a predefined set of common musical functions. This task challenges systems to perform both accurate boundary detection and correct functional classification.

This task builds upon a history of structural segmentation evaluations, first run in MIREX 2009. Recent works driving this updated focus include:
* Wang, J. C., Hung, Y. N., & Smith, J. B. (2022, May). To catch a chorus, verse, intro, or anything else: Analyzing a song with structural functions. In ''ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)'' (pp. 416-420). IEEE.
* Kim, T., & Nam, J. (2023, October). All-in-one metrical and functional structure analysis with neighborhood attentions on demixed audio. In ''2023 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)'' (pp. 1-5). IEEE.
* Buisson, M., McFee, B., Essid, S., & Crayencour, H. C. (2024). Self-supervised learning of multi-level audio representations for music segmentation. ''IEEE/ACM Transactions on Audio, Speech, and Language Processing''.

For MIREX 2026, participants are required to segment musical audio and classify each segment into one of seven functional categories: '''‘intro’, ‘verse’, ‘chorus’, ‘bridge’, ‘inst’ (instrumental), ‘outro’, or ‘other’'''. The 'other' category can be used for segments that do not fit into the primary six functional labels or for non-musical content if explicitly defined by the dataset annotations being mapped.

=== Data ===

Other collections may be used, but participants must clearly state their dataset usage and, when applicable, the data collection process in the extended abstract. The Harmonix Set train and validation splits may also be used for training and development, but the Harmonix Set test split is held out.

We use the relabeled Harmonix Set for evaluation. The test split is held out for evaluation, and participants may use the train and validation splits for training models.

https://huggingface.co/datasets/m-a-p/harmonixset_bigvgan/tree/main

==== Collections ====
The evaluation will utilize datasets previously established in MIREX. Annotations from these diverse collections will be mapped to the seven target functional labels for consistent evaluation.
* '''The MIREX 2009 Collection''': 297 pieces, largely derived from the work of the Beatles.
* '''MIREX 2010 RWC collection''': 100 pieces of popular music. This collection has two sets of ground truths. The first was originally included with the RWC dataset. The second set provides segment boundary annotations (see [http://hal.inria.fr/docs/00/47/34/79/PDF/PI-1948.pdf Pechuho et al., 2010] for details).
* '''MIREX 2012 dataset''': Over 1,000 annotated pieces covering a range of musical styles, with the majority annotated by two independent annotators.

Participants should be aware that original labels in these datasets (e.g., 'verse1', 'solo', 'fade-out') will need to be mapped to the seven specified functional categories for evaluation. Guidelines for this mapping will be provided, or a standard mapping will be applied during evaluation.

==== Audio Formats (Input to Algorithms) ====
Algorithms should be prepared to process audio with the following characteristics:
* Sample rate: 44.1 kHz
* Bit depth: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

=== Submission Format ===

Submissions will be handled via '''CodeBench'''. Participants are required to submit their results in a specific format, as detailed below. You will upload a single file containing the segmentation results for all test audio files.

==== Output Data Format ====
The output must be a '''list of dictionaries''' in a text-based format (e.g., JSON parsable). Each dictionary in the list corresponds to one audio file and must contain two keys: <tt>'id'</tt> (the identifier of the audio file, e.g., '1.wav') and <tt>'result'</tt> (a list of segment predictions). Each segment prediction is a list containing two elements: a two-element list with the <tt>[start_time, end_time]</tt> of the segment in seconds, and the <tt>label</tt> string for that segment.

The labels must be one of the seven target functional categories: <tt>'intro'</tt>, <tt>'verse'</tt>, <tt>'chorus'</tt>, <tt>'bridge'</tt>, <tt>'inst'</tt>, <tt>'outro'</tt>, <tt>'silence'</tt>.

Example of the content of the submitted file:
<pre>
[
{
'id': 'track01.wav',
'result': [
[[0.000, 15.500], 'intro'],
[[15.500, 45.230], 'verse'],
[[45.230, 75.800], 'chorus'],
[[75.800, 90.000], 'outro']
]
},
{
'id': 'track02.wav',
'result': [
[[0.000, 20.100], 'verse'],
[[20.100, 38.500], 'chorus'],
[[38.500, 55.000], 'verse'],
[[55.000, 72.600], 'chorus'],
[[72.600, 89.000], 'bridge'],
[[89.000, 105.000], 'chorus'],
[[105.000, 115.500], 'outro']
]
}
]
</pre>
Ensure that <tt>offset_time</tt> of one segment is the <tt>onset_time</tt> of the next segment, and segments cover the entire duration of the piece analyzed. The first segment must start at <tt>0.0</tt>.

=== Evaluation Procedures ===

Evaluation will focus on both the accuracy of the detected segment boundaries and the correctness of the assigned functional labels. The primary metrics are:

# '''Frame-Level Accuracy (ACC)''':
# Both the system output and the ground truth will be converted into time-series of labels at a fine temporal resolution (e.g., 10ms or 100ms frames). Accuracy is calculated as the proportion of frames that are correctly labeled by the system compared to the ground truth across the entire dataset. This metric evaluates the overall correctness of segment labels and their temporal extents.

# '''Boundary Retrieval Hit Rate F-Measures (HR.5F and HR3F)''':
# This metric assesses the system's ability to correctly identify segment boundaries.
# * A predicted boundary is considered a '''hit''' if it falls within a certain tolerance window of a ground truth boundary.
# * Two tolerance windows will be used:
# ** 0.5 seconds: For finer precision.
# ** 3.0 seconds: For coarser, more perceptually relevant boundaries.
# * Based on these hits, '''Precision (P)''', '''Recall (R)''', and '''F-measure (F1-score)''' will be calculated for boundary detection at both tolerance levels.
# <math>P = \frac{\text{Number of correctly retrieved boundaries}}{\text{Total number of retrieved boundaries}}</math>
# <math>R = \frac{\text{Number of correctly retrieved boundaries}}{\text{Total number of ground truth boundaries}}</math>
# <math>F = \frac{2 \times P \times R}{P + R}</math>
# * The reported metrics will be '''HR.5F''' (F-measure with 0.5s tolerance) and '''HR3F''' (F-measure with 3s tolerance).

==== Baseline ====
The performance of the method described in '''Kim, T., & Nam, J. (2023). All-in-one metrical and functional structure analysis with neighborhood attentions on demixed audio.''' will serve as a baseline for this task. Participants are encouraged to develop systems that surpass this baseline.

=== Relevant Development Collections ===
While the MIREX datasets will be used for evaluation, participants may find the following publicly available annotated corpora useful for development. Please note that the annotations in these corpora will also need to be mapped to the 7-class functional labeling scheme if used for training models for this task.

* Jouni Paulus's [http://www.cs.tut.fi/sgn/arg/paulus/structure.html structure analysis page] links to a corpus of 177 Beatles songs ([http://www.cs.tut.fi/sgn/arg/paulus/beatles_sections_TUT.zip zip file]). The TUTstructure07 dataset, containing 557 songs, is also listed [http://www.cs.tut.fi/sgn/arg/paulus/TUTstructure07_files.html here].
* Ewald Peiszer's [http://www.ifs.tuwien.ac.at/mir/audiosegmentation.html thesis page] links to a portion of his corpus: 43 non-Beatles pop songs (including 10 J-pop songs) ([http://www.ifs.tuwien.ac.at/mir/audiosegmentation/dl/ep_groundtruth_excl_Paulus.zip zip file]).

These public corpora offer over 200 songs that can be adapted for development purposes.

=== Time and Hardware Limits ===
Due to the nature of the CodeBench platform and the potentially high number of participants, limits on the runtime and computational resources for submissions may be imposed. Specific details regarding these limits will be provided closer to the submission deadline. A general guideline is that analysis should be computationally feasible. For reference, a hard limit of '''24 hours''' for total analysis time over the evaluation dataset was imposed in previous iterations, and a similar constraint might apply.

2026:New Task Proposals

Junyan — Mon, 29 Jun 2026 19:27:53 GMT

‎Continued Tasks

2026:Audio Downbeat Estimation

Junyan — Mon, 29 Jun 2026 19:27:22 GMT

Created page with "== Description == Standardized Evaluation Suites: [https://github.com/ismir-mirex/mirex-evaluation ismir-mirex/mirex-evaluation]. The aim of the automatic downbeat estimati..."

New page

== Description ==

Standardized Evaluation Suites: [https://github.com/ismir-mirex/mirex-evaluation ismir-mirex/mirex-evaluation].

The aim of the automatic downbeat estimation task is to identify the locations of downbeats in a collection of sound files. While this is similar to the Audio Beat Tracking task, here the aim is to find the first beat of each bar (measure) rather than all beat times. Algorithms are '''not''' required to estimate beat times or time-signature in addition to downbeats.

Submitted algorithms will be evaluated in terms of their accuracy in finding downbeat locations (only) as annotated by musical experts across several diverse datasets.

== Data ==

=== Test Data ===

Participants are not allowed to use any split of the following datasets for training, validation, model selection, parameter tuning, or any other development purpose:

'''GTZAN''': The GTZAN dataset contains 1,000 30-second excerpts covering ten music genres. It provides a broad genre-balanced collection for evaluating beat and downbeat estimation systems beyond ballroom dance music.

'''Hidden test set''': We may include an additional hidden test set, mainly consisting of pop-style music. Details about this hidden set will be announced later if it is included in the evaluation.

=== Other Collections ===

Other collections may be used, but participants must clearly state their dataset usage and, when applicable, the data collection process in the extended abstract.

'''Ballroom'''
The Ballroom dataset contains eight different dance styles (Cha Cha, Jive, Quickstep, Rumba, Samba, Tango, Viennese Waltz and Waltz). It consists of 697 excerpts of 30s in duration. We removed duplicates from the dataset as suggested by [http://media.aau.dk/null_space_pursuits/2014/01/ballroom-dataset.html Bob Sturm], which finally yields '''685''' excerpts.
We are using the audio files available [http://mtg.upf.edu/ismir2004/contest/tempoContest/node5.html here] (see Gouyon et al. (2006)) and ground truth annotations available [https://github.com/CPJKU/BallroomAnnotations here] (see Krebs et al. (2013)).

'''HarmonixSet'''
912 pop songs with beat, downbeat and structure annotations (https://github.com/urinieto/harmonixset/tree/main/dataset/beats_and_downbeats).

'''Turkish Data'''
The Turkish corpus is an extended version of the annotated data used in Srinivasamurthy et al. (2014). It includes '''82''' excerpts of one
minute length each, and each piece belongs to one of three
rhythm classes that are referred to as usul in Turkish Art
music. 32 pieces are in the 9/8-usul Aksak, 20 pieces
in the 10/8-usul Curcuna, 30 samples in the 8/8-usul
Düyek.

'''Cretan Data'''
The corpus of Cretan music consists of '''42''' full length pieces of Cretan leaping dances. While there are several dances that differ in terms of their steps, the differences in
the sound are most noticeable in the melodic content, and all pieces can be considered to belong to one rhythmic style. All these dances are usually notated using a 2/4 time signature,
and the accompanying rhythmical patterns are usually played on a Cretan lute. While a variety of rhythmic patterns exist, they do not relate to a specific dance and can be
assumed to occur in all of the 42 songs in this corpus.

'''Carnatic Data'''
The Carnatic music dataset is a subset of the CompMusic [http://compmusic.upf.edu/carnatic-rhythm-dataset Carnatic Music Rhythm Dataset]. It includes '''118''' two minute long excerpts spanning four most commonly used tālas (the rhythmic framework of Carnatic music, consisting of time cycles) of Carnatic music. There are 30 examples in each of ādi tāla (8 beats/cycle), rūpaka tāla (3 beats/cycle) and miśra chāpu tāla (7 beats/cycle), and 28 examples in khaṇḍa chāpu tāla (5 beats/cycle). The beats of the tāla in miśra chāpu and khaṇḍa chāpu are non-uniform, but for consistency with other datasets, a uniform beat pulse was obtained by interpolating the non-uniformly spaced beat locations. The recordings consist of both vocal and instrumental music recordings representative of the present day performance practice. All recordings contain percussion accompaniment, mainly the Mridangam.

'''HJDB'''
The HJDB dataset contains '''235''' excerpts of Hardcore, Jungle and Drum and Bass music between 30s and 2 minutes in length. All excerpts are in 4/4 and have a constant tempo.
For further information see Hockman et al (2012).

=== Audio Formats ===

The data are monophonic sound files

* CD-quality (PCM, 16-bit, 44100 Hz) for all except Ballroom (originally lower quality, but resampled to 44100 Hz)
* single channel (mono)

== Submission Format ==
Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.

=== Input Data ===
Participating algorithms will have to read audio in the following format:

* Sample rate: 44.1 KHz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

=== Output Data ===

The downbeat estimation algorithms will return downbeat times in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.

=== Output File Format (Audio Downbeat Estimation) ===

The downbeat output file format is an ASCII text format. Each downbeat time is specified, in seconds, on its own line. Specifically,

<downbeat time (in seconds)>\n

where \n denotes the end of line. The < and > characters are not included. An example output file would look something like:

0.243
1.486
2.729

=== Algorithm Calling Format ===

The submitted algorithm must take as arguments a SINGLE .wav file to perform the downbeat estimation as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:

foobar %input %output
foobar -i %input -o %output

Moreover, if your submission takes additional parameters, such as a detection threshold, foobar could be called like:

foobar .1 %input %output
foobar -param1 .1 -i %input -o %output

If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example:

foobar('%input','%output')
foobar(.1,'%input','%output')

=== README File ===

A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.

For instance, to test the program foobar with different values for parameters param1, the README file would look like:

foobar -param1 .1 -i %input -o %output
foobar -param1 .15 -i %input -o %output
foobar -param1 .2 -i %input -o %output
foobar -param1 .25 -i %input -o %output
foobar -param1 .3 -i %input -o %output
...

For a submission using MATLAB, the README file could look like:

matlab -r "foobar(.1,'%input','%output');quit;"
matlab -r "foobar(.15,'%input','%output');quit;"
matlab -r "foobar(.2,'%input','%output');quit;"
matlab -r "foobar(.25,'%input','%output');quit;"
matlab -r "foobar(.3,'%input','%output');quit;"
...

The different command lines to evaluate the performance of each parameter set over the whole database will be generated automatically from each line in the README file containing both '%input' and '%output' strings.

== Evaluation Procedure ==

For the evaluation procedure we will use
*'''F-measure''' - the standard calculation as used in onset and beat tracking evaluation with a +/-70ms window, see Dixon (2007).

Given the high diversity of musical styles included in the task, results will be reported per each individual dataset.

== Time and hardware limits ==
Due to the potentially high number of participants in this and other audio tasks, limits on the runtime of submissions may be imposed. Specific details will be announced with the evaluation instructions.

= Bibliography =

S. Dixon, F. Gouyon and G. Widmer, [http://ismir2004.ismir.net/proceedings/p093-page-509-paper165.pdf Towards Characterisation of Music via Rhythmic Patterns], In Proceedings of the 5th International Conference on Music Information Retrieval (ISMIR 2004), pp 509-516.

S. Dixon, [http://www.eecs.qmul.ac.uk/~simond/pub/2007/jnmr07.pdf Evaluation of audio beat tracking system BeatRoot], Journal of New Music Research, vol. 36, no. 1, pp. 39-51, 2007.

J. A. Hockman, M. E. P. Davies, I. Fujinaga.[http://ismir2012.ismir.net/event/papers/169-ismir-2012.pdf ONE IN THE JUNGLE: Downbeat Detection in Hardcore, Jungle, and Drum and Bass], In Proceedings of 13th International Society for Music Information Retrieval Conference (ISMIR), Porto, Portugal pp. 169-174, 2012.

F. Krebs, S. Boeck, and G. Widmer, [http://www.cp.jku.at/research/papers/Krebs_etal_ISMIR_2013.pdf Rhythmic Pattern Modeling for Beat- and Downbeat Tracking in Musical Audio], In Proceedings of 14th International Society for Music Information Retrieval Conference (ISMIR), Curitiba, Brazil, 2013.

M. Mauch, C. Cannam, M. E. P. Davies, S. Dixon, C. Harte, S. Kolozali and D. Tidhar, [http://ismir2009.ismir.net/proceedings/LBD-18.pdf OMRAS2 Metadata Project 2009], Late-breaking session at the 10th International Conference on Music Information Retrieval, 2009.

A. Srinivasamurthy, A. Holzapfel, and Xavier Serra, [http://www.tandfonline.com/doi/full/10.1080/09298215.2013.879902 In Search of Automatic Rhythm Analysis Methods for Turkish and Indian Art Music], Journal of New Music Research, vol. 43, no. 1, pp. 94-114, 2014.

F. Gouyon, A. Klapuri, S. Dixon, M. Alonso, G. Tzanetakis, C. Uhle, and P. Cano. An experimental comparison of audio tempo induction algorithms. IEEE Transactions on Audio, Speech and Language Processing 14(5), pp.1832-1844, 2006.

2026:Audio Key Detection

Junyan — Mon, 29 Jun 2026 19:20:01 GMT

Created page with "==Description== Standardized Evaluation Suites: [https://github.com/ismir-mirex/mirex-evaluation ismir-mirex/mirex-evaluation]. Audio Key Detection aims to identify the mus..."

New page

==Description==

Standardized Evaluation Suites: [https://github.com/ismir-mirex/mirex-evaluation ismir-mirex/mirex-evaluation].

Audio Key Detection aims to identify the musical key (e.g., C major, A minor) of an audio recording. This involves determining both the tonic (root pitch) and the mode (major or minor) from the audio signal.

== Data ==

=== Test Data ===

Participants are not allowed to use any split of the following datasets for training, validation, model selection, parameter tuning, or any other development purpose:

'''GiantSteps Key''': The GiantSteps Key Dataset comprises 604 two-minute excerpts of electronic dance music tracks, primarily sourced from Beatport. Each excerpt is annotated with one of 24 musical keys—12 major and 12 minor—providing a standardized benchmark for evaluating key classification performance.

=== Other Collections ===

Other collections may be used, but participants must clearly state their dataset usage and, when applicable, the data collection process in the extended abstract.

==== GiantSteps MTG Keys ====

1,077 additional songs that extend the GiantSteps Key dataset. NOTICE: You may use GiantSteps MTG Keys for training, but should exclude the songs from the original GiantSteps Key collection. Link: https://github.com/GiantSteps/giantsteps-mtg-key-dataset/tree/master/annotations/key

==== Isophonics Dataset ====
The Isophonics Dataset includes 225 songs from artists such as The Beatles, Queen, and Zweieck. Each track is annotated with key, chord, beat, and structural segmentation information, offering a comprehensive resource for evaluating key detection algorithms across a diverse range of popular music.

== Evaluation Procedures ==
The error analysis will center on comparing the key identified by the algorithm to the actual key of the piece. The key of the piece is the one defined by the composer in the title of the piece. We will then determine how "close" each identified key is to the corresponding correct key. Keys will be considered as "close" if they have one of the following relationships: distance of perfect fifth, relative major and minor, and parallel major and minor. A correct key assignment will be given a full point, and incorrect assignments will be allocated fractions of a point according to the following table:

{|border="1"
|'''Relation to Correct Key''' ||'''Points'''
|-
|Same||1.0
|-
|Perfect fifth||0.5
|-
|Relative major/minor||0.3
|-
|Parallel major/minor||0.2
|-
|Other||0.0
|}

The points are counted over all files and averaged. The number of correctly identified keys as well as the distribution of the errors is also reported.

== Submission Format ==

Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.

=== Input Data ===
Participating algorithms will have to read audio in the following format:

* Sample rate: 44.1 KHz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

=== Output Data ===

The audio key detection algorithms will return the estimated key in an individual ASCII text file for each input .wav audio file. The specification of this output file is immediately below.

=== Output File Format (Audio Key Detection) ===

The Audio Key Detection output file format is a single-line tab-delimited ASCII text format. The tonic is reported, followed by a TAB and the mode. For sharps, the "#" symbol is used (e.g. A# for A sharp), for flats, a lowercase "b" is used, e.g. (Bb for B flat). Therefore, the output file should be of the form:

<tonic {A, A#, Bb, ...}>\t<mode {major, minor}>\n

where \t denotes a tab, \n denotes the end of line. The < and > characters are not included. An example output file would look something like:

C major

or

G# minor

=== Algorithm Calling Format ===

The submitted algorithm must take as arguments a SINGLE .wav file to perform the melody extraction on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:

foobar %input %output
foobar -i %input -o %output

Moreover, if your submission takes additional parameters, foobar could be called like:

foobar .1 %input %output
foobar -param1 .1 -i %input -o %output

If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example:

foobar('%input','%output')
foobar(.1,'%input','%output')

=== Packaging submissions ===

All submissions should include a README file including the following the information:

* Command line calling format for all executables including examples
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Approximately how much scratch disk space will the submission need to store any feature/cache files?
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.
* Any special notice regarding to running your algorithm

Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.

==== README File ====

A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.

For instance, to test the program foobar with a specific value for parameter param1, the README file would look like:

foobar -param1 .1 -i %input -o %output

2026:Audio Beat Tracking

Junyan — Mon, 29 Jun 2026 19:12:01 GMT

Created page with "== Description == Standardized Evaluation Suites: [https://github.com/ismir-mirex/mirex-evaluation ismir-mirex/mirex-evaluation]. The aim of the automatic beat tracking task..."

New page

== Description ==

Standardized Evaluation Suites: [https://github.com/ismir-mirex/mirex-evaluation ismir-mirex/mirex-evaluation].

The aim of the automatic beat tracking task is to track each beat locations in a collection of sound files. Unlike the Audio Tempo Extraction task, which aim is to detect tempi for each file, the beat tracking task aims at detecting all beat locations in recordings. The algorithms will be evaluated in terms of their accuracy in predicting beat locations annotated by a group of listeners.

== Data ==

=== Test Data ===

Participants are not allowed to use any split of the following datasets for training, validation, model selection, parameter tuning, or any other development purpose:

'''SMC Dataset''': The SMC dataset contains challenging excerpts with beat annotations, designed to test beat tracking systems on material with ambiguous, expressive, or otherwise difficult rhythmic content.

'''GTZAN dataset''': The GTZAN dataset contains 1,000 30-second excerpts covering ten music genres. It provides a broad genre-balanced collection for evaluating beat tracking systems beyond rhythmically regular dance music.

=== Other Collections ===

Other collections may be used, but participants must clearly state their dataset usage and, when applicable, the data collection process in the extended abstract.

==== Ballroom Dataset ====
The Ballroom Dataset contains 685 short excerpts (30 seconds each) from various ballroom dance genres, such as Waltz, Tango, Rumba, and Jive. Each excerpt includes annotated beat and meter information, making the dataset ideal for evaluating beat tracking and tempo estimation methods in rhythmically regular dance music.

==== Hainsworth Dataset ====
The Hainsworth Dataset offers 222 music excerpts covering a wide range of genres, including classical, jazz, and popular music. Annotations include both beat and downbeat locations, providing a varied testing ground for assessing the performance of beat tracking systems across different musical styles.

== Submission Format ==
Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.

=== Input Data ===
Participating algorithms will have to read audio in the following format:

* Sample rate: 44.1 KHz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

=== Output Data ===

The beat tracking algorithms will return beat-times in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.

=== Output File Format (Audio Beat tracking) ===

The Beat Tracking output file format is an ASCII text format. Each beat time is specified, in seconds, on its own line. Specifically,

<beat time(in seconds)>\n

where \n denotes the end of line. The < and > characters are not included. An example output file would look something like:

0.243
0.486
0.729

=== Algorithm Calling Format ===

The submitted algorithm must take as arguments a SINGLE .wav file to perform the onset detection on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:

foobar %input %output
foobar -i %input -o %output

Moreover, if your submission takes additional parameters, such as a detection threshold, foobar could be called like:

foobar .1 %input %output
foobar -param1 .1 -i %input -o %output

If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example:

foobar('%input','%output')
foobar(.1,'%input','%output')

=== README File ===

A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.

For instance, to test the program foobar with different values for parameters param1, the README file would look like:

foobar -param1 .1 -i %input -o %output
foobar -param1 .15 -i %input -o %output
foobar -param1 .2 -i %input -o %output
foobar -param1 .25 -i %input -o %output
foobar -param1 .3 -i %input -o %output
...

For a submission using MATLAB, the README file could look like:

matlab -r "foobar(.1,'%input','%output');quit;"
matlab -r "foobar(.15,'%input','%output');quit;"
matlab -r "foobar(.2,'%input','%output');quit;"
matlab -r "foobar(.25,'%input','%output');quit;"
matlab -r "foobar(.3,'%input','%output');quit;"
...

The different command lines to evaluate the performance of each parameter set over the whole database will be generated automatically from each line in the README file containing both '%input' and '%output' strings.

== Evaluation Procedures ==

The evaluation methods are taken from the beat evaluation toolbox and
are described in the following technical report:

M. E. P. Davies, N. Degara and M. D. Plumbley. "Evaluation methods for musical audio beat tracking algorithms". [http://www.elec.qmul.ac.uk/people/markp/2009/DaviesDegaraPlumbley09-evaluation-tr.pdf ''Technical Report C4DM-TR-09-06'']. This link now works! :)

For further details on the specifics of the methods please refer to the
paper. However, here is a brief summary with appropriate references:

*'''F-measure''' - the standard calculation as used in onset evaluation but
with a 70ms window.

S. Dixon, "Onset detection revisited," in ''Proceedings of 9th
International Conference on Digital Audio Effects (DAFx)'', Montreal,
Canada, pp. 133-137, 2006.

S. Dixon, "Evaluation of audio beat tracking system beatroot," ''Journal
of New Music Research'', vol. 36, no. 1, pp. 39-51, 2007.

*'''Cemgil''' - beat accuracy is calculated using a Gaussian error function
with 40ms standard deviation.

A. T. Cemgil, B. Kappen, P. Desain, and H. Honing, "On tempo tracking:
Tempogram representation and Kalman filtering," ''Journal Of New Music
Research'', vol. 28, no. 4, pp. 259-273, 2001

*'''Goto''' - binary decision of correct or incorrect tracking based on
statistical properties of a beat error sequence.

M. Goto and Y. Muraoka, "Issues in evaluating beat tracking systems," in
''Working Notes of the IJCAI-97 Workshop on Issues in AI and Music -
Evaluation and Assessment'', 1997, pp. 9-16.

*'''PScore''' - McKinney's impulse train cross-correlation method as used in
2006.

M. F. McKinney, D. Moelants, M. E. P. Davies, and A. Klapuri,
"Evaluation of audio beat tracking and music tempo extraction
algorithms," ''Journal of New Music Research'', vol. 36, no. 1, pp. 1-16,
2007.

*'''CMLc''', '''CMLt''', '''AMLc''', '''AMLt''' - continuity-based evaluation methods based on
the longest continuously correctly tracked section.

S. Hainsworth, "Techniques for the automated analysis of musical audio,"
Ph.D. dissertation, Department of Engineering, Cambridge University,
2004.

A. P. Klapuri, A. Eronen, and J. Astola, "Analysis of the meter of
acoustic musical signals," IEEE Transactions on Audio, Speech and
Language Processing, vol. 14, no. 1, pp. 342-355, 2006.

*'''D''', '''Dg''' - information based criteria based on analysis of a beat error
histogram (note the results are measured in 'bits' and not percentages),
see the technical report for a description.

2026:Audio Chord Estimation

Junyan — Mon, 29 Jun 2026 16:27:15 GMT

‎Other Collections

2026:Audio Chord Estimation

Junyan — Mon, 29 Jun 2026 16:24:58 GMT

Created page with "= Description = Standardized Evaluation Suites: [https://github.com/ismir-mirex/mirex-evaluation ismir-mirex/mirex-evaluation]. This task requires participants to extract o..."

New page

= Description =

Standardized Evaluation Suites: [https://github.com/ismir-mirex/mirex-evaluation ismir-mirex/mirex-evaluation].

This task requires participants to extract or transcribe a sequence of chords from an audio music recording. For many applications in music information retrieval, extracting the harmonic structure of an audio track is very desirable, for example for segmenting pieces into characteristic segments, for finding similar pieces, or for semantic analysis of music. The extraction of the harmonic structure requires the estimation of a sequence of chords that is as precise as possible. This includes the full characterisation of chords – root, quality, and bass note – as well as their chronological order, including specific onset times and durations. Audio chord estimation has a long history in MIREX, and readers interested in this history, especially with respect to evaluation methodology, should review the work of Christopher Harte (2010), Pauwels and Peeters (2013), and the [https://www.music-ir.org/mirex/wiki/The_Utrecht_Agreement_on_Chord_Evaluation “Utrecht Agreement”] on evaluation metrics. For python evaluation code, please refer to [https://craffel.github.io/mir_eval/#module-mir_eval.chord “mir_eval”].

= Data =

== Held-out Datasets ==

Participants are not allowed to use any split of the following datasets for training, validation, model selection, parameter tuning, or any other development purpose:

* Billboard 2013

== Other Collections ==

Other collections may be used, but participants must clearly state their dataset usage and, when applicable, the data collection process in the extended abstract.

The following datasets are commonly used for development or background comparison in audio chord estimation. They are listed here for context; participants remain responsible for checking that any collections they use do not overlap with the held-out datasets.

We might include additional hidden test sets for the task.

; Isophonics
: The collected Beatles, Queen, and Zweieck datasets from the Centre for Digital Music at Queen Mary, University of London (http://www.isophonics.net/), as used for Audio Chord Estimation in MIREX for many years. Available from http://www.isophonics.net/. See also Matthias Mauch’s dissertation (2010) and Harte et al.’s introductory paper (2005).
; Billboard 2012
: An abridged version of the ''Billboard'' dataset from McGill University, including a representative sample of American popular music from the 1950s through the 1990s. Available from http://billboard.music.mcgill.ca. See also Ashley Burgoyne’s dissertation (2012) and Burgoyne et al.’s introductory paper (2011). Parsing tools for the data are available from http://hackage.haskell.org/package/billboard-parser/ and documented by De Haas and Burgoyne (2012).

== Training and Testing ==

The ground-truth files contain one line per unique chord, in the form <code>{start_time end_time chord}</code>, e.g.,
<pre>...
41.2631021 44.2456460 B:maj
44.2456460 45.7201230 E:maj
45.7201230 47.2061900 E:7/3
47.2061900 48.6922670 A:maj
48.6922670 50.1551240 A:min/b3
...</pre>
Start and end times are in seconds from the start of the file. Chord labels follow the syntax proposed by C. Harte et al. (2005). Please note that the syntax has changed slightly since since it was originally described; in particular, the root is no longer implied as a voiced element of a chord so a C major chord (notes C, E and G) should be written C:(1,3,5) instead of just C:(3,5) if using the interval list representation. As before, the labels C and C:maj are equivalent to C:(1,3,5).

= Evaluation =

To evaluate the quality of an automatic transcription, a transcription is compared to ground truth created by one or more human annotators. MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth:

<math>\textrm{CSR} = \frac{\text{total duration of segments where annotation equals estimation}} {\text{total duration of annotated segments}}</math>

In previous years, MIREX has used an approximate CSR calculated by sampling both the ground-truth and the automatic annotations every 10 ms and dividing the number of correctly annotated samples by the total number of samples. Following Christopher Harte (2010, §8.1.2), however, we can view the ground-truth and estimated annotations as continuous segmentations of the audio and calculate the CSR by considering the cumulative length of the correctly overlapping segments. This way of calculating the CSR is more precise, as the precision of the frame-based method is limited by the frame length, and computationally more efficient, as it reduces the number of segment comparisons. Because pieces of music come in a wide variety of lengths, we will weight the CSR by the length of the song when computing an average for a given corpus. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

== Chord Vocabularies ==

We propose a set of single chord evaluation measures for MIREX that extends the previous iterations of MIREX and combines it with evaluation measures proposed in the literature, providing a more complete assessment of the transcription quality. Following Pauwels and Peeters (2013), we suggest using the CSR with five different chord vocabulary mappings.

In each of these calculations, the full chord descriptions of either the estimated or the ground-truth transcriptions, which might contain complex chord annotations, would be mapped to the following classes:

# Chord root note only;
# Major and minor: {<code>N, maj, min</code>};
# Seventh chords: {<code>N, maj, min, maj7, min7, 7</code>};
# Major and minor with inversions: {<code>N, maj, min, maj/3, min/b3, maj/5, min/5</code>}; or
# Seventh chords with inversions: {<code>N, maj, min, maj7, min7, 7, maj/3, min/b3, maj7/3, min7/b3, 7/3, maj/5, min/5, maj7/5, min7/5, 7/5, maj7/7, min7/b7, 7/b7</code>}.

With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels. A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary. For instance, in the major and minor case, <code>G:7(#9)</code> is mapped to <code>G:maj</code> because the interval set of <code>G:maj</code>, {<code>1,3,5</code>}, is a subset of the interval set of the <code>G:7(#9)</code>, {<code>1,3,5,b7,#9</code>}. In the seventh-chord case, <code>G:7(#9)</code> is mapped to <code>G:7</code> instead because the interval set of <code>G:7</code> {<code>1, 3, 5, b7</code>} is also a subset of <code>G:7(#9)</code> but is larger than <code>G:maj</code>. If a chord cannot be represented by a certain class, e.g., mapping a <code>D:aug</code> or <code>F:sus4(9)</code> to {<code>maj, min</code>}, the chord is excluded from the evaluation if it occurs in the ground-truth, and it is considered a mismatch if it occurs in an estimated annotation.

{|
|+ Most frequent chord qualities in the McGill ''Billboard'' corpus.
! Quality
! Freq. (%)
! Cum. Freq (%)
|-
|maj
|52
|52
|-
|min
|13
|65
|-
|7
|10
|75
|-
|min7
|8
|83
|-
|maj7
|3
|86
|-
|5
|2
|88
|-
|1
|2
|90
|-
|maj(9)
|1
|91
|-
|maj6
|1
|92
|-
|sus4
|1
|93
|-
|sus7
|1
|94
|-
|sus9
|1
|94
|-
|7(#9)
|1
|95
|-
|min9
|1
|96
|}

Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus (see table above), which is a balanced sample of American popular music from the 1950s through the 1990s (J.A. Burgoyne, Wild, and Fujinaga 2011). Pure major and minor chords alone account for 65 percent of all chords encountered, whereas augmented and diminished triads account for 0.2 percent or less of the corpus each. Our arguments for our particular seventh-chord vocabulary as opposed to the set of all tetrads follows similar reasoning; our proposed vocabulary accounts for 86 percent of all chords, whereas no other standard type of seventh chord accounts for more than 0.2 percent of the corpus. In future years, the table suggests that we might consider introducing vocabularies including power chords, and possibly suspended chords or added sixths and ninths as well.

== Chord Segmentation ==

Besides CSR, the chord transcription literature includes several other metrics for evaluating chord transcriptions, which mainly focus on the segmentation of the automatic transcription. We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences ((S. A. Abdallah et al. 2005); (Mauch 2010, §2.3.3)). Depending on the order of application, the directional Hamming distance yields a measure of over- or under segmentation. Both directions can be combined to yield an overall quality metric (Christopher Harte 2010, §8.3.2):

<math>Q = 1 - \frac{\text{maximum of directional Hamming distances in either direction}} {\text{total duration of song}}</math>

= Submission Format =

== Audio Format ==

Audio tracks in the training directory will be encoded as 44.1 kHz 16bit mono WAV files.

== I/O Format ==

The algorithms should output text files with a similar format to that used in the ground truth transcriptions. That is to say, they should be flat text files with chord segment labels and times arranged thus:

<pre>start_time end_time chord_label</pre>
with elements separated by white spaces, times given in seconds, chord labels corresponding to the syntax described by C. Harte et al. (2005), and one chord segment per line. As in all benchmarks after 2008, end times are a mandatory component of the output. For the evaluation process we will assume enharmonic equivalence for chord roots. We will no longer accept participants who would only like to be evaluated on major/minor chords and want to use the number format.

== Command line calling format ==

Submissions using machine learning models must also submit their trained models. Training on the evaluation server is no longer supported starting from this year. We will execute the following commands for testing:

<pre>
your_program prepare
your_program do_chord_identification "/path/to/input1.wav" "/path/to/output1.wav.txt"
your_program do_chord_identification "/path/to/input2.wav" "/path/to/output2.wav.txt"
...
</pre>

In the results directory, there should be one file for each testfile with same name as the test file + <code>.txt</code>. Programs can use the folder <code>/app/temp</code> if they need to keep temporary cache files or to download pretrained models. Standard output and standard error will be logged.

No internet access is allowed during the inference stage (<code>do_chord_identification</code>). Please contact us if your model requires internet access (e.g., model API call) during inference.

== Packaging submissions ==

* You can directly upload your submission (up to 5GB) to the MIREX submission site. If you need more storage you may contact the evaluation organizers.

=== Non-docker submission ===

* We recommend the participants submit a docker image to ensure that the evaluation team can easily run it.
* Non-docker submissions should contain a README file to include the setup procedure.

=== Docker Submission ===

* A docker submission will be deployed and evaluated automatically with <code>docker run</code>

Here are the docker commands that will be used to evaluate all systems:

<pre>
docker run -v temp_folder:/app/temp your_image_name prepare
docker run -v temp_folder:/app/temp -v dataset_folder:/app/data do_chord_identification "/app/data/input1.wav" "/app/data/output1.wav.txt"
docker run -v temp_folder:/app/temp -v dataset_folder:/app/data do_chord_identification "/app/data/input2.wav" "/app/data/output2.wav.txt"
...
</pre>

==== Example project ====

A sample project containing code and <code>Dockerfile</code> for a simple audio chord estimation baseline can be found in [https://github.com/futuremirex/audio_chord_estimation_sample_project https://github.com/futuremirex/audio_chord_estimation_sample_project].

Notice that this is not a compiled docker image - you need to build the image by your own before submitting it.

= Time and Hardware limits =

A Linux server with one Nvidia GeForce RTX 3090 is used for evaluation. CPU, OS, and memory specifications will be announced later.

Time limit: within 5 times the total duration of the test set.

= Bibliography =

Abdallah, Samer A., Katy Noland, Mark B. Sandler, Michael Casey, and Christophe Rhodes. 2005. “Theory and Evaluation of a Bayesian Music Structure Extractor.” In ''Proceedings of the International Society for Music Information Retrieval Conference'', 420–425.

Burgoyne, J. A., J. Wild, and I. Fujinaga. 2011. “An expert ground truth set for audio chord recognition and music analysis.” In ''Proceedings of the 12th International Society for Music Information Retrieval Conference (ISMIR)'', 633–638.

Burgoyne, John Ashley. 2012. “Stochastic Processes and Database-Driven Musicology.” Ph.D. diss. Montréal, Québec, Canada: McGill University.

Haas, W. B. de, and John~Ashley Burgoyne. 2012. ''Parsing the Billboard Chord Transcriptions''. Technical report UU-CS- 2012-018, Department of Information and Computing Sciences, Utrecht University.

Harte, C., M. Sandler, S. Abdallah, and E. Gómez. 2005. “Symbolic representation of musical chords: A proposed syntax for text annotations.” In ''Proceedings of the 6th International Society for Music Information Retrieval Conference (ISMIR)'', 66–71.

Harte, Christopher. 2010. “Towards automatic extraction of harmony information from music signals.” Ph.D. diss. Queen Mary, University of London.

Mauch, Matthias. 2010. “Automatic Chord Transcription from Audio Using Computational Models of Musical Context.” Ph.D. diss. Queen Mary University of London.

Pauwels, Johan, and Geoffroy Peeters. 2013. “Evaluating automatically estimated chord sequences.” In ''Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)''. Vancouver, British Columbia, Canada.

2026:Audio-to-Score Transcription

Alexandre DHooge — Mon, 29 Jun 2026 08:23:11 GMT

‎Undisclosed Evaluation Set: add more details

@@ Line 25: / Line 25: @@
 * [[2026:AI-Generated Music Detection]] <TC: [mailto:ldzhangyx@outlook.com Yixiao Zhang] & [mailto:you.zhang@rochester.edu Neil Zhang]>
 * [[2026:Music Performance Difficulty Prediction]] <TC: [mailto:huan.zhang@qmul.ac.uk Huan Zhang] & [mailto:pedro@songscription.ai Pedro Ramoneda]>
 ==Call for Challenges==

@@ Line 41: / Line 41: @@
 Possible training sources include:
-* '''SONICS''': approximately 96k tracks generated by Suno v3.5 and Udio.
+* '''[https://huggingface.co/datasets/awsaf49/sonics SONICS]''': approximately 96k tracks generated by Suno v3.5 and Udio.
-* '''Muse''': approximately 116k tracks generated by Suno v5.
+* '''[https://huggingface.co/datasets/bolshyC/Muse/viewer/default/ Muse]''': approximately 116k tracks generated by Suno v5.
-The final training dataset will be announced before the submission phase.
+Additional candidate training sources include:
-Participants may also use their own public, private, synthetic, or self-constructed training data, provided that no part of the hidden evaluation set is used directly or indirectly for training, validation, model selection, prompt tuning, or threshold tuning.
+* '''[https://www.kaggle.com/datasets/miguelcivit/sunocaps?select=SunoCaps_dataset.csv SunoCaps]''': 256 tracks generated with Suno v3 from MusicCaps captions/aspect descriptions.
 = Input and Output Format =

← Older revision		Revision as of 10:12, 1 July 2026
Line 79:		Line 79:

	Responses were collected from 20 participants with diverse music backgrounds. 14 participants completed all 8 pages with an average completion time of 32 minutes.		Responses were collected from 20 participants with diverse music backgrounds. 14 participants completed all 8 pages with an average completion time of 32 minutes.
		+
		+	= Listening Samples =
		+	Play-along piano-roll continuations (visual + audio) for every prompt and system are available on the demo page:
		+
		+	: '''[https://futuremirex.com/demos/symbolic-music-generation/ ▶ Open the interactive demo]'''

@@ Line 52: / Line 52: @@
 Suggested datasets for training and validation:
-* '''[https://zenodo.org/records/12783403 PSyllabus]''' (Ramoneda et al., 2025): 7,901 audio recordings of classical piano pieces with 11-level pedagogical grades, together with mappings to multiple international grading systems. We specifically encourage participants to leverage the multiple ranking annotations available in PSyllabus (13 grading/ranking systems in total), not only the default labels, in order to study and improve generalization across heterogeneous pedagogical traditions and difficulty scales.
+* '''[https://zenodo.org/records/14794592 Psyllabus]''' (Ramoneda et al., 2025): 7,901 audio recordings of classical piano pieces with 11-level pedagogical grades, together with mappings to multiple international grading systems. We specifically encourage participants to leverage the multiple ranking annotations available in PSyllabus (13 grading/ranking systems in total), not only the default labels, in order to study and improve generalization across heterogeneous pedagogical traditions and difficulty scales.
-* '''[https://zenodo.org/records/8037327 CIPI (Can I Play It?)]''' (Ramoneda et al., 2024): 652 MusicXML piano pieces with 9-level Henle annotations, useful for participants interested in transcription-then-classification pipelines or distant supervision approaches.
+* '''[https://zenodo.org/records/8037327 Can I Play It? (CIPI)]''' (Ramoneda et al., 2024): 652 MusicXML piano pieces with 9-level Henle annotations, useful for participants interested in transcription-then-classification pipelines or distant supervision approaches.
 * '''[https://github.com/PRamoneda/Mikrokosmos-difficulty Mikrokosmos-difficulty]''' (Ramoneda et al., 2022) and other publicly available piano difficulty datasets and pedagogical collections.

@@ Line 42: / Line 42: @@
 Dataset-specific labels may be mapped to the official vocabulary when necessary. For example, labels such as "drum kit", "drums", and "drum set" may be mapped to "drums"; labels such as "synth" and "synthesizer" may be mapped to "synthesizer".
-The final label list and any label-mapping rules will be announced before evaluation.
+The official evaluation label set is expected to follow the 20-label OpenMIC-based vocabulary listed above. Any necessary dataset-specific label mapping or label exclusion after the final annotation audit will be documented and applied uniformly to all submitted systems.
 = Datasets =
@@ Line 174: / Line 174: @@
 The official leaderboard will be determined by clip-level macro-F1 on the hidden evaluation set. Results on any public reference dataset will be reported separately.
-For confidence-based outputs, the official evaluation script may use a fixed threshold, a predefined thresholding rule, or threshold-independent metrics such as mAP. The thresholding rule will be specified before evaluation.
+For confidence-based outputs, scores will be converted to binary predictions using a fixed threshold of 0.5. Scores greater than or equal to 0.5 will be treated as positive predictions; scores below 0.5 will be treated as negative predictions.
 = Time and Hardware Limits =
@@ Line 197: / Line 199: @@
 . J. J. Bosch, J. Janer, F. Fuhrmann, and P. Herrera, “A comparison of sound segregation techniques for predominant instrument recognition in musical audio signals,” in ''Proceedings of the 13th International Society for Music Information Retrieval Conference'', Porto, Portugal, 2012, pp. 559–564.
-. R. M. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Cannam, and J. P. Bello, “MedleyDB: A multitrack dataset for annotation-intensive MIR research,” in ''Proceedings of the 15th International Society for Music Information Retrieval Conference'', Taipei, Taiwan, 2014, pp. 155–160.
+. R. M. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Cannam, and P. Bello, “MedleyDB: A multitrack dataset for annotation-intensive MIR research,” in ''Proceedings of the 15th International Society for Music Information Retrieval Conference'', Taipei, Taiwan, 2014, pp. 155–160.
 . D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, “The MTG-Jamendo dataset for automatic music tagging,” in ''Machine Learning for Music Discovery Workshop, International Conference on Machine Learning'', Long Beach, CA, USA, 2019.

@@ Line 98: / Line 98: @@
 ==== Baseline ====
 The performance of the method described in '''Kim, T., & Nam, J. (2023). All-in-one metrical and functional structure analysis with neighborhood attentions on demixed audio.''' will serve as a baseline for this task. Participants are encouraged to develop systems that surpass this baseline.
 === Relevant Development Collections ===

← Older revision		Revision as of 00:54, 1 July 2026
(One intermediate revision by the same user not shown)
Line 4:		Line 4:

	In a long run, we want to make MIREX a platform for researchers to share their latest research results, to compare their systems with others, and to promote the development of the field.		In a long run, we want to make MIREX a platform for researchers to share their latest research results, to compare their systems with others, and to promote the development of the field.
		+
		+	==Task Descriptions==
		+
		+	Traditional MIR tasks
		+	* [[2026:Lyrics Transcription]] <TC: [mailto:dhooge@gbu.edu.cn Alexandre D'Hooge]>
		+	* [[2026:Audio Instrument Recognition]] <TC: [mailto:wenye.ma@mail.mcgill.ca Wenye Ma]>
		+	* Standardized Evaluation Suites <TC: [mailto:jj2731@nyu.edu Junyan Jiang] & [mailto:akira.maezawa@music.yamaha.com Akira Maezawa]>
		+	** [[2026:Audio Chord Estimation]]
		+	** [[2026:Audio Beat Tracking]]
		+	** [[2026:Audio Key Detection]]
		+	** [[2026:Audio Downbeat Estimation]]
		+	** [[2026:Music Structure Analysis]]
		+	** [[2026:Lyrics-to-Audio Alignment]]
		+
		+	Modern MIR Tasks
		+	* [[2026:Symbolic Music Generation]] <TC: [mailto:ziyu.wang@nyu.edu Ziyu Wang], [mailto:Xinyue.Li@mbzuai.ac.ae Xinyue Li] [mailto:jzhao@u.nus.edu Jingwei Zhao]>
		+	* [[2026:Audio-to-Score Transcription]] <TC: [mailto:dhooge@gbu.edu.cn Alexandre D'Hooge]>
		+	* [[2026:Rhythm Game Chart Generation]] <TC: [mailto:ziyun.liu@univ-lille.fr Ziyun Liu] & [mailto:carolina.carusi@kaist.ac.kr Carolina Carusi]>
		+	* [[2026:Music Evaluation via CMI-RewardBench]] <TC: [mailto:yinghao.ma@qmul.ac.uk Yinghao Ma], Yonghyun Kim, Haiwen Xia, Junwon Lee>
		+	* [[2026:AI-Generated Music Detection]] <TC: [mailto:ldzhangyx@outlook.com Yixiao Zhang] & [mailto:you.zhang@rochester.edu Neil Zhang]>
		+	* [[2026:Music Performance Difficulty Prediction]] <TC: [mailto:huan.zhang@qmul.ac.uk Huan Zhang] & [mailto:pedro@songscription.ai Pedro Ramoneda]>

	==Call for Challenges==		==Call for Challenges==

@@ Line 27: / Line 27: @@
 '''HarmonixSet'''
-pop songs with beat, downbeat and structure annotations (https://github.com/urinieto/harmonixset/tree/main/dataset/beats_and_downbeats).
+pop songs with beat, downbeat and structure annotations ([https://github.com/urinieto/harmonixset/tree/main/dataset/beats_and_downbeats Annotation Files], [https://huggingface.co/datasets/m-a-p/harmonixset_bigvgan/tree/main Dataset]).
 '''Turkish Data'''

@@ Line 25: / Line 25: @@
 * [[2026:Audio Beat Tracking]]
 * [[2026:Audio Key Detection]]
-* [[2026:Audio Downbeat Tracking]]
+* [[2026:Audio Downbeat Estimation]]
 * [[2026:Music Structure Analysis]]
 * [[2026:Lyrics-to-Audio Alignment]]

@@ Line 8: / Line 8: @@
 = Data =
-== Held-out Datasets ==
+== Test Data ==
 Participants are not allowed to use any split of the following datasets for training, validation, model selection, parameter tuning, or any other development purpose:
 * Billboard 2013
 == Other Collections ==
@@ Line 18: / Line 20: @@
 Other collections may be used, but participants must clearly state their dataset usage and, when applicable, the data collection process in the extended abstract.
-The following datasets are commonly used for development or background comparison in audio chord estimation. They are listed here for context; participants remain responsible for checking that any collections they use do not overlap with the held-out datasets.
+The following datasets are commonly used for development or background comparison in audio chord estimation. They are listed here for context.
 ; Isophonics

@@ Line 23: / Line 23: @@
 '''Notice:''' We particularly encourage submissions that move beyond isolated AMT components and produce complete, valid symbolic scores.
 = Evaluation =
@@ Line 91: / Line 133: @@
 * 16 bit FLAC
-* 44.1 kHz for MuseSyn, 22 050 Hz for Quartets
+* 22 050 Hz
-* 30s (for Quartets) to over 7 minutes (for MuseSyn)
+* 30s (for Quartets)
 If your pipeline expects characteristics different from the ones described above, the conversion should be done automatically in your algorithm.
@@ Line 162: / Line 204: @@
 * [https://huggingface.co/datasets/PRAIG/quartets-quartets Quartets dataset on Hugging Face]
 = Evaluation Datasets =
@@ Line 185: / Line 215: @@
 Evaluation will include the test split of the Quartets dataset. Results on this dataset will be reported separately.
 === Undisclosed Evaluation Set ===
@@ Line 197: / Line 220: @@
 An additional custom evaluation set will be used to ensure fairness and out-of-distribution data.
-General details on this dataset are:
+It will also include quartets, but with composers other than Haydn, Mozart, and Beethoven.
-TBD
+The data structure will replicate that of the Quartets dataset, and should therefore be usable directly by the submitted algorithms.
 = Time and Hardware Limits =