2026:Music Performance Difficulty Prediction - Revision history

Huanz: /* Training Datasets */

2026-07-01T09:36:49Z

‎Training Datasets

Huanz: /* Training Datasets */

2026-07-01T09:34:58Z

‎Training Datasets

Huanz: /* Timeline */

2026-05-31T14:25:12Z

‎Timeline

Huanz: /* Organizers */

2026-05-31T13:57:24Z

‎Organizers

Huanz: /* Organizers */

2026-05-31T13:57:09Z

‎Organizers

Huanz: /* Music Performance Difficulty Prediction 2026 */

2026-05-30T23:06:56Z

‎Music Performance Difficulty Prediction 2026

Show changes

Huanz: Created page with "= Music Performance Difficulty Prediction 2026 = Welcome to the official MIREX wiki page for the '''Music Performance Difficulty Prediction''' task, new for MIREX 2026. This..."

2026-05-30T14:22:20Z

Created page with "= Music Performance Difficulty Prediction 2026 = Welcome to the official MIREX wiki page for the '''Music Performance Difficulty Prediction''' task, new for MIREX 2026. This..."

New page

= Music Performance Difficulty Prediction 2026 =

Welcome to the official MIREX wiki page for the '''Music Performance Difficulty Prediction''' task, new for MIREX 2026. This task targets the automatic estimation of how technically and musically demanding a piece is to perform from audio recordings, supporting applications in music education, library cataloguing, and pedagogical recommendation systems.

== Overview ==

Music performance difficulty prediction has emerged as an active MIR sub-field over the past five years, with foundational datasets (Mikrokosmos-difficulty, CIPI, PSyllabus) and methods spanning symbolic, image, and audio modalities ([https://arxiv.org/abs/2203.13010 Ramoneda et al., ICASSP 2022]; [https://arxiv.org/abs/2306.08480 2023]; [https://arxiv.org/abs/2309.16287 ISMIR 2023]; [https://arxiv.org/abs/2403.03947 TASLP 2024]; [https://arxiv.org/abs/2408.00473 ISMIR 2024]). Despite this growth, there has been no community-wide evaluation: published results use inconsistent splits, label scales, and metrics, making cross-method comparison difficult.

This task establishes the first shared benchmark for audio-based music performance difficulty prediction, with a consistent ordinal label space, a held-out test set never publicly released, and standard ordinal regression metrics under composer-disjoint splits.

== Task Description ==

Participants submit systems that take a '''solo piano audio recording''' as input and output a predicted difficulty grade on an ordinal '''1–9 scale''', aligned to the Henle Verlag grading system.

* '''Input format''': WAV, 44.1 kHz, mono or stereo.
* '''Output format''': integer grade 1–9 together with a 9-dimensional confidence vector.
* '''Two recordings per piece''': each test piece is represented by at least two recordings — one human performance and one synthesized rendering — scored independently and aggregated per piece. This is designed so that systems cannot game the task by relying purely on performance-quality or recording-condition cues.

This year we deliberately focus on '''audio only''' to align with the broader applicability of audio-based difficulty prediction tools (no machine-readable score required) and to attract participants from the audio representation learning community. A symbolic-input track may be added in future editions.

== Evaluation Criteria ==

Submissions are ranked using standard ordinal regression metrics computed on the held-out test set:

* '''Mean Squared Error (MSE)''' — primary ranking metric
* '''Accuracy within one level (Acc±1)''' — fraction of predictions within one grade of the true label
* '''Balanced accuracy''' — per-class accuracy averaged across grade levels, to handle class imbalance
* '''Spearman ρ''' — rank correlation between predicted and ground-truth grades

The primary ranking uses MSE; ties are broken by Acc±1. We additionally report per-difficulty-level confusion matrices, per-composer error, and the gap between human-recording and synthesized-recording scores on the same piece (to surface systems that overfit to performance-quality cues).

== Training Datasets ==

Participants are welcome to train their systems on any dataset, including publicly available corpora, proprietary collections, or internally curated material. There are no restrictions on dataset origin, but we require full transparency in the technical report.

Suggested datasets for training and validation:

* '''[https://zenodo.org/records/12783403 PSyllabus]''' (Ramoneda et al., 2024): 7,901 audio recordings of classical piano pieces with 11-level grades. A provided mapping to the Henle 1–9 label space will be released alongside the task.
* '''[https://zenodo.org/records/8037327 CIPI (Can I Play It?)]''' (Ramoneda et al., 2023): 652 MusicXML pieces with 9-level Henle labels. Useful for participants who wish to transcribe-then-classify or for distant supervision.
* '''[https://github.com/PRamoneda/Mikrokosmos-difficulty Mikrokosmos-difficulty]''': 147 Bartók pieces with 3 difficulty levels.
* Other publicly available difficulty collections (Pianostreet-difficulty, Freescore-difficulty, Hidden Voices) may also be used.

'''Important''': no part of the held-out test set may be used for training. A list of forbidden pieces will be published with the task call. Please describe in your technical report:
* Dataset name and source
* Size and number of pieces
* Any preprocessing, cleaning, or label-space remapping applied

== Held-Out Test Set ==

To ensure label trust and prevent training-set contamination, the held-out test set (target size '''≈ 200 pieces''') is constructed by the task captain in three stages and is '''not publicly released''':

# '''Piece selection''': drawn from sources outside existing public difficulty datasets — (a) Henle Verlag catalogue additions published after the CIPI cutoff; (b) ABRSM, Trinity, and RCM 2024–2026 syllabus pieces not present in PSyllabus; (c) a curated set of contemporary and historically underrepresented composers (e.g. Hidden Voices) graded specifically for this task.
# '''Grade verification''': each piece is graded independently by at least two expert pianists recruited from the QMUL music programme and the ABRSM examiner network on the Henle 1–9 scale. Inter-annotator agreement is reported alongside the final results. Disagreements >1 level are resolved by a third annotator.
# '''Audio rendering''': for each piece, at least two reference recordings are produced — (i) a human performance, either recorded on a Yamaha Disklavier at QMUL or sourced from licensed commercial recordings where redistribution of features is permitted; and (ii) a high-quality synthesized rendering from the verified score, generated with a fixed renderer.

'''Test set governance''': the held-out audio files and labels remain private to the task captain. Participants submit systems; the task captain runs inference on the held-out set and reports metrics back. Test recordings and labels are not released after the competition, enabling re-use and incremental growth across editions.

== Submission Requirements ==

The following items are required for submission:

* '''System''': packaged as a Docker container with a standardised inference interface — input is a path to a WAV file; output is an integer grade 1–9 with a 9-dimensional confidence vector. A reference Docker template and inference wrapper script will be provided.
* '''Technical report''': 2–4 pages in ISMIR LBD format describing training data, model architecture, label-space handling, and any post-processing.
* '''Compute declaration''': GPU memory footprint and average inference time per piece.

Final submission must be made through the [[2026_Submission_Guidelines|MIREX submission system]]. Submissions exceeding the 24-hour total inference budget on the full test set, or failing on more than 5% of test items, are reported but excluded from the primary ranking.

== Timeline ==

(To be confirmed in line with the overall MIREX 2026 schedule.)

* '''TBD''': Task call published, training-data mapping released
* '''TBD''': Submission system opens
* '''TBD''': Submission deadline
* '''TBD''': Results announced at ISMIR 2026

== Organizers ==

* '''Huan Zhang''' (Task Captain, Queen Mary University of London)
* '''Pedro Ramoneda''' (Universitat Pompeu Fabra)

Contact: [mailto:huan.zhang@qmul.ac.uk huan.zhang@qmul.ac.uk]

@@ Line 52: / Line 52: @@
 Suggested datasets for training and validation:
-* '''[https://zenodo.org/records/12783403]''' (Ramoneda et al., 2025): 7,901 audio recordings of classical piano pieces with 11-level pedagogical grades, together with mappings to multiple international grading systems. We specifically encourage participants to leverage the multiple ranking annotations available in PSyllabus (13 grading/ranking systems in total), not only the default labels, in order to study and improve generalization across heterogeneous pedagogical traditions and difficulty scales.
+* '''[https://zenodo.org/records/14794592 Psyllabus]''' (Ramoneda et al., 2025): 7,901 audio recordings of classical piano pieces with 11-level pedagogical grades, together with mappings to multiple international grading systems. We specifically encourage participants to leverage the multiple ranking annotations available in PSyllabus (13 grading/ranking systems in total), not only the default labels, in order to study and improve generalization across heterogeneous pedagogical traditions and difficulty scales.
-* '''[https://zenodo.org/records/8037327]''' (Ramoneda et al., 2024): 652 MusicXML piano pieces with 9-level Henle annotations, useful for participants interested in transcription-then-classification pipelines or distant supervision approaches.
+* '''[https://zenodo.org/records/8037327 Can I Play It? (CIPI)]''' (Ramoneda et al., 2024): 652 MusicXML piano pieces with 9-level Henle annotations, useful for participants interested in transcription-then-classification pipelines or distant supervision approaches.
 * '''[https://github.com/PRamoneda/Mikrokosmos-difficulty Mikrokosmos-difficulty]''' (Ramoneda et al., 2022) and other publicly available piano difficulty datasets and pedagogical collections.

@@ Line 52: / Line 52: @@
 Suggested datasets for training and validation:
-* '''[https://zenodo.org/records/12783403 PSyllabus]''' (Ramoneda et al., 2025): 7,901 audio recordings of classical piano pieces with 11-level pedagogical grades, together with mappings to multiple international grading systems. We specifically encourage participants to leverage the multiple ranking annotations available in PSyllabus (13 grading/ranking systems in total), not only the default labels, in order to study and improve generalization across heterogeneous pedagogical traditions and difficulty scales.
+* '''[https://zenodo.org/records/12783403]''' (Ramoneda et al., 2025): 7,901 audio recordings of classical piano pieces with 11-level pedagogical grades, together with mappings to multiple international grading systems. We specifically encourage participants to leverage the multiple ranking annotations available in PSyllabus (13 grading/ranking systems in total), not only the default labels, in order to study and improve generalization across heterogeneous pedagogical traditions and difficulty scales.
-* '''[https://zenodo.org/records/8037327 CIPI (Can I Play It?)]''' (Ramoneda et al., 2024): 652 MusicXML piano pieces with 9-level Henle annotations, useful for participants interested in transcription-then-classification pipelines or distant supervision approaches.
+* '''[https://zenodo.org/records/8037327]''' (Ramoneda et al., 2024): 652 MusicXML piano pieces with 9-level Henle annotations, useful for participants interested in transcription-then-classification pipelines or distant supervision approaches.
 * '''[https://github.com/PRamoneda/Mikrokosmos-difficulty Mikrokosmos-difficulty]''' (Ramoneda et al., 2022) and other publicly available piano difficulty datasets and pedagogical collections.

@@ Line 85: / Line 85: @@
 == Timeline ==
-(To be confirmed in line with the overall MIREX 2026 schedule.)
+* '''July 1, 2026''': Submission system opens
+* '''Oct 1, 2026''': Submission deadline
-* '''TBD''': Task call published, training-data mapping released
+* '''Oct 15, 2026''': Results published
 == Long-term Plan ==

@@ Line 101: / Line 101: @@
 * '''Pedro Ramoneda''' (Songscription) — [mailto:pedro@songscription.ai pedro@songscription.ai]
-* '''Huan Zhang''' (Clefer) — [mailto:huan@clefer.com huan.zhang@qmul.ac.uk]
+* '''Huan Zhang''' (Clefer) — [mailto:huan@clefer.com huan@clefer.com]
 == References ==

@@ Line 100: / Line 100: @@
 == Organizers ==
 * '''Pedro Ramoneda''' (Songscription) — [mailto:pedro@songscription.ai pedro@songscription.ai]
 == References ==