2024:Music Description & Captioning

Task Description

The MIREX 2024 Music Captioning Task invites participants to develop models capable of generating accurate and descriptive captions for music clips. This task aims to push the boundaries of music understanding by advancing models that can interpret and describe musical content in natural language, thereby enhancing accessibility and comprehension of music.

Participants are tasked with creating systems that generate captions for a collection of music clips from the Song Describer dataset. The generated captions will be assessed using several evaluation metrics to gauge the effectiveness and performance of the models.

Dataset

Description

The Song Describer dataset (SDD) serves as the benchmark for this task. SDD is a meticulously curated collection of music clips, each paired with detailed textual descriptions crafted by volunteers. This dataset provides a robust foundation for evaluating music captioning models.

SDD comprises 1,106 captions for 706 music recordings, with a validated subset of 746 captions for 547 recordings. The clips represent a wide array of genres and styles, ensuring a comprehensive representation of musical content. The dataset is annotated with an emphasis on capturing the intricate details of the audio through precise textual descriptions.

Description of Audio Files

• The audio clips in SDD are carefully selected from the MTG-Jamendo dataset. Each clip is up to 2 minutes long (95% are 2 minutes), providing a rich diversity of musical genres and styles.
• Audio files are provided in 320kbps 44.1 kHz MP3 audio encoding.

Description of Text

• Each clip in the SDD is accompanied by one to five free-text captions written by volunteers. These captions focus on describing musical elements such as genre, instrumentation, mood, and other relevant characteristics.
• Captions are single-sentence descriptions, with an average length of 21.7 words in the full dataset and 18.2 words in the validated subset.

Description of Split

• While there is no recommended split for training and evaluation, the dataset is intended to be used solely for evaluation purposes. Participants should not use any part of SDD for training or validation.
• A validated subset is provided, containing manually reviewed captions that adhere strictly to the annotation guidelines.

Baseline

SD-MusicCaps: Model Architecture

• LP-MusicCaps utilizes a cross-modal encoder-decoder transformer architecture, designed to generate high-quality captions for music clips. The encoder processes 10-second audio signals by converting them into log-mel spectrograms, which are then refined through convolutional layers with GELU activation to extract critical audio features. These features, combined with positional encoding, are further processed by transformer blocks that understand the sequence and context of the audio data.

• The decoder is responsible for generating text captions from these encoded audio features. It uses transformer blocks similar to those in the encoder, processes tokenized text, and employs multi-head attention to ensure that the generated captions are contextually relevant.

• A key feature of LP-MusicCaps is the augmentation with a large language model (LLM), which enhances the model's ability to generate sophisticated and contextually rich captions.

Metrics

• The evaluation of submitted systems will be based on multiple metrics, with ROUGE-L serving as the primary metric for determining the final ranking. The metrics include:

ROUGE-L

Measures the overlap of the longest common subsequence between the generated and reference captions, serving as the main determinant of the final ranking.

BLEU (B1~B4)

Evaluates n-gram precision, with B1 to B4 representing unigram to 4-gram matches.

METEOR

Incorporates precision, recall, and synonymy matching to improve alignment with human judgment.

BERT-Score

Computes token similarity using contextual embeddings from BERT.

• While each metric will contribute to a ranking, ROUGE-L will primarily determine the final standings.

Download

The Song Describer dataset, including both the audio clips and their corresponding captions, is available for download from Zenodo (DOI: https://doi.org/10.5281/zenodo.10072001). Participants should download the dataset from this source to ensure they are using the correct version for the challenge.

Rules

• Participants are allowed to utilize external datasets and pre-trained models in developing their systems. However, the use of the Song Describer dataset for training or validation is strictly prohibited.
• Participants must ensure that their models do not use any information from the MTG-Jamendo dataset beyond what is provided in the Song Describer dataset.
• All submissions must respect the CC BY-SA 4.0 license under which the Song Describer dataset is released.

Submission

• Submissions will be evaluated using CodaBench (https://www.codabench.org/) for automated assessment.

• Submission Deadline: October 15, AOE

• Participants are required to submit the following:

JSON file

A JSON file containing the generated captions for the evaluation dataset. The format should match the structure provided in the Song Describer dataset.

PDF file

A PDF file detailing the system architecture, training process, and any external data or models used. This should include a clear statement that the Song Describer dataset was not used for training or validation.

Example

{
  "1004034": "Upbeat electronic dance music with a pulsing synthesizer melody and rhythmic drum patterns, suitable for a lively party atmosphere.",
  "1007274": "Gentle acoustic guitar instrumental featuring intricate fingerpicking and a soothing melody, perfect for a calm and reflective mood.",
  "1009321": "Energetic rock song with distorted electric guitars, powerful drumming, and passionate vocals, ideal for an intense workout session."
...
}

Note: Although an audio file in the dataset may correspond to multiple captions, participants only need to submit one generated caption for each audio file (identified by track_id). During the evaluation phase, multiple reference captions will be reflected in the calculation of metrics through the multi-reference evaluation method. The number of entries in the submitted JSON file should match the number of audio files in the assessment dataset, not the total number of original description texts.

• Each participant or team may submit up to four versions of their system. The final ranking will be based on the metrics outlined above.

Paper

Research paper submission

Participants are encouraged to submit the technical report to the MIREX track at ISMIR 2024.

Workshop presentation

We will invite top-ranked participants to present their work during the workshop session. The format will be hybrid to accommodate remote participation.

Bibliography

[1] Doh, S., Choi, K., Lee, J., & Nam, J. (2023, November). LP-MusicCaps: LLM-Based Pseudo Music Captioning. In ISMIR 2023 Hybrid Conference.

[2] Manco, I., Weck, B., Doh, S., Won, M., Zhang, Y., Bodganov, D., ... & Nam, J. The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation.