Difference between revisions of "2007:Main Page"

From MIREX Wiki
(MIREX 2007 Task Proposal Pages)
(Paper)
Line 79: Line 79:
 
- What file format should be used for ground truth and output?
 
- What file format should be used for ground truth and output?
  
== Paper ==
 
  
1. Harte, Sandler, Abdallah, G├│mez (2005): '''Symbolic Representation Of Musical Chords: A Proposed Syntax For Text Annotations.''' Proceedings of 6th International Conference on Music Information Retrieval
 
  
 
==Something New To Consider: MIREX DIY Demonstration==
 
==Something New To Consider: MIREX DIY Demonstration==

Revision as of 05:48, 9 February 2007

Welcome to MIREX 2007

This is the main page for the Third Music Information Retrieval Evaluation eXchange (MIREX 2007). The International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL) at the Graduate School of Library and Information Science (GSLIS), University of Illinois at Urbana-Champaign (UIUC) is the principal organizer of MIREX 2007.

The MIREX 2007 community will hold its annual plenary meeting as part of The 8th International Conference on Music Information Retrieval, ISMIR 2007, which will be held at the Vienna University of Technology, from September 23rd to September 27th, 2007.

J. Stephen Downie
Director, IMIRSEL

Getting Involved in MIREX 2007

MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2007 the best yet.

Mailing List Participation

If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2007 tasks. Subscription information at: https://mail.isrl.uiuc.edu/mailman/listinfo/evalfest

Wiki Participation

Please note that you must create a NEW login for this wiki even if you have a login that you previously used for editing the MIREX 2005 or 2006 wikis.

Please create an account via: Create Account.

MIREX 2005 and 2006 Wikis

This is the new wiki for MIREX 2007. The wikis for MIREX 2005 and 2006 are available at:

MIREX 2005 https://www.music-ir.org/mirex2005/

MIREX 2006 https://www.music-ir.org/mirex2006/

You can interlink between this wiki and the previous wikis using 2005: prefix on links to connect to pages in MIREX 2005 and 2006: for MIREX 2006.

Introduction

For many applications in music information retrieval, extracting the harmonic structure is very desirable, for example for segmenting pieces into characteristic segments, for finding similar pieces, or for semantic analysis of music.

The extraction of the harmonic structure requires the detection of as many chords as possible in a piece. That includes the characterisation of chords with a key and type as well as a chronological order with onset and duration of the chords.

Although some publications are available on this topic, comparison of the results is difficult, because different measures are used to assess the performance. To overcome this problem an accurately defined methodology is needed. This includes a repertory of the findable chords, a defined test set along with ground truth and unambiguous calculation rules to measure the performance.

Regarding this we suggest to introduced the new evaluation task Audio Chord Detection.


Data

As this is intended for music information retrieval, the analysis should be performed on real world audio, not resynthesized MIDI or special renditions of single chords. We suggest the test bed consists of WAV-files in CD quality (with a sampling rate of 44,1kHz and a solution of 16 bit). A representative test bed should consist of more than 50 songs of different genres like pop, rock, jazz and so on.

For each song in the test bed, a ground truth is needed. This should comprise all detectable chords in this piece with their tonic, type and temporal position (onset and duration) in a machine readable format that is still to be specified.

To define the ground truth, a set of detectable chords has to be identified. We propose to use the following set of chords build upon each of the twelve semitones.

Triads: major, minor, diminished, augmented, suspended4
 Quads: major maj7, major 7, major add9, major maj7/#5 
        minor maj7, minor 7, minor add9, minor 7/b5
        maj7/sus4, 7/sus4

An approach for text annotation of musical chords is presented in [1].

We could contribute excerpts of approximately 30 pop and rock songs including a ground truth.


Evaluation

Two common measures from field of information retrieval are recall and precision. They can be used to evaluate a chord detection system.

Recall: number of time units where the correct chords have been detected by the algorithm divided by the number of time units which contain detectable chords in the ground truth

Precision: number of time units where the correct chords have been detected divided by number of time units where the algorithm has detected any chord at all.


Points to discuss:

- What temporal resolution should be used for ground truth and results?

- How should enharmonic and other confusions of chords be handled?

- What is the maximal acceptable onset deviation between ground truth and result?

- What file format should be used for ground truth and output?


Something New To Consider: MIREX DIY Demonstration

Please take a look at: http://cluster3.lis.uiuc.edu:8080/mirexdiydemo/. This is a prototype "proof-of-concept" demonstration of the Music Information Retrieval Evaluation eXchange (MIREX) "Do-It-Yourself" (DIY) web service system.

A recent article in D-Lib Magazine is a good introduction to what we are trying to accomplish:

Downie, J. Stephen. 2006. The Music Information Retrieval Evaluation eXchange (MIREX). In D-Lib Magazine 12 (Issue 12). Available: http://dlib.org/dlib/december06/downie/12downie.html.

Would like to try to have a least one task (or some task beta-testers) use a similar set up for MIREX 2007. A MIREX DIY Discussion page has been started for community input.

Descriptions of the Past Evaluation Tasks

MIREX 2006 Evaluation Tasks

MIREX 2005 Evaluation Tasks