SIGIR 2003: Workshop on the Evaluation of Music Information Retrieval (MIR) Systems, August 1, 2003, Toronto, Canada.

Greetings Colleagues:

If you have not already done so, please familiarize yourself with the background information found at

At the above URL you will find example White Papers from the previous workshops. A general introduction to the "MIR/MDL Evaluation Frameworks Project," of which this workshop is a part, can be found at:

Information about SIGIR 2003:

If you have any comments, suggestions or questions please contact me, J. Stephen Downie, at

Special thanks to the Andrew W. Mellon Foundation for its support of the "MIR/MDL Evalution Frameworks Project".

General Information:

Two classes of participants are envisioned: 1) presenters;  and 2) audience members.

Presenters will submit written briefing documents (i.e., White Papers) prior to the workshop. I plan on including these briefing documents in the growing collection at Based upon prior experience, there will be 8 to 12 formal presenters. Audience members, while not acting as formal presenters, will be encourage to respond to the presentations as active debate on recommendations being put forward is a key goal of the workshop.

We are looking for the submission of scholarly "White Papers" or "Position Papers". These papers will be scholarly in the sense that they must conform to the standards of attribution and referencing generally accepted in the academic world. The "White Paper" notion, however, does afford us latitude as to content. Your personal and professional expertise, experiences and judgements are explicitly be sought to help the MIR/MDL communities design and manage an ongoing framework for meaningful and rigourous MIR/MDL evaluation. As such, if you have an opinion or argument, either for, or against, a particular aspect of MIR/MDL evaluation and you can provide sufficient background information (i.e., supporting references) that will help clarify your opinion or argument, then I am almost certain that your submission will be accepted. Remember, the MIR/MDL research communities are multi-disciplinary and this fact should be taken into account as you present your opinions (i.e., provide pointers to background information so that non-subject experts can better appreciate your arguments and world-view).

Basic Open Questions and Topics:

The following, non-exclusive (nor all-encompassing) list of open questions should help you understand just a few of the many possible paper topics: These are just a few of the possible questions/topics that can be addressed. The underlying questions are:
  1. How do we determine, and then appropriately classify, the tasks that should make up the legitimate purviews of the MIR/MDL domains?
  2. What do we mean by "success"? What do we mean by "failure"?
  3. How will we decide that one MIR/MDL approach works better than another?
  4. How do we best decide which MIR/MDL approach is best suited for a particular task?

Major Workshop Themes

Major general themes to be addressed in the workshop include:
To address these majors themes, participants will be prompted to provide recommendations and commentary on specific sub-components of the themes. For example, a non-exclusive list of possible presentations includes suggestions and commentary on:

Submission Instructions:

Unlike many other Workshops, you need not necessarily attend the Workshop for your submission to be considered for inclusion in the "White Paper" collection. It is much more important that your expertise be heard and considered by those interested in MIR/MDL evaluation issue.

If you plan on attending and/or submitting a "White Paper" please drop me an email at that briefly outlines your intentions at your earliest convenience. I will be structuring Workshop presentation schedules with an eye toward being as inclusive as possible.

Due to email constraints I would appreciate that submissions, if possible, be mounted on the Web and the URL of the submission emailed to me. I or my assistant will then download your submission.

Papers should range between 2000-5000 words.

Deadline for submission: 5 July  2003

Formatting Information:

We will be using the basic ACM-style formatting structure. This is the structure that was used for the 2nd International Symposium on Music Information Retrieval, Bloomington, IN (ISMIR 2001). The formatting instructions and templates can be found at the ISMIR 2001 site You may use any standard bibliographic reference style including those in the (Author-Date) genre.

Turn around time from submission to printing/mounting on the Web will be VERY TIGHT so I ask that submissions be made in the PDF format if at all possible. In cases of undue hardship MS-Word or RTF files *may be* considered. Please be considerate of this formatting request: we simply do not have the resources nor time to reformat documents into shape for printing. is hosted by the ISRL (Information Science Research Laboratories) which is part of GSLIS (the Graduate School of Library and Information Science at UIUC (the University of Illinois at Urbana-Champaign).

J. Stephen Downie
15 April  2003