The International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL) Project

The objective of the International Music Information Retrieval Systems Evaluation Laboratory project (IMIRSEL) is the establishment of the necessary resources for the scientifically valid development and evaluation of emerging Music Information Retrieval (MIR) and Music Digital Library (MDL) techniques and technologies. Part of the project is the creation of secure, yet accessible, large-scale collections of music materials in a variety of audio, symbolic and metadata forms. These collections, when coupled with a set of standardized experimental tasks and standardized evaluation metrics, will allow members of the international MIR/MDL research community to participate in TREC-like evaluation "contests" so they can scientifically compare and contrast their various approaches to making the world's vast store of musical heritage materials ever more available.

IMIRSEL is located at the Graduate School of Library and Information Science (GSLIS) at the University of Illinois at Urbana-Champaign (UIUC). Project Principal Investigator is J. Stephen Downie of GSLIS and Co-Principal Investigator is Prof. Michael Welge of the Automated Learning Group (ALG) of the National Center for Supercomputing Applications (NCSA).

Principal Project Components

The IMIRSEL project comprises to major subprojects:

The Virtual Research Labs (VRL) using Music-to-Knowledge (M2K) project

The VRL subproject is being undertaken to provide a uniform mechanism for the international MIR/MDL community to access the standardized resources of IMIRSEL in a robust, yet secure, manner. The VRLs are constructed using IMIRSEL's M2K rapicd prototyping and evalution environment. M2K is an open-sourced extension of the D2K (Data-to-Knowlege)/Text-to-Knowlege (T2K) Java-based datamining framework, developed by the ALG at NCSA. For more information on M2K please read our M2K (Music-to-Knowledge): A tool set for MIR/MDL development and evaluation pages.

The Human Use of Music Information Retrieval Systems (HUMIRS)

The HUMIRS subproject is designed to provide answers to the Who, What, Where, When , Why and How questions as they pertain to the use of MIR and MDL systems. By focusing on real-world examples of music information seeking the HUMIRS subproject will allow IMIRSEL to develop a set of experimental MIR/MDL evaluation task grounded in reality. This real-world grounding will thus make the set of evaluation tasks much more meaningful as developers prepare their MIR/MDL systems for real-world deployment.

Project Sponsors

The Andrew W. Mellon Foundation
The National Science Foundation (NSF): Grants No. IIS-0340597 and No. IIS-0327371

International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL) Project related:

J. Stephen Downie. "Report on ISMIR 2002 Conference Panel I: Music Information Retrieval Evaluation Frameworks." D-Lib Magazine 8(11). November 2002.
Available at: http://www.dlib.org/dlib/november02/11inbrief.html#DOWNIE
J. Stephen Downie. "The TREC Like Evaluation of Music Retrieval Systems" Appendix C of The MIR/MDL Evaluation Project White Paper Collection, Edition #3
Available at: http://www.music-ir.org/evaluation/wp3/wp3_appendixC.pdf
J. Stephen Downie (ed.) The MIR/MDL Evaluation Project White Paper Collection, Edition #3 Includes: Part I. Papers Presented at the Workshop on the Creation of Standardized Test Collections, Tasks, and Metrics for Music Information Retrieval (MIR) and Music Digital Library (MDL) Evaluation, 18 July, 2002. Part II. Panel on Music Information Retrieval Evaluation Frameworks at ISMIR 2002, 17 October, 2002. and Part III. Workshop on the Evaluation of Music Information Retrieval (MIR) Systems at SIGIR 2003, 1 August, 2003.
Available at:http://www.music-ir.org/evaluation/wp.html
J. Stephen Downie. (2004). "Supercomputing in the Humanities: A Robust Model for Interacting with Copyright-Sensitive Multimedia Content." In Proceedings of ACH/ALLC 2004.
Available at: http://www.hum.gu.se/allcach2004/AP/html/prop127.html
J. Stephen Downie, Andreas F. Ehmann and Xiao Hu. (2005). Music-to-Knowledge (M2K): a prototyping and evaluation environment for music digital library research. In Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2005), Denver, CO, 7-11 June, 2005. New York, NY: ACM Press, p.376.
J. Stephen Downie, Andreas F. Ehmann and David Tcheng. (2005). Music-to-knowledge (M2K): a prototyping and evaluation environment for music information retrieval research. In Proceedings of the 28th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (SIGIR 2005), Salvador, Brazil, August 15 - 19, 2005. New York, NY: ACM Press, pp. 676-676.
J. Stephen Downie, Jin Ha Lee, Anatoliy Gruzd and M. Cameron Jones. (2007). Toward an understanding of similarity judgments for music digital library evaluation. In Proceedings of the ACM IEEE Joint Conference on Digital Libraries 2007, Vancouver, Canada. [In print].

Human Use of Music Information Retrieval Systems (HUMIRS) Project related:

J. Stephen Downie. (2004). The Creation of Music Query Documents: Framework and Implications of the HUMIRS Project. In Proceedings of ACH/ALLC 2004.
Available at: http://www.hum.gu.se/allcach2004/AP/html/prop134.html
Jin Ha Lee and J. Stephen Downie. (2004). Survey Of Music Information Needs, Uses, And Seeking Behaviours: Preliminary Findings. In Proceedings of the Fifth International Conference on Music Information Retrieval: ISMIR 2004, Barcelona, Spain, 441-446.
Available at: http://ismir2004.ismir.net/proceedings/p081-page-441-paper232.pdf
Jin Ha Lee, J. Stephen Downie and Sally Jo Cunningham (2005). Challenges in cross-cultural/multilingual music information seeking. In Proceedings of the Sixth International Conference on Music Information Retrieval: ISMIR 2005, London, UK, 1-7.
Available at: http://ismir2005.ismir.net/proceedings/1100.pdf
Jin Ha Lee, Xiao Hu and J. Stephen Downie (2005). Q&A websites: Rich research resources for contextualizing information retrieval behaviors. In Proceedings of the ACM SIGIR 2005 Workshop on Information Retrieval in Context (IRIX), Salvador, Brazil, 33-36.
Available at: http://irix.umiacs.umd.edu/ACM-SIGIR2005-IRiX-proceedings.pdf
Xiao Hu, J. Stephen Downie, and Andreas Ehmann (2006). Exploiting Recommended Usage Metadata: Exploratory Analyses. In Proceedings of the Seventh International Conference on Music Information Retrieval: ISMIR 2006, Victoria, Canada, 19-22.
Available at: http://ismir2006.ismir.net/PAPERS/ISMIR06157_Paper.pdf
Jin Ha Lee, M. Cameron Jones, and J. Stephen Downie (2006). Factors affecting the response rates of real-life MIR queries. In Proceedings of the Seventh International Conference on Music Information Retrieval: ISMIR 2006, Victoria, Canada, 371-372.
Available at: http://ismir2006.ismir.net/PAPERS/ISMIR0660_Paper.pdf

Maintained by :J Stephen Downie
Comments to : jdownie at uiuc dot edu

Last modified: 7 April 2007