MIREX Wiki - User contributions [en]

User:UCASlsc

2022-01-23T14:56:02Z

Djevans: Creating user page for new user.

Mainly focus on sound event detection.

User talk:UCASlsc

2022-01-23T14:56:02Z

Djevans: Welcome!

'''Welcome to ''MIREX Wiki''!'''
We hope you will contribute much and well.
You will probably want to read the [https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents help pages].
Again, welcome and have fun! [[User:Djevans|Djevans]] ([[User talk:Djevans|talk]]) 08:56, 23 January 2022 (CST)

User:H7878778h

2021-12-14T19:17:50Z

Djevans: Creating user page for new user.

AI developer && music enthusiast. bla bla bla bla bla bla

User talk:H7878778h

2021-12-14T19:17:50Z

Djevans: Welcome!

User:Pipiqiang2

2021-12-14T19:17:34Z

Djevans: Creating user page for new user.

music drum transcript and speech recognition

User talk:Pipiqiang2

2021-12-14T19:17:34Z

Djevans: Welcome!

User:Iehppp2010

2021-12-14T19:17:25Z

Djevans: Creating user page for new user.

an audio algorithm engineer Mainly focus on ASR TTS MIR

User talk:Iehppp2010

2021-12-14T19:17:25Z

Djevans: Welcome!

User talk:Xiaoqiang2

2021-12-14T19:16:50Z

Djevans: Welcome!

User:Xiaoqiang2

2021-12-14T19:16:50Z

Djevans: Creating user page for new user.

music drum transcript and speech recognition

User:AIBurashnikova

2021-12-14T19:16:23Z

Djevans: Creating user page for new user.

I'm finishing my PhD at Skolkovo Institute of Science and Technology. I'm also interested in Music Generation and Style Transfer Tasks, Beat tracking, Alignment tasks.

User talk:AIBurashnikova

2021-12-14T19:16:23Z

Djevans: Welcome!

User:Tshpak2

2021-12-14T19:16:15Z

Djevans: Creating user page for new user.

I am RND researcher and work in Music Processing domain.

User talk:Tshpak2

2021-12-14T19:16:15Z

Djevans: Welcome!

User:Ju-chiang.wang

2021-12-14T19:16:08Z

Djevans: Creating user page for new user.

I am a Research Scientist and a member of the Speech, Audio, and Music Intelligence (SAMI) team at ByteDance in Mountain View, California. My research interests are currently focusing on music AI and machine learning with applications to music content understanding, intelligent music editing & remixing, and audiovisual cross-modal retrieval. I received my Ph.D. degree in Electrical Engineering from National Taiwan University. From 2013 to 2015, I did my postdoc at Academia Sinica, Taiwan, and University of California, San Diego, USA. In 2013, I was a Visiting Researcher at the Sound and Music Computing Lab in National University of Singapore. Prior to joining ByteDance in 2019, I worked at Cisco WebEx for 3.75 years on media engine and intelligent audio quality monitoring.

User talk:Ju-chiang.wang

2021-12-14T19:16:08Z

Djevans: Welcome!

User:Axle

2021-12-14T19:15:31Z

Djevans: Creating user page for new user.

Hi!
My name is Axel Marmoret, I'm 24 years old, and I'm a PhD Student at IRISA, Rennes in France.
My PhD focuses on structural segmentation of music, that is, techniques to retrieve simplified organisation of a song.
Previously to my PhD, I was graduated as a Computer Science engineer at the "Mines de Douai" School (in 2018), and graduated from a Research Master's degree in Machine and Deep Learning at INSA Rennes/University of Rennes 1 (in 2019).

User talk:Axle

2021-12-14T19:15:31Z

Djevans: Welcome!

User:Gongyibei

2021-12-14T19:15:05Z

Djevans: Creating user page for new user.

music information retrieval, music visualization, guitarist

User talk:Gongyibei

2021-12-14T19:15:05Z

Djevans: Welcome!

User:Satvikvenkatesh

2021-12-14T19:14:56Z

Djevans: Creating user page for new user.

Satvik holds a Bachelor of Technology in Information and Communication Technology from SASTRA University, India and a Research Masters in Computer Music from ICCMR, University of Plymouth, UK. He currently is studying for a PhD in ICCMR on the topic of audio segmentation and intelligent mixing for live radio broadcast. His research interests include Deep Learning, Brain-Computer Music Interfaces, and Unconventional Computing for music. Satvik is also an accomplished musician and performer.

Website: http://satvik-venkatesh.github.io/

User talk:Satvikvenkatesh

2021-12-14T19:14:56Z

Djevans: Welcome!

User:SineWu

2021-12-14T19:14:22Z

Djevans: Creating user page for new user.

I'm currently working toward the master's degree at School of Cyber Science and Engineering, Sichuan University, Chengdu, China. My research interests include music information retrieval, audio fingerprinting, deep hashing, and audio watermarking.

User talk:SineWu

2021-12-14T19:14:22Z

Djevans: Welcome!

User:Hanhaodi Zhang

2021-12-14T19:13:45Z

Djevans: Creating user page for new user.

My name is Hanhaodi Zhang, graduated from University of Queensland.

User talk:Hanhaodi Zhang

2021-12-14T19:13:45Z

Djevans: Welcome!

MIREX HOME

2021-11-12T18:16:08Z

Djevans: /* MIREX 2021 Possible Evaluation Tasks */

==Welcome to MIREX 2021==

This is the main page for the 17th running of the Music Information Retrieval Evaluation eXchange (MIREX 2021). The International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL) at [https://ischool.illinois.edu School of Information Sciences], University of Illinois at Urbana-Champaign ([https://illinois.edu UIUC]) is the principal organizer of MIREX 2021.

The MIREX 2021 community will hold its annual meeting as part of [https://ismir2021.ismir.net/ The 21st International Society for Music Information Retrieval Conference], ISMIR 2021, which will be held in an online format, November 8–12, 2021.

J. Stephen Downie 
Director, IMIRSEL 

==Task Leadership Model==

Like previous years, we are prepared to improve the distribution of tasks for the upcoming MIREX 2021. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead a task, please complete the form [https://forms.gle/fAACmt9qtXxEf97G8 here]. Current information about task captains can be found on the [[2021:Task Captains]] page. Please direct any communication to the [https://lists.ischool.illinois.edu/lists/admin/evalfest EvalFest] mailing list.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

We really need leaders to help us!

==MIREX 2021 Possible Evaluation Tasks==
* [[2021:Audio Chord Estimation]]
* [[2021:Audio Cover Song Identification]]
* [[2021:Audio Melody Extraction]]
* [[2021:Lyrics Transcription (former: Automatic Lyrics-to-Audio Alignment)]](site under-construction)
* [[2021:Drum Transcription]]
* [[2021:Music Detection]]
* [[2021:Query by Singing/Humming]]
* [[2021:Set List Identification]]

==MIREX 2021 Deadline Dates==

Due to the extraneous circumstances brought on by COVID-19, we do not anticipate having all tasks wrapped up by the ISMIR conference. However, we still hope to meet with partial results and continue working on this after the conclusion of ISMIR.

==MIREX 2021 Submission Instructions==
* Be sure to read through the rest of this page
* Be sure to read though the task pages for which you are submitting
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2021 Submission Instructions]] including both the tutorial video and the text
* The MIREX 2021 Submission System is coming soon at: https://www.music-ir.org/mirex/sub/.

==MIREX 2021 Evaluation==

===Note to New Participants===
Please take the time to read the following review articles that explain the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

Downie, J. Stephen, Andreas F. Ehmann, Mert Bay and M. Cameron Jones. (2010). 
The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. 
''Advances in Music Information Retrieval'' Vol. 274, pp. 93-115 
Available at: [http://bit.ly/KpM5u5 http://bit.ly/KpM5u5]

===Runtime Limits===

We reserve the right to stop any process that exceeds runtime limits for each task. We will do our best to notify you in enough time to allow revisions, but this may not be possible in some cases. Please respect the published runtime limits.

===Note to All Participants===

Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted program(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2020 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same).
# present a poster at the MIREX 2020 poster session at ISMIR 2020, if there is a physical component to the conference.

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL currently supports some of the software/architecture dependencies for your submission, please contact [mailto:yunhao2@illinois.edu IMIRSEL team] as early as possible. Failing to notify the team might result in your submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2021==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2020 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2021 tasks. To subscribe to EvalFest, send a message to [mailto:lists@ischool.illinois.edu lists@ischool.illinois.edu] with the subject line “subscribe evalfest”

If you are participating in MIREX 2021, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2021 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2020 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
If you find that you cannot edit a MIREX wiki page, you will need to create a new account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2020 Wikis==
Content from MIREX 2005 - 2020 are available at:
'''[[2020:Main_Page|MIREX 2020]]'''
'''[[2019:Main_Page|MIREX 2019]]'''
'''[[2018:Main_Page|MIREX 2018]]'''
'''[[2017:Main_Page|MIREX 2017]]'''
'''[[2016:Main_Page|MIREX 2016]]'''
'''[[2015:Main_Page|MIREX 2015]]'''
'''[[2014:Main_Page|MIREX 2014]]'''
'''[[2013:Main_Page|MIREX 2013]]'''
'''[[2012:Main_Page|MIREX 2012]]'''
'''[[2011:Main_Page|MIREX 2011]]'''
'''[[2010:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

2021:Audio Beat Tracking

2021-10-28T19:42:27Z

Djevans: /* Collections */

== Description ==
The text of this section was copied from the 2012 Wiki. Please add your comments and discussion at the bottom of this page.

The aim of the automatic beat tracking task is to track each beat locations in a collection of sound files. Unlike the Audio Tempo Extraction task, which aim is to detect tempi for each file, the beat tracking task aims at detecting all beat locations in recordings. The algorithms will be evaluated in terms of their accuracy in predicting beat locations annotated by a group of listeners.

== Data ==
=== Collections ===
The original 2006 dataset contains 160 30-second excerpts (WAV format) used for the Audio Tempo and Beat contests in 2006. Beat locations have been annotated in each excerpt by 40 different listeners (39 listeners for a few excerpts. The length of each excerpt is 30 seconds. These audio recordings were selected to provide a stable tempo value, a wide distribution of tempi values, and a large variety of instrumentation and musical styles. About 20% of the files contain non-binary meters, and a small number of examples contain changing meters. One disadvantage of using this set for beat tracking is that the tempi are rather stable and this set will not test beat-tracking algorithms in their ability to track tempo changes.

The second collection is comprised of 367 Chopin Mazurkas, represented as full audio tracks (WAV format). The Mazurka dataset contains tempo changes so it will evaluate the ability of algorithms to track these.

The third collection was assembled and donated in 2012. This dataset contains 217 excerpts around 40s each, of which 19 are "easy" and the remaining 198 are "hard". The harder excerpts were drawn from the following musical styles: Romantic music, ﬁlm soundtracks, blues, chanson and solo guitar.

This dataset has been designed for radically new techniques which can contend with challenging beat tracking situations like: quiet accompaniment, expressive timing, changes in time signature, slow tempo, poor sound quality etc. So, if your beat tracker likes a 4/4 time-signature with a steady tempo and needs clear percussive onsets, don't expect it to do very well!
But don't be deterred, this is for the good of beat tracking.

You can read in detail about how the dataset was made here:
[http://dx.doi.org/10.1109/TASL.2012.2205244 ''Selective Sampling for Beat Tracking Evaluation'']

The dataset can be found under the [[#Relevant Development Collections| Relevant Development Collections]]

=== Audio Formats ===

The data are monophonic sound files, with the associated onset times and data about the annotation robustness.

* CD-quality (PCM, 16-bit, 44100 Hz)
* single channel (mono)
* file length between 2 and 36 seconds (total time: 14 minutes)

== Submission Format ==
Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.

=== Input Data ===
Participating algorithms will have to read audio in the following format:

* Sample rate: 44.1 KHz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

=== Output Data ===

The beat tracking algorithms will return beat-times in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.

=== Output File Format (Audio Beat tracking) ===

The Beat Tracking output file format is an ASCII text format. Each beat time is specified, in seconds, on its own line. Specifically,

<beat time(in seconds)>\n

where \n denotes the end of line. The < and > characters are not included. An example output file would look something like:

0.243
0.486
0.729

=== Algorithm Calling Format ===

The submitted algorithm must take as arguments a SINGLE .wav file to perform the onset detection on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:

foobar %input %output
foobar -i %input -o %output

Moreover, if your submission takes additional parameters, such as a detection threshold, foobar could be called like:

foobar .1 %input %output
foobar -param1 .1 -i %input -o %output

If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example:

foobar('%input','%output')
foobar(.1,'%input','%output')

=== README File ===

A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.

For instance, to test the program foobar with different values for parameters param1, the README file would look like:

foobar -param1 .1 -i %input -o %output
foobar -param1 .15 -i %input -o %output
foobar -param1 .2 -i %input -o %output
foobar -param1 .25 -i %input -o %output
foobar -param1 .3 -i %input -o %output
...

For a submission using MATLAB, the README file could look like:

matlab -r "foobar(.1,'%input','%output');quit;"
matlab -r "foobar(.15,'%input','%output');quit;"
matlab -r "foobar(.2,'%input','%output');quit;"
matlab -r "foobar(.25,'%input','%output');quit;"
matlab -r "foobar(.3,'%input','%output');quit;"
...

The different command lines to evaluate the performance of each parameter set over the whole database will be generated automatically from each line in the README file containing both '%input' and '%output' strings.

== Evaluation Procedures ==

The evaluation methods are taken from the beat evaluation toolbox and
are described in the following technical report:

M. E. P. Davies, N. Degara and M. D. Plumbley. "Evaluation methods for musical audio beat tracking algorithms". [http://www.elec.qmul.ac.uk/people/markp/2009/DaviesDegaraPlumbley09-evaluation-tr.pdf ''Technical Report C4DM-TR-09-06'']. This link now works! :)

For further details on the specifics of the methods please refer to the
paper. However, here is a brief summary with appropriate references:

*'''F-measure''' - the standard calculation as used in onset evaluation but
with a 70ms window.

S. Dixon, "Onset detection revisited," in ''Proceedings of 9th
International Conference on Digital Audio Effects (DAFx)'', Montreal,
Canada, pp. 133-137, 2006.

S. Dixon, "Evaluation of audio beat tracking system beatroot," ''Journal
of New Music Research'', vol. 36, no. 1, pp. 39-51, 2007.

*'''Cemgil''' - beat accuracy is calculated using a Gaussian error function
with 40ms standard deviation.

A. T. Cemgil, B. Kappen, P. Desain, and H. Honing, "On tempo tracking:
Tempogram representation and Kalman filtering," ''Journal Of New Music
Research'', vol. 28, no. 4, pp. 259-273, 2001

*'''Goto''' - binary decision of correct or incorrect tracking based on
statistical properties of a beat error sequence.

M. Goto and Y. Muraoka, "Issues in evaluating beat tracking systems," in
''Working Notes of the IJCAI-97 Workshop on Issues in AI and Music -
Evaluation and Assessment'', 1997, pp. 9-16.

*'''PScore''' - McKinney's impulse train cross-correlation method as used in
2006.

M. F. McKinney, D. Moelants, M. E. P. Davies, and A. Klapuri,
"Evaluation of audio beat tracking and music tempo extraction
algorithms," ''Journal of New Music Research'', vol. 36, no. 1, pp. 1-16,
2007.

*'''CMLc''', '''CMLt''', '''AMLc''', '''AMLt''' - continuity-based evaluation methods based on
the longest continuously correctly tracked section.

S. Hainsworth, "Techniques for the automated analysis of musical audio,"
Ph.D. dissertation, Department of Engineering, Cambridge University,
2004.

A. P. Klapuri, A. Eronen, and J. Astola, "Analysis of the meter of
acoustic musical signals," IEEE Transactions on Audio, Speech and
Language Processing, vol. 14, no. 1, pp. 342-355, 2006.

*'''D''', '''Dg''' - information based criteria based on analysis of a beat error
histogram (note the results are measured in 'bits' and not percentages),
see the technical report for a description.

== Relevant Development Collections ==
You can find it here:

(data has been uploaded in both .tgz and .zip format)

''User: beattrack Password: b34trx''

https://www.music-ir.org/evaluation/MIREX/data/2006/beat/beattrack_train_2006.tgz OR

https://www.music-ir.org/evaluation/MIREX/data/2006/beat/beattrack_train_2006.zip

''User: tempo Password: t3mp0''

https://www.music-ir.org/evaluation/MIREX/data/2006/tempo/tempo_train_2006.tgz OR

https://www.music-ir.org/evaluation/MIREX/data/2006/tempo/tempo_train_2006.zip

== Time and hardware limits ==
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.

A hard limit of 12 hours will be imposed on analysis times. Submissions exceeding this limit may not receive a result.

== Potential Participants ==
name / email

Ju-Chiang Wang / ju-chiang.wang@bytedance.com

== Discussion ==
name / email

User:Yiting

2021-09-11T20:45:46Z

Djevans: Creating user page for new user.

I am an informatics phd student at UIUC.

User talk:Yiting

2021-09-11T20:45:46Z

Djevans: Welcome!

2021:Main Page

2021-09-11T20:31:57Z

Djevans: /* MIREX 2021 Deadline Dates */

==Welcome to MIREX 2021==

This is the main page for the 17th running of the Music Information Retrieval Evaluation eXchange (MIREX 2021). The International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL) at [https://ischool.illinois.edu School of Information Sciences], University of Illinois at Urbana-Champaign ([https://illinois.edu UIUC]) is the principal organizer of MIREX 2021.

The MIREX 2021 community will hold its annual meeting as part of [https://ismir2021.ismir.net/ The 21st International Society for Music Information Retrieval Conference], ISMIR 2021, which will be held in an online format, November 8–12, 2021.

J. Stephen Downie 
Director, IMIRSEL 

==Task Leadership Model==

Like previous years, we are prepared to improve the distribution of tasks for the upcoming MIREX 2021. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead a task, please complete the form [https://forms.gle/fAACmt9qtXxEf97G8 here]. Current information about task captains can be found on the [[2021:Task Captains]] page. Please direct any communication to the [https://lists.ischool.illinois.edu/lists/admin/evalfest EvalFest] mailing list.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

We really need leaders to help us!

==MIREX 2021 Possible Evaluation Tasks==
* [[2021:Audio Beat Tracking]]
* [[2021:Audio Chord Estimation]]
* [[2021:Audio Cover Song Identification]]
* [[2021:Audio Downbeat Estimation]]
* [[2021:Audio Fingerprinting]]
* [[2021:Audio Key Detection]]
* [[2021:Audio Melody Extraction]]
* [[2021:Audio Onset Detection]]
* [[2021:Audio Tag Classification]]
* [[2021:Audio Tempo Estimation]]
* [[2021:Lyrics Transcription (former: Automatic Lyrics-to-Audio Alignment)]](site under-construction)
* [[2021:Drum Transcription]]
* [[2021:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2021:Music Detection]]
* [[2021:Patterns for Prediction]] (offshoot of [[2017:Discovery of Repeated Themes & Sections]])
* [[2021:Query by Singing/Humming]]
* [[2021:Query by Tapping]]
* [[2021:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2021:Set List Identification]]
* [[2021:Structural Segmentation]]

==MIREX 2021 Deadline Dates==

Due to the extraneous circumstances brought on by COVID-19, we do not anticipate having all tasks wrapped up by the ISMIR conference. However, we still hope to meet with partial results and continue working on this after the conclusion of ISMIR.

==MIREX 2021 Submission Instructions==
* Be sure to read through the rest of this page
* Be sure to read though the task pages for which you are submitting
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2020 Submission Instructions]] including both the tutorial video and the text
* The MIREX 2021 Submission System is coming soon at: https://www.music-ir.org/mirex/sub/.

==MIREX 2021 Evaluation==

===Note to New Participants===
Please take the time to read the following review articles that explain the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

Downie, J. Stephen, Andreas F. Ehmann, Mert Bay and M. Cameron Jones. (2010). 
The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. 
''Advances in Music Information Retrieval'' Vol. 274, pp. 93-115 
Available at: [http://bit.ly/KpM5u5 http://bit.ly/KpM5u5]

===Runtime Limits===

We reserve the right to stop any process that exceeds runtime limits for each task. We will do our best to notify you in enough time to allow revisions, but this may not be possible in some cases. Please respect the published runtime limits.

===Note to All Participants===

Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted program(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2020 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same).
# present a poster at the MIREX 2020 poster session at ISMIR 2020, if there is a physical component to the conference.

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL currently supports some of the software/architecture dependencies for your submission, please contact [mailto:yunhao2@illinois.edu IMIRSEL team] as early as possible. Failing to notify the team might result in your submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2021==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2020 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2021 tasks. To subscribe to EvalFest, send a message to [mailto:lists@ischool.illinois.edu lists@ischool.illinois.edu] with the subject line “subscribe evalfest”

If you are participating in MIREX 2021, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2021 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2020 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
If you find that you cannot edit a MIREX wiki page, you will need to create a new account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2020 Wikis==
Content from MIREX 2005 - 2020 are available at:
'''[[2020:Main_Page|MIREX 2020]]'''
'''[[2019:Main_Page|MIREX 2019]]'''
'''[[2018:Main_Page|MIREX 2018]]'''
'''[[2017:Main_Page|MIREX 2017]]'''
'''[[2016:Main_Page|MIREX 2016]]'''
'''[[2015:Main_Page|MIREX 2015]]'''
'''[[2014:Main_Page|MIREX 2014]]'''
'''[[2013:Main_Page|MIREX 2013]]'''
'''[[2012:Main_Page|MIREX 2012]]'''
'''[[2011:Main_Page|MIREX 2011]]'''
'''[[2010:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

MIREX HOME

2021-09-11T20:31:40Z

Djevans: /* MIREX 2021 Deadline Dates */

==Welcome to MIREX 2021==

This is the main page for the 17th running of the Music Information Retrieval Evaluation eXchange (MIREX 2021). The International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL) at [https://ischool.illinois.edu School of Information Sciences], University of Illinois at Urbana-Champaign ([https://illinois.edu UIUC]) is the principal organizer of MIREX 2021.

The MIREX 2021 community will hold its annual meeting as part of [https://ismir2021.ismir.net/ The 21st International Society for Music Information Retrieval Conference], ISMIR 2021, which will be held in an online format, November 8–12, 2021.

J. Stephen Downie 
Director, IMIRSEL 

==Task Leadership Model==

Like previous years, we are prepared to improve the distribution of tasks for the upcoming MIREX 2021. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead a task, please complete the form [https://forms.gle/fAACmt9qtXxEf97G8 here]. Current information about task captains can be found on the [[2021:Task Captains]] page. Please direct any communication to the [https://lists.ischool.illinois.edu/lists/admin/evalfest EvalFest] mailing list.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

We really need leaders to help us!

==MIREX 2021 Possible Evaluation Tasks==
* [[2021:Audio Beat Tracking]]
* [[2021:Audio Chord Estimation]]
* [[2021:Audio Cover Song Identification]]
* [[2021:Audio Downbeat Estimation]]
* [[2021:Audio Fingerprinting]]
* [[2021:Audio Key Detection]]
* [[2021:Audio Melody Extraction]]
* [[2021:Audio Onset Detection]]
* [[2021:Audio Tag Classification]]
* [[2021:Audio Tempo Estimation]]
* [[2021:Lyrics Transcription (former: Automatic Lyrics-to-Audio Alignment)]](site under-construction)
* [[2021:Drum Transcription]]
* [[2021:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2021:Music Detection]]
* [[2021:Patterns for Prediction]] (offshoot of [[2017:Discovery of Repeated Themes & Sections]])
* [[2021:Query by Singing/Humming]]
* [[2021:Query by Tapping]]
* [[2021:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2021:Set List Identification]]
* [[2021:Structural Segmentation]]

==MIREX 2021 Deadline Dates==

Due to the extraneous circumstances brought on by COVID-19, we do not anticipate having all tasks wrapped up by the ISMIR conference. However, we still hope to meet with partial results and continue working on this after the conclusion of ISMIR.

==MIREX 2021 Submission Instructions==
* Be sure to read through the rest of this page
* Be sure to read though the task pages for which you are submitting
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2021 Submission Instructions]] including both the tutorial video and the text
* The MIREX 2021 Submission System is coming soon at: https://www.music-ir.org/mirex/sub/.

==MIREX 2021 Evaluation==

===Note to New Participants===
Please take the time to read the following review articles that explain the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

Downie, J. Stephen, Andreas F. Ehmann, Mert Bay and M. Cameron Jones. (2010). 
The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. 
''Advances in Music Information Retrieval'' Vol. 274, pp. 93-115 
Available at: [http://bit.ly/KpM5u5 http://bit.ly/KpM5u5]

===Runtime Limits===

We reserve the right to stop any process that exceeds runtime limits for each task. We will do our best to notify you in enough time to allow revisions, but this may not be possible in some cases. Please respect the published runtime limits.

===Note to All Participants===

Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted program(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2020 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same).
# present a poster at the MIREX 2020 poster session at ISMIR 2020, if there is a physical component to the conference.

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL currently supports some of the software/architecture dependencies for your submission, please contact [mailto:yunhao2@illinois.edu IMIRSEL team] as early as possible. Failing to notify the team might result in your submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2021==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2020 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2021 tasks. To subscribe to EvalFest, send a message to [mailto:lists@ischool.illinois.edu lists@ischool.illinois.edu] with the subject line “subscribe evalfest”

If you are participating in MIREX 2021, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2021 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2020 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
If you find that you cannot edit a MIREX wiki page, you will need to create a new account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2020 Wikis==
Content from MIREX 2005 - 2020 are available at:
'''[[2020:Main_Page|MIREX 2020]]'''
'''[[2019:Main_Page|MIREX 2019]]'''
'''[[2018:Main_Page|MIREX 2018]]'''
'''[[2017:Main_Page|MIREX 2017]]'''
'''[[2016:Main_Page|MIREX 2016]]'''
'''[[2015:Main_Page|MIREX 2015]]'''
'''[[2014:Main_Page|MIREX 2014]]'''
'''[[2013:Main_Page|MIREX 2013]]'''
'''[[2012:Main_Page|MIREX 2012]]'''
'''[[2011:Main_Page|MIREX 2011]]'''
'''[[2010:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

MIREX HOME

2021-09-10T21:31:50Z

Djevans:

==Welcome to MIREX 2021==

This is the main page for the 17th running of the Music Information Retrieval Evaluation eXchange (MIREX 2021). The International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL) at [https://ischool.illinois.edu School of Information Sciences], University of Illinois at Urbana-Champaign ([https://illinois.edu UIUC]) is the principal organizer of MIREX 2021.

The MIREX 2021 community will hold its annual meeting as part of [https://ismir2021.ismir.net/ The 21st International Society for Music Information Retrieval Conference], ISMIR 2021, which will be held in an online format, November 8–12, 2021.

J. Stephen Downie 
Director, IMIRSEL 

==Task Leadership Model==

Like previous years, we are prepared to improve the distribution of tasks for the upcoming MIREX 2021. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead a task, please complete the form [https://forms.gle/fAACmt9qtXxEf97G8 here]. Current information about task captains can be found on the [[2021:Task Captains]] page. Please direct any communication to the [https://lists.ischool.illinois.edu/lists/admin/evalfest EvalFest] mailing list.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

We really need leaders to help us!

==MIREX 2021 Possible Evaluation Tasks==
* [[2021:Audio Beat Tracking]]
* [[2021:Audio Chord Estimation]]
* [[2021:Audio Cover Song Identification]]
* [[2021:Audio Downbeat Estimation]]
* [[2021:Audio Fingerprinting]]
* [[2021:Audio Key Detection]]
* [[2021:Audio Melody Extraction]]
* [[2021:Audio Onset Detection]]
* [[2021:Audio Tag Classification]]
* [[2021:Audio Tempo Estimation]]
* [[2021:Lyrics Transcription (former: Automatic Lyrics-to-Audio Alignment)]](site under-construction)
* [[2021:Drum Transcription]]
* [[2021:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2021:Music Detection]]
* [[2021:Patterns for Prediction]] (offshoot of [[2017:Discovery of Repeated Themes & Sections]])
* [[2021:Query by Singing/Humming]]
* [[2021:Query by Tapping]]
* [[2021:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2021:Set List Identification]]
* [[2021:Structural Segmentation]]

==MIREX 2021 Deadline Dates==
To be announced.

==MIREX 2021 Submission Instructions==
* Be sure to read through the rest of this page
* Be sure to read though the task pages for which you are submitting
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2021 Submission Instructions]] including both the tutorial video and the text
* The MIREX 2021 Submission System is coming soon at: https://www.music-ir.org/mirex/sub/.

==MIREX 2021 Evaluation==

===Note to New Participants===
Please take the time to read the following review articles that explain the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

Downie, J. Stephen, Andreas F. Ehmann, Mert Bay and M. Cameron Jones. (2010). 
The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. 
''Advances in Music Information Retrieval'' Vol. 274, pp. 93-115 
Available at: [http://bit.ly/KpM5u5 http://bit.ly/KpM5u5]

===Runtime Limits===

We reserve the right to stop any process that exceeds runtime limits for each task. We will do our best to notify you in enough time to allow revisions, but this may not be possible in some cases. Please respect the published runtime limits.

===Note to All Participants===

Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted program(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2020 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same).
# present a poster at the MIREX 2020 poster session at ISMIR 2020, if there is a physical component to the conference.

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL currently supports some of the software/architecture dependencies for your submission, please contact [mailto:yunhao2@illinois.edu IMIRSEL team] as early as possible. Failing to notify the team might result in your submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2021==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2020 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2021 tasks. To subscribe to EvalFest, send a message to [mailto:lists@ischool.illinois.edu lists@ischool.illinois.edu] with the subject line “subscribe evalfest”

If you are participating in MIREX 2021, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2021 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2020 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
If you find that you cannot edit a MIREX wiki page, you will need to create a new account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2020 Wikis==
Content from MIREX 2005 - 2020 are available at:
'''[[2020:Main_Page|MIREX 2020]]'''
'''[[2019:Main_Page|MIREX 2019]]'''
'''[[2018:Main_Page|MIREX 2018]]'''
'''[[2017:Main_Page|MIREX 2017]]'''
'''[[2016:Main_Page|MIREX 2016]]'''
'''[[2015:Main_Page|MIREX 2015]]'''
'''[[2014:Main_Page|MIREX 2014]]'''
'''[[2013:Main_Page|MIREX 2013]]'''
'''[[2012:Main_Page|MIREX 2012]]'''
'''[[2011:Main_Page|MIREX 2011]]'''
'''[[2010:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

MIREX 2021 Submission Instructions

2021-09-10T21:31:11Z

Djevans: Created page with "==Some Reminders== * Be sure to read through the rest of this page * Be sure to read through the 2021 MIREX Home page * Be sure to read through the task pa..."

==Some Reminders==
* Be sure to read through the rest of this page
* Be sure to read through the [[2021:Main_Page| 2021 MIREX Home]] page
* Be sure to read through the task pages for which you are submitting
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]

==Begin with the Video Tutorial==
Go watch the [https://www.music-ir.org//mirex/2010/submission_tutorial/ MIREX 2010 Submission System Video Tutorial]

==Basic Steps==

# Tell us about yourself by creating an identity profile. If you are participating under multiple affiliations, repeat this step for each affiliation.
# Create a submission record. You'll need to add all your contributors. This is easiest if they have also registered and completed step 1, but you can create profiles for them when you create your submission.
# Upload your submission via SFTP to the dropbox. Specific instructions are given after you complete your submission.
# Upload your abstract via the webform.

==Very Important Things to Note==

# NOTA BENE: We are REQUIRING that EACH person involved in a MIREX 2021 submission MUST create an identity for themselves on the submission system. Identities are important to us as they help us better manage the submissions. Even if a colleague of yours is going to do the actual submitting, you still need to create an identity for yourself in the system.
# When you create your personal identity in the system, review your input carefully for errors! Once your personal identity is created and the "submit" button is pressed, it is not possible for you to edit your identity information.
# If you are submitting on behalf of a team you will need to make sure that the identity for each team member is associated with your submission. Your first job is to find out if they have already created identities in the system by using the search tool. If they have, simply click on the identity to add them.
# If you cannot find an identity for one or more of your colleagues, the best way to proceed is get them to create an identity for themselves on the system. This way, they are responsible for the accuracy of their information.
# If your colleague, for some reason, cannot create an identity for themselves, you will need to create an identity for them. Do your best to create as accurate an identity for them as possible.
# If you plan to submit more than one algorithm or algorithm variant to a given task, EACH algorithm or variant needs its own complete submission to be made including the README and binary bundle upload. Each package will be given its own unique identifier. Tell us in the README the priority of a given algorithm in case we have to limit a task to only one or two algorithms/variants per submitter/team.

==Getting Help==
If things do not work or if you make a major mistake or if you are simply confused, please contact the MIREX team at djevans4 [at] illinois.edu.

==Participant Identity Information Fields==
(* = required field)
*First name*:
*Last name*:
*Organization*:
*Department:
*Unit/Lab:
*URL*:
*Title*:
*From (year)*: To:
*Email:
*Street Address:
*Street Address 2:
*Street Address 3:
*City:
*State, Region:
*Postal Code:
*Country:

==Extended Abstract Details==
The extended abstracts provide the outside world with a general understanding of what each submission is trying to accomplish. The extended abstracts need NOT be cutting edge/never-before-published materials. The extended abstracts will be revised by the authors after the data has been collected (to allow for commentary on results data); however, we at MIREX still need the first-pass drafts at submission time to help us understand what is happening in the submission. Like last year we will post the final versions of the extended abstracts as part of the MIREX 2021 results page.

The MIREX 2021 extended abstracts:
# Are two to four pages long.
# Must conform to the guidelines in the following templates: [https://www.music-ir.org/mirex/templates/2010/MIREX2010_tex_template.zip LaTeX template] [https://www.music-ir.org/mirex/templates/2010/MIREX2010_doc_template.zip Word template]
# Must be submitted in PDF format.
# Should include, if exists, references to other publications about your work (yes, self-reference is encouraged!)
# Should have the same general look and feel as these examples from last year:
## https://www.music-ir.org/mirex/abstracts/2019/AR2.pdf
## https://www.music-ir.org/mirex/abstracts/2019/KN1.pdf

The MIREX 2021 Submission System can be found at: https://www.music-ir.org/mirex/sub/ .

2021:MIREX2020 Results

2021-09-10T21:25:03Z

Djevans: Created page with "==Results by Task (More results are coming) =="

==Results by Task (More results are coming) ==

MediaWiki:Sidebar

2021-09-10T21:24:24Z

Djevans:

2021:Main Page

2021-09-10T21:15:31Z

Djevans: Created page with "==Welcome to MIREX 2021== This is the main page for the 17th running of the Music Information Retrieval Evaluation eXchange (MIREX 2021). The International Music Information..."

==Welcome to MIREX 2021==

This is the main page for the 17th running of the Music Information Retrieval Evaluation eXchange (MIREX 2021). The International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL) at [https://ischool.illinois.edu School of Information Sciences], University of Illinois at Urbana-Champaign ([https://illinois.edu UIUC]) is the principal organizer of MIREX 2021.

The MIREX 2021 community will hold its annual meeting as part of [https://ismir2021.ismir.net/ The 21st International Society for Music Information Retrieval Conference], ISMIR 2021, which will be held in an online format, November 8–12, 2021.

J. Stephen Downie 
Director, IMIRSEL 

==Task Leadership Model==

Like previous years, we are prepared to improve the distribution of tasks for the upcoming MIREX 2021. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead a task, please complete the form [https://forms.gle/fAACmt9qtXxEf97G8 here]. Current information about task captains can be found on the [[2021:Task Captains]] page. Please direct any communication to the [https://lists.ischool.illinois.edu/lists/admin/evalfest EvalFest] mailing list.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

We really need leaders to help us!

==MIREX 2021 Possible Evaluation Tasks==
* [[2021:Audio Beat Tracking]]
* [[2021:Audio Chord Estimation]]
* [[2021:Audio Cover Song Identification]]
* [[2021:Audio Downbeat Estimation]]
* [[2021:Audio Fingerprinting]]
* [[2021:Audio Key Detection]]
* [[2021:Audio Melody Extraction]]
* [[2021:Audio Onset Detection]]
* [[2021:Audio Tag Classification]]
* [[2021:Audio Tempo Estimation]]
* [[2021:Lyrics Transcription (former: Automatic Lyrics-to-Audio Alignment)]](site under-construction)
* [[2021:Drum Transcription]]
* [[2021:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2021:Music Detection]]
* [[2021:Patterns for Prediction]] (offshoot of [[2017:Discovery of Repeated Themes & Sections]])
* [[2021:Query by Singing/Humming]]
* [[2021:Query by Tapping]]
* [[2021:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2021:Set List Identification]]
* [[2021:Structural Segmentation]]

==MIREX 2021 Deadline Dates==
To be announced.

==MIREX 2021 Submission Instructions==
* Be sure to read through the rest of this page
* Be sure to read though the task pages for which you are submitting
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2020 Submission Instructions]] including both the tutorial video and the text
* The MIREX 2021 Submission System is coming soon at: https://www.music-ir.org/mirex/sub/.

==MIREX 2021 Evaluation==

===Note to New Participants===
Please take the time to read the following review articles that explain the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

Downie, J. Stephen, Andreas F. Ehmann, Mert Bay and M. Cameron Jones. (2010). 
The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. 
''Advances in Music Information Retrieval'' Vol. 274, pp. 93-115 
Available at: [http://bit.ly/KpM5u5 http://bit.ly/KpM5u5]

===Runtime Limits===

We reserve the right to stop any process that exceeds runtime limits for each task. We will do our best to notify you in enough time to allow revisions, but this may not be possible in some cases. Please respect the published runtime limits.

===Note to All Participants===

Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted program(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2020 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same).
# present a poster at the MIREX 2020 poster session at ISMIR 2020, if there is a physical component to the conference.

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL currently supports some of the software/architecture dependencies for your submission, please contact [mailto:yunhao2@illinois.edu IMIRSEL team] as early as possible. Failing to notify the team might result in your submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2021==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2020 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2021 tasks. To subscribe to EvalFest, send a message to [mailto:lists@ischool.illinois.edu lists@ischool.illinois.edu] with the subject line “subscribe evalfest”

If you are participating in MIREX 2021, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2021 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2020 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
If you find that you cannot edit a MIREX wiki page, you will need to create a new account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2020 Wikis==
Content from MIREX 2005 - 2020 are available at:
'''[[2020:Main_Page|MIREX 2020]]'''
'''[[2019:Main_Page|MIREX 2019]]'''
'''[[2018:Main_Page|MIREX 2018]]'''
'''[[2017:Main_Page|MIREX 2017]]'''
'''[[2016:Main_Page|MIREX 2016]]'''
'''[[2015:Main_Page|MIREX 2015]]'''
'''[[2014:Main_Page|MIREX 2014]]'''
'''[[2013:Main_Page|MIREX 2013]]'''
'''[[2012:Main_Page|MIREX 2012]]'''
'''[[2011:Main_Page|MIREX 2011]]'''
'''[[2010:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

MIREX HOME

2021-09-10T21:15:02Z

Djevans:

==Welcome to MIREX 2021==

This is the main page for the 17th running of the Music Information Retrieval Evaluation eXchange (MIREX 2021). The International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL) at [https://ischool.illinois.edu School of Information Sciences], University of Illinois at Urbana-Champaign ([https://illinois.edu UIUC]) is the principal organizer of MIREX 2021.

The MIREX 2021 community will hold its annual meeting as part of [https://ismir2021.ismir.net/ The 21st International Society for Music Information Retrieval Conference], ISMIR 2021, which will be held in an online format, November 8–12, 2021.

J. Stephen Downie 
Director, IMIRSEL 

==Task Leadership Model==

Like previous years, we are prepared to improve the distribution of tasks for the upcoming MIREX 2021. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead a task, please complete the form [https://forms.gle/fAACmt9qtXxEf97G8 here]. Current information about task captains can be found on the [[2021:Task Captains]] page. Please direct any communication to the [https://lists.ischool.illinois.edu/lists/admin/evalfest EvalFest] mailing list.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

We really need leaders to help us!

==MIREX 2021 Possible Evaluation Tasks==
* [[2021:Audio Beat Tracking]]
* [[2021:Audio Chord Estimation]]
* [[2021:Audio Cover Song Identification]]
* [[2021:Audio Downbeat Estimation]]
* [[2021:Audio Fingerprinting]]
* [[2021:Audio Key Detection]]
* [[2021:Audio Melody Extraction]]
* [[2021:Audio Onset Detection]]
* [[2021:Audio Tag Classification]]
* [[2021:Audio Tempo Estimation]]
* [[2021:Lyrics Transcription (former: Automatic Lyrics-to-Audio Alignment)]](site under-construction)
* [[2021:Drum Transcription]]
* [[2021:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2021:Music Detection]]
* [[2021:Patterns for Prediction]] (offshoot of [[2017:Discovery of Repeated Themes & Sections]])
* [[2021:Query by Singing/Humming]]
* [[2021:Query by Tapping]]
* [[2021:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2021:Set List Identification]]
* [[2021:Structural Segmentation]]

==MIREX 2021 Deadline Dates==
To be announced.

==MIREX 2021 Submission Instructions==
* Be sure to read through the rest of this page
* Be sure to read though the task pages for which you are submitting
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2020 Submission Instructions]] including both the tutorial video and the text
* The MIREX 2021 Submission System is coming soon at: https://www.music-ir.org/mirex/sub/.

==MIREX 2021 Evaluation==

===Note to New Participants===
Please take the time to read the following review articles that explain the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

Downie, J. Stephen, Andreas F. Ehmann, Mert Bay and M. Cameron Jones. (2010). 
The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. 
''Advances in Music Information Retrieval'' Vol. 274, pp. 93-115 
Available at: [http://bit.ly/KpM5u5 http://bit.ly/KpM5u5]

===Runtime Limits===

We reserve the right to stop any process that exceeds runtime limits for each task. We will do our best to notify you in enough time to allow revisions, but this may not be possible in some cases. Please respect the published runtime limits.

===Note to All Participants===

Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted program(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2020 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same).
# present a poster at the MIREX 2020 poster session at ISMIR 2020, if there is a physical component to the conference.

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL currently supports some of the software/architecture dependencies for your submission, please contact [mailto:yunhao2@illinois.edu IMIRSEL team] as early as possible. Failing to notify the team might result in your submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2021==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2020 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2021 tasks. To subscribe to EvalFest, send a message to [mailto:lists@ischool.illinois.edu lists@ischool.illinois.edu] with the subject line “subscribe evalfest”

If you are participating in MIREX 2021, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2021 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2020 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
If you find that you cannot edit a MIREX wiki page, you will need to create a new account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2020 Wikis==
Content from MIREX 2005 - 2020 are available at:
'''[[2020:Main_Page|MIREX 2020]]'''
'''[[2019:Main_Page|MIREX 2019]]'''
'''[[2018:Main_Page|MIREX 2018]]'''
'''[[2017:Main_Page|MIREX 2017]]'''
'''[[2016:Main_Page|MIREX 2016]]'''
'''[[2015:Main_Page|MIREX 2015]]'''
'''[[2014:Main_Page|MIREX 2014]]'''
'''[[2013:Main_Page|MIREX 2013]]'''
'''[[2012:Main_Page|MIREX 2012]]'''
'''[[2011:Main_Page|MIREX 2011]]'''
'''[[2010:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

2021:Task Captains

2021-09-10T21:03:33Z

Djevans:

Like MIREX 2019, we are prepared to improve the distribution of tasks for the upcoming MIREX 2020. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead one or more tasks, please complete the form [https://goo.gl/forms/0w5nowHkHzYxHl4l1 here]. Please direct any communication to the [https://lists.ischool.illinois.edu/lists/admin/evalfest EvalFest] mailing list.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

{| class="wikitable" style="margin-left: 20px"
!ID !! Task !! Captain(s)
|-
|act
|[[2021:Audio Classification (Train/Test) Tasks]]
|
|-
|abt
|[[2021:Audio Beat Tracking]]
|
|-
|ace
|[[2021:Audio Chord Estimation]]
|
|-
|acs
|[[2021:Audio Cover Song Identification]]
|
|-
|ade
|[[2021:Audio Downbeat Estimation]]
|
|-
|afp
|[[2021:Audio Fingerprinting]]
|
|-
|akd
|[[2021:Audio Key Detection]]
|
|-
|ame
|[[2021:Audio Melody Extraction]]
|
|-
|aod
|[[2021:Audio Onset Detection]]
|
|-
|atc
|[[2021:Audio Tag Classification]]
|
|-
|ate
|[[2021:Audio Tempo Estimation]]
|
|-
|ala
|[[2021:Automatic Lyrics-to-Audio Alignment]]
|
|-
|dt
|[[2021:Drum Transcription]]
|
|-
|mf0
|[[2021:Multiple Fundamental Frequency Estimation & Tracking]]
|
|-
|mdt
|[[2021:Music Detection]]
|
|-
|pfp
|[[2021:Patterns for Prediction]]
|
|-
|qbsh
|[[2021:Query by Singing/Humming]]
|
|-
|qbt
|[[2021:Query by Tapping]]
|
|-
|rtas
|[[2021:Real-time Audio to Score Alignment (a.k.a Score Following)]]
|
|-
|sli
|[[2021:Set List Identification]]
|
|-
|stse
|[[2021:Structural Segmentation]]
|
|}

2021:Task Captains

2021-09-10T20:15:28Z

Djevans: Created page with "Like MIREX 2019, we are prepared to improve the distribution of tasks for the upcoming MIREX 2020. To do so, we really need leaders to help us organize and run each task. To..."

Like MIREX 2019, we are prepared to improve the distribution of tasks for the upcoming MIREX 2020. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead one or more tasks, please complete the form [https://goo.gl/forms/0w5nowHkHzYxHl4l1 here]. Please direct any communication to the [https://lists.ischool.illinois.edu/lists/admin/evalfest EvalFest] mailing list.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

{| class="wikitable" style="margin-left: 20px"
!ID !! Task !! Captain(s)
|-
|act
|[[2021:Audio Classification (Train/Test) Tasks]]
|
|-
|abt
|[[2021:Audio Beat Tracking]]
|
|-
|ace
|[[2021:Audio Chord Estimation]]
|
|-
|acs
|[[2021:Audio Cover Song Identification]]
|
|-
|ade
|[[2021:Audio Downbeat Estimation]]
|
|-
|afp
|[[2021:Audio Fingerprinting]]
|
|-
|akd
|[[2021:Audio Key Detection]]
|
|-
|ame
|[[2021:Audio Melody Extraction]]
|
|-
|aod
|[[2021:Audio Onset Detection]]
|
|-
|atc
|[[2021:Audio Tag Classification]]
|
|-
|ate
|[[2021:Audio Tempo Estimation]]
|
|-
|ala
|[[2021:Automatic Lyrics-to-Audio Alignment]]
|
|-
|dt
|[[2021:Drum Transcription]]
|
|-
|mf0
|[[2021:Multiple Fundamental Frequency Estimation & Tracking]]
|
|-
|mdt
|[[2021:Music Detection]]
|
|-
|pfp
|[[2021:Patterns for Prediction]]
|
|-
|sli
|[[2021:Set List Identification]]
|
|-
|ame
|[[2021:Audio Melody Extraction]]
|
|-
|mscd
|[[2021:Music Detection]]
|
|-
|qbsh
|[[2021:Query by Singing/Humming]]
|
|-
|qbt
|[[2021:Query by Tapping]]
|
|-
|rtas
|[[2021:Real-time Audio to Score Alignment (a.k.a Score Following)]]
|
|-
|sli
|[[2021:Set List Identification]]
|
|-
|stse
|[[2021:Structural Segmentation]]
|
|}

2021:Structural Segmentation

2021-09-10T20:03:02Z

Djevans: Created page with "== Description == The aim of the MIREX structural segmentation evaluation is to identify the key structural sections in musical audio. The segment structure (or form) is one..."

== Description ==

The aim of the MIREX structural segmentation evaluation is to identify the key structural sections in musical audio. The segment structure (or form) is one of the most important musical parameters. It is furthermore special because musical structure -- especially in popular music genres (e.g. verse, chorus, etc.) -- is accessible to everybody: it needs no particular musical knowledge. This task was first run in 2009.

== Data ==

=== Collections ===
* The MIREX 2009 Collection: 297 pieces, most of it derived from the work of the Beatles.

* MIREX 2010 RWC collection. 100 pieces of popular music. There are two ground truths. The first is the one originally included with the RWC dataset. The explanation of the second set of annotations can be found at http://hal.inria.fr/docs/00/47/34/79/PDF/PI-1948.pdf. The second set of annotations contains no labels for segments, but rather provides an annotation of segment boundaries.

* MIREX 2012 dataset. The new data set contains over 1,000 annotated pieces covering a range of musical styles. The majority of the pieces have been annotated by two independent annotators.

=== Audio Formats ===

* CD-quality (PCM, 16-bit, 44100 Hz)
* single channel (mono)

== Submission Format ==

Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.

=== Input Data ===
Participating algorithms will have to read audio in the following format:

* Sample rate: 44.1 KHz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

=== Output Data ===

The structural segmentation algorithms will return the segmentation in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.

=== Output File Format (Structural Segmentation) ===

The Structural Segmentation output file format is a tab-delimited ASCII text format. This is the same as Chris Harte's chord labelling files (.lab), and so is the same format as the ground truth as well. Onset and offset times are given in seconds, and the labels are simply letters: 'A', 'B', ... with segments referring to the same structural element having the same label.

Three column text file of the format

<onset_time(sec)>\t<offset_time(sec)>\t<label>\n
<onset_time(sec)>\t<offset_time(sec)>\t<label>\n
...

where \t denotes a tab, \n denotes the end of line. The < and > characters are not included. An example output file would look something like:

0.000 5.223 A
5.223 15.101 B
15.101 20.334 A

=== Algorithm Calling Format ===

The submitted algorithm must take as arguments a SINGLE .wav file to perform the structural segmentation on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:

foobar %input %output
foobar -i %input -o %output

Moreover, if your submission takes additional parameters, foobar could be called like:

foobar .1 %input %output
foobar -param1 .1 -i %input -o %output

If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example:

foobar('%input','%output')
foobar(.1,'%input','%output')

=== README File ===

A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.

For instance, to test the program foobar with a specific value for parameter param1, the README file would look like:

foobar -param1 .1 -i %input -o %output

For a submission using MATLAB, the README file could look like:

matlab -r "foobar(.1,'%input','%output');quit;"

== Evaluation Procedures ==
At the last ISMIR conference [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich] proposed a measure for segmentation evaluation. Because of the complexity of the structural segmentation task definition, several different evaluation measures will be employed to address different aspects. It should be noted that none of the evaluation measures cares about the true labels of the sections: they only denote the clustering. This means that it does not matter if the systems produce true labels such as "chorus" and "verse", or arbitrary labels such as "A" and "B".

=== Boundary retrieval ===
'''Hit rate''' Found segment boundaries are accepted to be correct if they are within 0.5s ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007]) or 3s ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008]) from a border in the ground truth. Based on the matched hits, ''boundary retrieval recall rate'', ''boundary retrieval precision rate'', and ''boundary retrieval F-measure'' are be calculated.

'''Median deviation''' Two median deviation measure between boundaries in the result and ground truth are calculated: ''median true-to-guess'' is the median time from boundaries in ground truth to the closest boundaries in the result, and ''median guess-to-true'' is similarly the median time from boundaries in the result to boundaries in ground truth. ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007])

=== Frame clustering ===
Both the result and the ground truth are handled in short frames (e.g., beat or fixed 100ms). All frame pairs in a structure description are handled. The pairs in which both frames are assigned to the same cluster (i.e., have the same label) form the sets <math>P_E</math> (for the system result) and <math>P_A</math> (for the ground truth). The ''pairwise precision rate'' can be calculated by <math>P = \frac{|P_E \cap P_A|}{|P_E|}</math>, ''pairwise recall rate'' by <math>R = \frac{|P_E \cap P_A|}{|P_A|}</math>, and ''pairwise F-measure'' by <math>F=\frac{2 P R}{P + R}</math>. ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008])

=== Normalised conditional entropies ===
Over- and under segmentation based evaluation measures proposed in [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich ISMIR2008].
Structure descriptions are represented as frame sequences with the associated cluster information (similar to the Frame clustering measure). Confusion matrix between the labels in ground truth and the result is calculated. The matrix C is of size |L_A| * |L_E|, i.e., number of unique labels in the ground truth times number of unique labels in the result. From the confusion matrix, the joint distribution is calculated by normalising the values with the total number of frames F:

<math>p_{i,j} = C_{i,j} / F</math>

Similarly, the two marginals are calculated:

<math>p_i^a = \sum_{j=1}^{|L_E|} C{i,j}/F</math>, and

<math>p_j^e = \sum_{i=1}^{|L_A|} C{i,j}/F</math>

Conditional distributions:

<math>p_{i,j}^{a|e} = C_{i,j} / \sum_{i=1}^{|L_A|} C{i,j}</math>, and

<math>p_{i,j}^{e|a} = C_{i,j} / \sum_{j=1}^{|L_E|} C{i,j}</math>

The conditional entropies will then be

<math>H(E|A) = - \sum_{i=1}^{|L_A|} p_i^a \sum_{j=1}^{|L_E|} p_{i,j}^{e|a} \log_2(p_{i,j}^{e|a})</math>, and

<math>H(A|E) = - \sum_{j=1}^{|L_E|} p_j^e \sum_{i=1}^{|L_A|} p_{i,j}^{a|e} \log_2(p_{i,j}^{a|e})</math>

The final evaluation measures will then be the oversegmentation score

<math>S_O = 1 - \frac{H(E|A)}{\log_2(|L_E|)}</math> , and the undersegmentation score

<math>S_U = 1 - \frac{H(A|E)}{\log_2(|L_A|)}</math>

== Relevant Development Collections ==
*Jouni Paulus's [http://www.cs.tut.fi/sgn/arg/paulus/structure.html structure analysis page] links to a corpus of 177 Beatles songs ([http://www.cs.tut.fi/sgn/arg/paulus/beatles_sections_TUT.zip zip file]). The Beatles annotations are not a part of the TUTstructure07 dataset. That dataset contains 557 songs, a list of which is available [http://www.cs.tut.fi/sgn/arg/paulus/TUTstructure07_files.html here].

*Ewald Peiszer's [http://www.ifs.tuwien.ac.at/mir/audiosegmentation.html thesis page] links to a portion of the corpus he used: 43 non-Beatles pop songs (including 10 J-pop songs) ([http://www.ifs.tuwien.ac.at/mir/audiosegmentation/dl/ep_groundtruth_excl_Paulus.zip zip file]).

These public corpora give a combined 220 songs.

== Time and hardware limits ==
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.

A hard limit of 24 hours will be imposed on analysis times. Submissions exceeding this limit may not receive a result.

== Potential Participants ==

name / email

== Discussion ==

2021:Set List Identification

2021-09-10T20:02:14Z

Djevans: Created page with "__TOC__ ==Description== This task requires that algorithm identify the '''set list''' (See [http://en.wikipedia.org/wiki/Set_list Set list]). Set list is the song sequence i..."

__TOC__

==Description==

This task requires that algorithm identify the '''set list''' (See [http://en.wikipedia.org/wiki/Set_list Set list]). Set list is the song sequence in a live concert. It shows the order of songs will be performed in a live concert.

Recently, more and more full-length live concert videos have become available on website (e.g. [https://www.youtube.com/ Youtube]). Most of them are lacking sufficient information to describe itself, such as the set list, and start/end time of each song. In this task, we collect the audio of live concerts and studio songs, applying music information retrieval techniques to answer this question -- what songs had been sung in this concert and when are the songs start and end.

For the first step of this task, we assume that '''artist is known'''. In the live concert, '''the performers play their studio songs only''', however the ultimate goal is granted a full-length live concert audio and studio song database, we still can find out the set list and the start/end time of each song.

here are two sub tasks in this task:

===Sub task 1: Song sequence identification===
*To identify the order of songs which be performed in a live concert.

In this sub task, the participants known the the artist and artist's studio song collection. Assigning a live concert audio and studio songs collection of a specific artist, all songs in live concert are included in studio songs collection, to identify the order of songs in this live concert.

===Sub task 2: Time boundary identification===
*To identify the start/end time of each song in song sequence

In this sub task, the participants known the artist, artist's studio song collection and the '''song sequence'''. Assigning a live concert audio, song sequence and studio songs collection of a specific artist, all songs in live concert are included in studio songs collection, to identify start time and end time of each song in the live concert.

== Data ==
To satisfy our assessment, we pre-process all audio -- '''remove the "out of artist song" form live concert audio''' for following our assumption. (See the [https://www.music-ir.org/mirex/wiki/2016:Set_List_Identification#Description description])

We provide two set for this task,participating algorithms will have to read audio in the following format.

* Sample rate: 22050 Hz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

===Developing set===
This set contain 3 artists and 7 live concerts, the following information would be release ([https://www.dropbox.com/sh/t83ogdrxi0f050n/AABb11MCcQUokqSjOsqhArOFa?dl=0 Dropbox])
* artist
* live concert name and links
* studio collection list
* start/end time tags

We extract features for the convenience of participants, the links is the tool we used. ([https://www.dropbox.com/s/bote36k8pkmt2f8/MIREX_2015_Setlist_ID_Developing_set_chroma_fea.rar?dl=0 Dropbox])
*chroma (CRP features [http://resources.mpi-inf.mpg.de/MIR/chromatoolbox/ Chroma Toolbox])

Collection statistics:
* 3 artists
* 7 live concerts
* 279 tracks

=== Testing set ===
This set contain 7 artists and 13 live concerts, no information would be release.

Collection statistics:
* 7 artists
* 13 live concerts
* 873 tracks

== Evaluation ==

For two tasks, the evaluation metrics were different.

=== Sub task 1===

* Edit distance (see [http://en.wikipedia.org/wiki/Edit_distance Edit distance])

We evaluated the two sequence (ground truth and your result) by edit distance, there three errors included
* insertion error <math>I</math>
* substitution error <math>S</math>
* deletion error <math>D</math>

Edit Distance: <big><math>ED = I+S+D </math> </big>

Percent Correct: <big><math>Corr = \frac{N-D-S}{N}</math></big>

Percent Accuracy: <big><math> Acc = \frac{N-D-S-I}{N}</math></big>

=== Sub task 2===

* average time boundary

We will evaluate two time boundaries as follow: average start time boundary and average end time boundary. The evaluation function is described below:

* Set list contains '''<math>N</math>''' songs

''' Ground truth: '''

* Start time of song '''<math>i</math>'''：<math>sBD_{GT_i}</math>

* End time of song '''<math>i</math>'''：<math>eBD_{GT_i}</math>

''' Identification result: '''

* Start time of song '''<math>i</math>'''：<math>sBD_{ID_i}</math>

* End time of song '''<math>i</math>'''：<math>eBD_{ID_i}</math>

<math> AVGsBD =\frac{\sum_{i=1}^N |sBD_{GT_i} - sBD_{ID_i}|}{N} </math>,

<math> AVGeBD =\frac{\sum_{i=1}^N |eBD_{GT_i} - eBD_{ID_i}|}{N} </math>,

=== Runtime performance ===
In addition computation times for feature extraction and training/classification will be measured.

== Submission Format ==
* '''\n''' is end of line

Submission to this task will have to conform to a specified format detailed below.
=== Implementation details ===
we recommend your submission folder construction as follow:
/root_folder/... all the code you submitted
/root_folder/extract_feature/... all feature your extracted
/root_folder/output/... the folder to save results

=== Sub task 1 ===

Two inputs : live file list and studio song file list

One output: song ID sequence

==== Input file ====
The input for studio songs list file format will be of the form:

/path/to/artist_1/studio/song/001.wav\n 1st
/path/to/artist_1/studio/song/002.wav\n 2nd
/path/to/artist_1/studio/song/003.wav\n 3rd
...

The input for live concert list file format will be of the form:

/path/to/artist_1/live/concert/001.wav\n

==== Output file ====
The output is a list file (song ID sequence), '''the song ID is the order of input list file''', not the file name of *.wav file.

3\n <-- 003.wav is the first song of set list for your identification result
17\n
59\n
...

=== Sub task 2 ===

Three inputs : song ID sequence list, live file list and studio song file list

One output: time label of song list

==== Input file ====

The input is a list of song ID (song ID sequence), '''the song ID is the order of studio songs list file'''.

Your system should read the *.wav file according that order and find the time boundary of the song.

3\n
17\n
59\n
...

The input for live concert list file format will be of the form:

/path/to/artist_1/live/concert/001.wav\n

The input for studio songs list file format will be of the form:

/path/to/artist_1/studio/song/001.wav\n 1st
/path/to/artist_1/studio/song/002.wav\n 2nd
/path/to/artist_1/studio/song/003.wav\n 3rd
...

==== Output file ====

The output for studio songs time boundary list file format will be of the form:
* please round the time boundary to millisecond
* '''\t''' is tab space
Start time end time
hours.minutes.seconds.milliseconds \t hours.minutes.seconds.milliseconds\n (for input input sond ID:3)
hours.minutes.seconds.milliseconds \t hours.minutes.seconds.milliseconds\n (for input input sond ID:17)
hours.minutes.seconds.milliseconds \t hours.minutes.seconds.milliseconds\n (for input input sond ID:59)
...

Examples:
0.7.23.521 0.13.24.512
0.14.3.021 0.19.53.38
0.20.9.893 0.27.15.987
...
...
0.56.22.433 1.1.46.593
1.3.51.146 1.9.21.138
...

=== Packaging submissions ===

All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guarenteed).

All submissions should include a README file including the following the information:

* Which task you want to participate (sub task1, sub task2 or all)
* Command line calling format for all executables and an example formatted set of commands
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Any required environments (and versions), e.g. python, java, bash, matlab.

== Time and hardware limits ==
Due to the potentially high number of particpants in this and other audio tasks, hard limits on the runtime of submissions are specified.

A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.

== Potential Participants ==
name / email

2021:Real-time Audio to Score Alignment (a.k.a Score Following)

2021-09-10T20:01:07Z

Djevans: Created page with "''Real-time Audio to Score Alignment'', also known as ''Score Following'' == Description == Score Following is the real-time alignment of an incoming music signal to the musi..."

''Real-time Audio to Score Alignment'', also known as ''Score Following''

== Description ==
Score Following is the real-time alignment of an incoming music signal to the music score. The music signal can be symbolic (MIDI) or audio, but we will concentrate here on audio following, unless there are some candidates who'd want their symbolic followers to be evaluated and can propose reference data.

This page describes a proposal for evaluation of score following systems.

Submissions will be required to estimate alignment precision according to the indexed times. In order for your system to participate, please specify the type of alignment (monophonic, polyphonic), type of training and realtime performance, also separated into two domains (upon enough submissions) for symbolic and audio systems. Note that we also do accept systems that don't run in real-time in practice, as soon as their algorithm is on-line, i.e. without making use of global knowledge of the input.

=== Task specific mailing list ===
In the past we have use a specific mailing list for the discussion of this task and related tasks. This year, however, we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.

== Data ==
46 recordings and their corresponding MIDI representations of the score will be used in the evaluation. These 46 excerpts were extracted from 4 distinct musical pieces.
Recordings are in 44.1Khz 16bit wav format. The reference scores are in MIDI format.

Zhiyao Duan and Prof. Bryan Pardo contributed another polyphonic dataset. This dataset consists of 10 pieces of four-part J.S. Bach chorales. The audio file was performed by a quartet of instruments: violin, clarinet, saxophone and bassoon. The ground-truth alignment between audio and midi were generated by human annotation.

Andreas Arzt contributed a heavily polyphonic dataset consisting of 3 piano performances of the Prelude in G minor op. 23-5 by Sergei Rachmaninoff. The 3 performances (by Ashkenazy, Gavrilov and Shelley) differ heavily in their style of interpretation. The ground truth data was compiled by extensive manual correction of off-line alignments. ''Due to an oversight this data was not used for the evaluation runs.''

Marius Miron, Juan Bosch, Julio Carabias and Jordi Janer contributed with four passages of symphonic music from the Classical and Romantic periods. The first passage is a soprano aria of Donna Elvira from the opera Don Giovanni by W. A. Mozart (1756-1791), corresponding to the Classical period, and traditionally played by a small group of musicians. The second passage is from L. van Beethoven's (1770-1827) Symphony no. 7, featuring big chords and string crescendo. The chords and pauses make the reverberation tail of a concert hall clearly audible. The third passage is from Bruckner's (1824-1896) Symphony no. 8, and represents the late Romantic period. It features large dynamics and size of the orchestra. Finally, G. Mahler's Symphony no. 1, also featuring a large orchestra is another example of late romanticism. The ground-truth alignment between audio and midi were generated by human annotation.

== Evaluation procedures ==

Evaluation procedure consists of running score followers on a database of aligned audio to score where the database contains score, and performance audio (for system call) and a reference alignment (for evaluations) --
See http://ismir2007.ismir.net/proceedings/ISMIR2007_p315_cont.pdf for details.

See the details of 2006 proposal on the [[2006:Score_Following_Proposal|MIREX 2006 Wiki]]

=== I/O Format ===
Each system should conform to the following format:

''doScofo.sh "/path/to/audiofile.wav" "/path/to/midi_score_file.mid" "/path/to/result/filename.txt"

The stdout and stderr will be logged.

"/path/to/result/filenam.txt" should be have one line per detected note with the following 4 columns

1. estimated note onset time in performance audio file (ms)
2. detection time relative to performance audio file (ms)
3. note start time in score (ms)
4. MIDI note number in score (int)

Example :
''1800 1800 0 75''
''2021 2022 187.5 73''
''... ... ... ...''

Remarks: The third column with the detected note's start time in score serves as the unique identifier of a note (or chord for polyphonic scores) that links it to the ground truth onset of that note within the reference alignment files. The fourth column of MIDI note number is there only for your convenience, to know your way around in the result files, if you know the melody in MIDI.

=== Packaging submissions ===
All submissions should be statically linked to all libraries (the presence of
dynamically linked libraries cannot be guarenteed).

All submissions should include a README file including the following the
information:

* Command line calling format for all executables and an example formatted set of commands
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Any required environments (and versions), e.g. python, java, bash, matlab.

== Time and hardware limits ==
Due to the potentially high number of particpants in this and other audio tasks,
hard limits on the runtime of submissions are specified.

A hard limit of 12 hours will be imposed on rthe total runtime of algorithms. Submissions that exceed this runtime may not receive a result.

== Potential Participants ==

Francisco Rodriguez / fjrodrig@ujaen.es

2021:Query by Tapping

2021-09-10T20:00:01Z

Djevans:

== Overview ==
The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2021 ==

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <code><dbMidi.list></code> is the input list of database midi files named as <code>uniq_key.mid</code>. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <code><dir_workspace_root></code>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list_train> [dir_workspace_root]

Where <code><dbMidi_list></code> is a list of the MIDI files in the database to match against (see Step 0), and <code><query_file_list_train></code> maps each query to its associated ground truth. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <code><query_file_list_train></code> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are <code>.onset</code> and <code>.y_onset</code> files (for subtask 3), then the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same <code>.onset</code> file extension as previous symbolic input query datasets.

An <code>.onset</code> file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an <code>.onset</code> file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding <code>.onset</code> file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list_test> <result_file> [dir_workspace_root]

Where <code><query_file_list_test></code> is a single-column text file of input queries (the <code>.onset</code> files only, not the <code>.mid</code> files), and <code><result_file></code> is the filename where your script should store results. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

<code>query_file_list_test</code> thus has the following format:

qbtQuery/query_00001.onset
qbtQuery/query_00002.onset
qbtQuery/query_00003.onset
...

<code><result_file></code> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <code><result_file></code> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2021:Query by Tapping

2021-09-10T19:59:28Z

Djevans: Created page with "== Overview == The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone..."

== Overview ==
The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have three corpora for evaluation:

* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
** 890 onset & .wav queries; 136 ground-truth MIDI files
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).
** 410 onset queries; 143 ground-truth MIDI files (128 of which have at least one query)
* Kaneshiro et al.'s [http://ccrma.stanford.edu/groups/qbtextended/data/qbt-extended-onset.zip QBT-Extended]: This dataset contains only onset files (obtained from users tapping on a touchscreen). Documentation can be found [http://ccrma.stanford.edu/groups/qbtextended/dataset.html here].
** 3,365 onset queries (1,412 from long-term memory and 1,953 from short-term memory) from 60 participants; 51 ground-truth MIDI files
** A hidden dataset is currently being collected, from 20 new participants

== Discussions for 2020 ==

== Task description ==
* '''Evaluations are performed separately on each dataset'''

=== Subtask 1: QBT with symbolic input ===
* '''Test database''': The set of ground-truth MIDI files corresponding to each dataset.
* '''Query files''': Text files of onset times to retrieve target MIDIs from all datasets listed above. These onset files can help participant concentrate on similarity matching instead of onset detection. Onset files derived from .wav files cannot guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 2: QBT with wave input ===
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

=== Subtask 3: QBT-Extended with symbolic input (new for 2014) ===
* This subtask uses a longer query vector concatenating tap times and (pitch) positions.
* '''Development dataset''': The set of ground-truth MIDI files in the QBT-Extended dataset. Both onset times and MIDI note numbers are used.
* '''Query files''': Text files of onset times in the QBT-Extended dataset (long-term and short-term memory queries). Both onset times and vertical coordinates of tasks are considered.
* '''Development evaluation''': Return top 10 candidates for each query file in the development dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.
* '''Test evaluation''': Return top 10 candidates for each query file in the hidden dataset. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). We may also consider Top-5 and Top-1 scoring.

== Command formats ==

=== Step 0: Indexing the MIDIs collection ===
If your algorithm needs to pre-process (e.g., index) the database, your code should do so using the following command-line format (Note that this step is not required unless you want to index or preprocess the MIDI database).

Command format should look like this:

indexing <dbMidi.list> <dir_workspace_root>

where <code><dbMidi.list></code> is the input list of database midi files named as <code>uniq_key.mid</code>. For example:

QBT/database/00001.mid
QBT/database/00002.mid
QBT/database/00003.mid
QBT/database/00004.mid
...

Output indexed files are placed into <code><dir_workspace_root></code>.

=== Step 1: Training ===

The command format should be like this:

qbtTraining <dbMidi_list> <query_file_list_train> [dir_workspace_root]

Where <code><dbMidi_list></code> is a list of the MIDI files in the database to match against (see Step 0), and <code><query_file_list_train></code> maps each query to its associated ground truth. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

==== Per-task input specification ====
If the input query files are onset files (for subtask 1), then the format of <code><query_file_list_train></code> is like this (tab-separated):

qbtQuery/query_00001.onset 00001.mid
qbtQuery/query_00002.onset 00001.mid
qbtQuery/query_00003.onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

If the input query files are wave files (for subtask 2), the the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.wav 00001.mid
qbtQuery/query_00002.wav 00001.mid
qbtQuery/query_00003.wav 00002.mid
...

If the input query files are <code>.onset</code> and <code>.y_onset</code> files (for subtask 3), then the format of <code><query_file_list></code> is like this (tab-separated):

qbtQuery/query_00001.onset qbtQuery/query_00001.y_onset 00001.mid
qbtQuery/query_00002.onset qbtQuery/query_00002.y_onset 00001.mid
qbtQuery/query_00003.onset qbtQuery/query_00003.y_onset 00002.mid
...

See details of [[#Onset files format|Onset files format]]

===== Onset files format =====
To preserve compatibility with the original task, the QBT-E query files share the same <code>.onset</code> file extension as previous symbolic input query datasets.

An <code>.onset</code> file is a space-separated text file of elapsed onset times (in milliseconds) from the first onset, which is always 0.0. Example of 5 onsets:

0.0 479.922 720.069 976.071 1215.694

For subtask 3, the additional dimension (position/pitch/contour) is provided in a file with the same name, but with extension <code>.y_onset</code>. Similarly to an <code>.onset</code> file, <code>.y_onset</code> files are space-separated text files, the same length as the corresponding <code>.onset</code> file, that lists the absolute vertical position of each tap on the touchscreen. Example (corresponding to above):

291.000000 293.500000 305.500000 302.000000 239.000000

=== Step 2: Testing ===
The command format should be like this:

qbtTesting <query_file_list_test> <result_file> [dir_workspace_root]

Where <code><query_file_list_test></code> is a single-column text file of input queries (the <code>.onset</code> files only, not the <code>.mid</code> files), and <code><result_file></code> is the filename where your script should store results. You can use <code>[dir_workspace_root]</code> to store any temporary indexing/database structures. (You can omit <code>[dir_workspace_root]</code> if you do not need it at all.)

<code>query_file_list_test</code> thus has the following format:

qbtQuery/query_00001.onset
qbtQuery/query_00002.onset
qbtQuery/query_00003.onset
...

<code><result_file></code> gives ranked top-10 candidates for each query (note that ranking of the candidates is new for 2014). For instance <code><result_file></code> should have the following format for subtasks 1 and 3:

qbtQuery/query_00001.onset: 00025 01003 02200 ...
qbtQuery/query_00002.onset: 01547 02313 07653 ...
qbtQuery/query_00003.onset: 03142 00320 00973 ...
...

And for subtask 2:

qbtQuery/query_00001.wav: 00025 01003 02200 ...
qbtQuery/query_00002.wav: 01547 02313 07653 ...
qbtQuery/query_00003.wav: 03142 00320 00973 ...
...

Where 00025 is the top-ranked (closest match) MIDI file for query_00001, followed by 01003, 02200, etc. Note that the output should be the names of the MIDI files (e.g., <code>00025</code> means <code>00025.mid</code>); they are not necessary 5-digit numbers.

== Potential Participants ==

== References ==
Chen JCC, and Chen ALP (1998). Query by rhythm: An approach for song retrieval in music databases. Research Issues in Data Engineering, Proceedings of IEEE Eighth International Workshop on Continuous-Media Databases and Applications, 139-146.

Eisenberg G, Batke JM, and Sikora T (2004). BeatBank - an MPEG-7 compliant query by tapping system. Audio Engineering Society Convention 116, paper 6136.

Eisenberg G, Batke JM, and Sikora T (2004). Efficiently computable similarity measures for query by tapping systems. Proceedings of the Seventh International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, 189-192.

Hanna P, and Robine M (2009) Query by tapping system based on alignment algorithm. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1881-1884.

Hébert S, and Peretz I (1997). Recognition of music in long-term memory: Are melodic and temporal patterns equal partners? Memory & Cognition 25:4, 518-533.

Jang JSR, Lee HR, and Yeh CH (2001). Query by tapping: A new paradigm for content-based music retrieval from acoustic input. Advances in Multimedia Information Processing PCM, 590-597.

Kaneshiro B, Kim HS, Herrera J, Oh J, Berger J, and Slaney M (2013). QBT-extended: An annotated dataset of melodically contoured tapped queries. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil, 329-334.

Peters G, Anthony C, and Schwartz M (2005). Song search and retrieval by tapping. Proceedings of the National Conference on Artificial Intelligence 20, 1696.

Peters G, Cukierman D, Anthony C, and Schwartz M (2006). Online music search by tapping. Ambient Intelligence in Everyday Life, 178-197.

2021:Query by Singing/Humming

2021-09-10T19:58:31Z

Djevans: Created page with "== Description == The text of this section is copied from the 2010 page. Please add your comments and discussions for 2021. The goal of the Query-by-Singing/Humming (QBSH)..."

== Description ==

The text of this section is copied from the 2010 page. Please add your comments and discussions for 2021.

The goal of the Query-by-Singing/Humming (QBSH) task is the evaluation of MIR systems that take as query input queries sung or hummed by real-world users. More information can be found in:

* [[2016:Query_by_Singing/Humming]]
* [[2015:Query_by_Singing/Humming]]
* [[2014:Query_by_Singing/Humming]]
* [[2013:Query_by_Singing/Humming]]
* [[2012:Query_by_Singing/Humming]]
* [[2011:Query_by_Singing/Humming]]
* [[2010:Query_by_Singing/Humming]]
* [[2009:Query_by_Singing/Humming]]
* [[2008:Query_by_Singing/Humming]]
* [[2007:Query_by_Singing/Humming]]
* [[2006:QBSH:_Query-by-Singing/Humming]]

=== Subtask 1: Classic QBSH evaluation ===
This is the classic QBSH problem where we need to find the ground-truth midi from a user's singing or humming.
* '''Queries''': human singing/humming snippets (.wav). Queries are from MIR-QBSH and IOACAS corpora described below.
* '''Database''': ground-truth and noise MIDI files(which are monophonic). Comprised of ground-truth MIDIs from MIR-QBSH corpus (48), and IOACAS corpus (106), along with a cleaned version of Essen Database(2000+ MIDIs which are not available to the participator)
* '''Output''': top-10 candidate list.
* '''Evaluation''': Top-10 hit rate (1 point is scored for a hit in the top 10 and 0 is scored otherwise).

=== Subtask 2: Variants QBSH evaluation ===
This is based on Prof. Downie's idea that queries are variants of "ground-truth" midi. In fact, this becomes more important since user-contributed singing/humming is an important part of the song database to be searched, as evidenced by the QBSH search service at [http://www.midomi.com/ www.midomi.com].
* '''Queries''': human singing/humming snippets (.wav). Queries are from Roger Jang's corpus and IOACAS corpus.
* '''Database''': human singing/humming snippets (.wav) from all available corpora (excluding the query input being searched).
* '''Output''': top-10 candidate list.
* '''Evaluation''': Top-10 hit rate (1 point is scored for a hit in the top 10 and 0 is scored otherwise).

To make algorithms able to share intermediate steps, participants are encouraged to submit separate tracker and matcher modules instead of integrated ones, which is according to Rainer Typke's suggestion. So trackers and matchers from different submissions could work together with the same pre-defined interface and thus for us it's possible to find the best combination.

== Data ==
Currently we have 2 publicly available corpora for QBSH:

* Roger Jang's [https://music-ir.org/evaluation/MIREX/data/qbsh/MIR-QBSH-corpus.tar.gz MIR-QBSH corpus] which is comprised of 4431 queries along with 48 ground-truth MIDI files. All queries are from the beginning of references. Manually labeled pitch for each recording is available.

* [https://music-ir.org/evaluation/MIREX/data/qbsh/IOACAS_QBH_Corpus.tar.gz IOACAS corpus] comprised of 759 queries and 298 monophonic ground-truth MIDI files (with MIDI 0 or 1 format). There are no "singing from beginning" guarantee.

The noise MIDI will be the 5000+ Essen collection(can be accessed from http://www.esac-data.org/).

To build a large test set which can reflect real-world queries, it is suggested that every participant makes a contribution to the evaluation corpus. Sometimes this is hard in practice. So we shall adopt "no hidden dataset" policy if there are not enough user-contributed copora.

== Evaluation Corpus Contribution ==
Every participant will be asked to contribute 100~200 wave queries (8k 16bits) as well as the ground truth MIDI as test data.

A simple tool for recording query data will be public soon. You may need to have .NET 2.0 or above installed in your system in order to run this program. The generated files conform to the format used in the IOACAS corpus. Of course you are also welcomed to use your own program to record the query data.

If there are not enough user-contributed corpora, then we shall adopt "no hidden dataset" policy for QBSH task as usual.

== Submission Format ==

=== Breakdown Version ===
1. Database indexing/building. Command format should look like this:

indexing %dbMidi.list% %dir_workspace_root%

where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example:

./QBSH/midiDatabase/00001.mid
./QBSH/midiDatabase/00002.mid
./QBSH/midiDatabase/00003.mid
./QBSH/midiDatabase/00004.mid
...

Output indexed files are placed into %dir_workspace_root%. (For task 2, %dbMidi.list% is in fact a list of wav files in the database.)

2. Pitch tracker. Command format:

pitch_tracker %queryWave.list% %dir_query_pitch%

where %queryWave.list% looks like

queryWave/query_00001.wav
queryWave/query_00002.wav
queryWave/query_00003.wav
...

Each input file dir_query/query_xxxxx.wav in %queryWave.list% outputs a corresponding transcription %dir_query_pitch%/query_xxxxx.pitch which gives the pitch sequence in midi note scale with the resolution of 10ms:

0
0
62.23
62.25
62.21
...

Thus a query with x seconds should output a pitch file with 100*x lines. Places of silence/rest are set to be 0.

3. Pitch matcher. Command format:

pitch_matcher %dbMidi.list% %queryPitch.list% %resultFile%

where %queryPitch.list% looks like

queryPitch/query_00001.pitch
queryPitch/query_00002.pitch
queryPitch/query_00003.pitch
...

and the result file gives top-10 candidates(if has) for each query:

queryPitch/query_00001.pitch: 00025 01003 02200 ...
queryPitch/query_00002.pitch: 01547 02313 07653 ...
queryPitch/query_00003.pitch: 03142 00320 00973 ...
...

=== Integrated Version ===
If you want to pack everything together, the command format should be much simpler:

qbshProgram %dbMidi.list% %queryWave.list% %resultFile% %dir_workspace_root%

You can use %dir_workspace_root% to store any temporary indexing/database structures. The result file should have the same format as mentioned previously. (For task 2, %dbMidi.list% is in fact a list of wav files in the database to be retrieved.)

=== Packaging submissions ===
All submissions should be statically linked to all libraries (the presence of
dynamically linked libraries cannot be guarenteed).

All submissions should include a README file including the following the
information:

* Command line calling format for all executables and an example formatted set of commands
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Any required environments (and versions), e.g. python, java, bash, matlab.

== Time and hardware limits ==
Due to the potentially high number of particpants in this and other audio tasks,
hard limits on the runtime of submissions are specified.

A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.

== Potential Participants ==

2021:Patterns for Prediction

2021-09-10T19:56:47Z

Djevans: Created page with "== Description == '''In brief''', there are two subtasks: * Subtask 1) '''The Explicit Task''' - Algorithms that take an excerpt of music as input (the ''prime''), and output..."

== Description ==
'''In brief''', there are two subtasks:
* Subtask 1) '''The Explicit Task''' - Algorithms that take an excerpt of music as input (the ''prime''), and output a predicted ''continuation'' of the excerpt
* Subtask 2) '''The Implicit Task''' - Algorithms that are given a prime and a candidate continuation and return the probability that this is the true continuation of the provided prime (i.e. the notes which occur immediately after)

Your task captains are: [http://beritjanssen.com/ Berit Janssen] (berit.janssen), [https://sites.google.com/view/iyr/home Iris YuPing Ren] (yuping.ren.iris), [https://jamesowers.github.io/ James Owers] (james.f.owers), and [http://tomcollinsresearch.net/ Tom Collins] (tomthecollins, all at gmail.com). Please copy in all four of us if you have questions/comments.
For participants interested in the 2021 edition, please also copy in Zongyu Yin (zy728 at york.ac.uk) instead of James Owers.

The '''submission deadline''' is Monday September 9th, 2019 (any time as long as it's Sep 9th somewhere on Earth!).

'''Relation to the pattern discovery task''': The Patterns for Prediction task is an offshoot of the [https://www.music-ir.org/mirex/wiki/2013:Discovery_of_Repeated_Themes_%26_Sections Discovery of Repeated Themes & Sections task] (2013-2017). We hope to run the former (Patterns for Prediction) task and pause the latter (Discovery of Repeated Themes & Sections). In future years we may run both.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next ''N'' musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; [https://www.music-ir.org/mirex/wiki/2013:Discovery_of_Repeated_Themes_%26_Sections MIREX Discovery of Repeated Themes & Sections task]; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt ([https://www.music-ir.org/mirex/abstracts/2013/DM10.pdf Meredith, 2013]). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

==Data==
The Patterns for Prediction Development Dataset (PPDD-Sep2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it is not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Here are the PPDD-Sep2018 variants for download:
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_aud_mono_small.zip audio, monophonic, small] (92 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_aud_mono_medium.zip audio, monophonic, medium] (850 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_aud_mono_large.zip audio, monophonic, large] (8.46 GB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_aud_poly_small.zip audio, polyphonic, small] (137 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_aud_poly_medium.zip audio, polyphonic, medium] (1.35 GB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_aud_poly_large.zip audio, polyphonic, large] (13.44 GB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_sym_mono_small.zip symbolic, monophonic, small] (< 1 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_sym_mono_medium.zip symbolic, monophonic, medium] (3 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_sym_mono_large.zip symbolic, monophonic, large] (32 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_sym_poly_small.zip symbolic, polyphonic, small] (< 1 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_sym_poly_medium.zip symbolic, polyphonic, medium] (9 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_sym_poly_large.zip symbolic, polyphonic, large] (64 MB)
("Large" datasets were compressed using the [https://www.mankier.com/1/7za p7zip] package, installed via "brew install p7zip on Mac".)

===Some examples===
[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-jul2018/examples/0a983538-61b5-4b9d-9ad9-23e05f548e5c.wav This prime] finishes with two G’s followed by a D above. Looking at the [http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-jul2018/examples/0a983538-61b5-4b9d-9ad9-23e05f548e5c.png piano roll] or listening to the linked file, we can see/hear that this pitch pattern, in the exact same rhythm, has happened before (see bars 17-18 transition in the piano roll). Therefore, we and/or an algorithm, might predict that the first note of the continuation will follow the pattern established in the previous occurrence, returning to G 1.5 beats later.

[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-jul2018/examples/001f5992-527d-4e04-8869-afa7cbb74cd0.wav This] is another example where a previous occurrence of a pattern might help predict the contents of the continuation. Not all excerpts contain patterns (in fact, one of the motivations for running the task is to interrogate the idea that patterns are abundant in music and always informative in terms of predicting what comes next). [http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-jul2018/examples/fc2fda7c-9f55-4bf3-8fa8-f337e35aa20f.wav This one], for instance, does not seem to contain many clues for what will come next. And finally, [http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-jul2018/examples/b9261e74-125a-429e-ae27-5b51abdc7d81.wav this one] might not contain any obvious patterns, but other strategies (such as schematic or tonal expectations) might be recruited in order to predict the contents of the continuation.

(These examples are from an earlier version of the dataset, PPDD-Jul2018, but the above observations apply also to the current version of the dataset.)

===Preparation of the data===
Preparation of the monophonic datasets was more involved than the polyphonic datasets: for both, we imported each MIDI file, quantised it using a subset of the Farey sequence of order 6 (Collins, Krebs, et al., 2014), and then excerpted a prime and continuation at a randomly selected time. For the monophonic datasets, we filtered for:
*channels that contained at least 20 events in the prime;
*channels that were at least 80% monophonic at the outset, meaning that at least 80% of their segments (Pardo & Birmingham, 2002) contained no more than one event;
*channels where the maximum inter-ontime interval in the prime was no more than 8 quarter-note beats.
*we then "skylined" these channels (independently) so that no two events had the same start time (maximum MNN chosen in event of a clash), and double-checked that they still contained at least 20 events;
*one suitable channel was then selected at random, and the prime appears in the dataset if the continuation contained at least 10 events.
If any of the above could not be satisfied for the given input, we skipped this MIDI file.

For the polyphonic data, we applied the minimum note criteria of 20 in the prime and 10 in the continuation, as well as the prime maximum inter-ontime interval of 8, but it was not necessary to measure monophony or perform skylining.

Audio files were generated by importing the corresponding CSV and descriptor files and using a sample bank of piano notes from the [https://magenta.tensorflow.org/datasets/nsynth Google Magenta NSynth dataset] (Engel et al., 2017) to construct and export the waveform.

The foil continuations were generated using a Markov model of order 1 over the whole texture (polyphonic) or channel (monophonic) in question, and there was '''no''' attempt to nest this generation process in any other process cognisant of repetitive or phrasal structure. See Collins and Laney (2017) for details of the state space and transition matrix.

==Submission Format==
In terms of input representations, we will evaluate 4 largely independent versions of the task: audio, monophonic; audio, polyphonic; symbolic, monophonic; symbolic, polyphonic. Participants may submit algorithms to 1 or more of these versions, and should list these versions clearly in their readme. '''Irrespective of input representation''', all output for subtask (1) should be in "ontime", "MNN" CSV files. The CSV may contain other information, but "ontime" and "MNN" should be in the first two columns, respectively. All output for subtask (2) should be an indication whether of the two presented continuations, "A" or "B" is judged by the algorithm to be genuine. This should be one CSV file for an entire dataset, with first column "id" referring to the file name of a prime-continuation pair, second column "A" containing a likelihood value in [0, 1] for the genuineness of the continuation in folder A, and column "B" similarly for the continuation in folder B.

All submissions should be statically linked to all dependencies and include a README file including the following information:

*input representation(s), should be 1 or more of "audio, monophonic"; "audio, polyphonic"; "symbolic, monophonic"; "symbolic, polyphonic";
*subtasks you would like your algorithm to be evaluated on, should be "1", "2", or "1 and 2" (see first sentences of [[2018:Patterns_for_Prediction#Description]] for a reminder);
*command line calling format for all executables and an example formatted set of commands;
*number of threads/cores used or whether this should be specified on the command line;
*expected memory footprint;
*expected runtime;
*any required environments and versions, e.g. Python, Java, Bash, MATLAB.

===Example Command Line Calling Format===

Python:

python <your_script_name.py> -i <input_folder> -o <output_folder>

==Evaluation Procedure==
'''In brief''':
* For subtask 1), we match the algorithmic output with the original continuation and compute a match score which is described below. We additionally provide a pitch score which ignores rhythm.
* For subtask 2), we return the accuracy: the count of how many times an algorithm judged the genuine continuation as most likely. Additionally we return the mean and variance of the predicted probability for the true continuation (the higher the mean and lower the variance the better)

The code used for all evaluations is available on [https://github.com/BeritJanssen/PatternsForPrediction/tree/mirex2019 GitHub].

===Notation===

The input excerpt ends with a final note event: <math>(x_0, y_0)</math>, where <math>x_0</math> is ontime (start time measured in quarter-note beats starting with 0 for bar 1 beat 1), <math>y_0</math> is pitch, represented by MNN.

For subtask 1, the algorithm predicts the continuations: <math>(\hat{x}_1, \hat{y}_1)</math>, <math>(\hat{x}_2, \hat{y}_2)</math>, ..., <math>(\hat{x}_{n^\prime}, \hat{y}_{n^\prime})</math>, where <math>\hat{x}_i</math> are predicted ontimes, and <math>\hat{y}_i</math> are predicted MNNs. The true continuations are notated <math>(x_1, y_1), (x_2, y_2),..., (x_n, y_n)</math>. The predicted continuation ontimes are strictly non-decreasing, that is <math>\hat{x}_0 \leq \hat{x}_1 \leq \cdots \leq \hat{x}_{n^\prime}</math>, and so are the true continuation ontimes, that is <math>x_0 \leq x_1 \leq \cdots \leq x_n</math>.

===Subtask 1 - Explicit Task===

For subtask 1, the algorithm is provided with a prime, and produces a continuation. We clip this continuation at 10 beats long (plus the time for the last note to finish). We then compare the generated continuation with the true continuation using two different scores: the cardinality and pitch score. In short, the cardinality score attempts to find the best overlap be between the continuations, and the pitch score makes a histogram of pitches for both continuations and checks the overlap between them.

====The Cardinality Score====
We represent each note in the true and algorithmic continuation as a point in a two-dimensional space of onset and pitch, giving the point-set <math>\mathbf{P} = \{ (x_1, y_1), (x_2, y_2),..., (x_n, y_n) \}</math> for the true continuation, and <math>\mathbf{Q} = \{ (\hat{x}_1, \hat{y}_1), (\hat{x}_2, \hat{y}_2),..., (\hat{x}_{n^\prime}, \hat{y}_{n^\prime}) \}</math> for the algorithmic continuation. We calculate differences between all points <math>p = (x_i, y_i)</math> in <math>\mathbf{P}</math> and <math>q = (\hat{x}_j, \hat{y}_j)</math> in <math>\mathbf{Q}</math>, which represent the translation vectors <math>\mathbf{T}</math> that transform a given algorithmically generated note into a note from the true continuation:

<math>\mathbf{T} = \left\{p - q \; \forall \; p \in \mathbf{P},\, q \in \mathbf{Q}\right\}</math>

From this, we find the <math>t \in \mathbf{T}</math> which maximises the cardinality of the set of notes which now overlap under this translation. The maximum cardinality is the Cardinality Score (CS):

<math>
\text{CS}(\mathbf{P},\mathbf{Q}) = \max_{t \in \mathbf{T}} \left|\left\{q \; \forall \; q \in \mathbf{Q} \;\, | \;\, (q + t) \in \mathbf{P}\right\}\right|
</math>

Here is an illustration of the search for the CS:

[[File:card_score_animation.gif|600px]]

'''Figure 1.''' An illustration of the search for the cardinality score. In this case, translating the candidate continuation up by 5 semitones (and no temporal shift) gives the CS of 4.

We define recall as the number of correctly predicted notes, divided by the cardinality of the true continuation point set <math>\mathbf{P}</math>. Since there exists at least one point in <math>\mathbf{Q}</math> which can be translated by any vector to a point in <math>\mathbf{P}</math>, we subtract <math>1</math> from numerator and denominator to scale to <math>[0,1]</math>.

<math>
\text{Rec} = (\text{cs}(\mathbf{P},\mathbf{Q}) - 1) / (|\mathbf{P}| - 1)
</math>

Precision is the number of correctly predicted notes, divided by the cardinality of the point set of the algorithmic continuation <math>\mathbf{Q}</math>, scaled in the same way:

<math>
\text{Prec} = (\text{cs}(\mathbf{P},\mathbf{Q}) - 1) / (|\mathbf{Q}| - 1)
</math>

====The Pitch Score====
The cardinality rewards continuations that are the correct 'shape' - it will give the maximum score to a continuation which is a copy of the true continuation but transposed one semitone down. This is not a very musically correct continuation! To counter this in some way, we also asses the pitches of the continuations more directly.

To do this we create two '''normalised''' histograms of the pitches in both continuations. The score is simply the overlap between the overlaid histograms (maximum score is 1, minimum is 0).

We then repeat the process but disregard octave (i.e. take the MNN pitch modulo 12).

====A note on Entropy====
Some existing work in this area (e.g., Conklin & Witten, 1995; Pearce & Wiggins, 2006; Temperley, 2007) evaluates algorithm performance in terms of entropy. If we have time to collect human listeners' judgments of likely (or not) continuations for given excerpts, then we will be in a position to compare the entropy of listener-generated distributions with the corresponding algorithm distributions. This would open up the possibility of entropy-based metrics, but we consider this of secondary importance to the metrics outlined above.

===Subtask 2 - Implicit Task===

For subtask two, we provide each model with a prime and continuation and it returns the probability that this is the true continuation. We only provide two continuations - the true continuation, and a foil continuation generated by our baseline markov model. We then softmax these values such that the sum of probabilities for the two continuations is 1. The best model should assign a high probability to the true continuation (1) and a low continuation to the foil continuation (0).

====Accuracy====
To get the accuracy of the model, we simply sum the number of times it assigns a higher probability for the true continuation.

====Average Probability====
It's possible models could attain a similar accuracy but assign different probabilities e.g. one model gets an accuracy of 1 but gives the true continuations a probability of around .51, whereas another could predict near 1 for the true continuations. To identify more confident models, and get a slightly more in depth understanding of model performance, we also take the mean and variance of the probability assigned to the true continuation. The non-informative model (assigning .5 to all inputs), will get a mean probability score of .5, the minimum score is 0, and the maximum score is of course 1.

==Questions (Q), Answers (A), and Comments (C)==

Q. Instead of evaluating continuations, have you considered evaluating an algorithm's ability to predict content between two timepoints, or before a timepoint?

A. Yes we considered including this also, but opted not to for sake of simplicity. Furthermore, these alternatives do not have the same intuitive appeal as predicting future events.

Q. Why do some files sound like they contain a drum track rendered on piano?

A. Some of the MIDI files import as a single channel, but upon listening to them it is evident that they contain multiple instruments. For the sake of simplicity, we removed percussion channels where possible, but if everything was squashed down into a single channel, there was not much we could do.

C. to_the_sun--at--gmx.com writes: "This is exactly what I'm interested in! I have an open-source project called The Amanuensis (https://github.com/to-the-sun/amanuensis) that uses an algorithm to predict where in the future beats are likely to fall.

"Amanuensis constructs a cohesive song structure, using the best of what you give it, looping around you and growing in real-time as you play. All you have to do is jam and fully written songs will flow out behind you wherever you go.

"My algorithm right now is only rhythm-based and I'm sure it's not sophisticated enough to be entered into your contest, but I would be very interested in the possibility of using any of the algorithms that are, in place of mine in The Amanuensis. Would any of your participants be interested in some collaboration? What I can bring to the table would be a real-world application for these algorithms, already set for implementation."

Q. I'm interested in performing this task on the symbolic dataset, but I don't have an audio-based algorithm. It was unclear to me if the inputs are audio, symbolic, both, or either.

A. We have clarified, at the top of [[2018:Patterns_for_Prediction#Submission_Format]], that submissions in 1-4 representational categories are acceptable. It's also OK, say, for an audio-based algorithm to make use of the descriptor file in order to determine beat locations. (You could do this by looking at the <math>u = \mathrm{bpm}</math> value, and then you would know that the main beats in the WAV file are at <math>0, 60/u, 2 \cdot 60/u,\ldots</math> sec.)

==Time and Hardware Limits==

A total runtime limit of 72 hours will be imposed on each submission.

==Seeking Contributions==

*We would like to evaluate against real (not just synthesized-from-MIDI) audio versions. If you have a good idea of how we might make this available to participants, let us know. We would be happy to acknowledge individuals and/or companies for helping out in this regard.

*More suggestions/comments/ideas on the task is always welcome!

==Acknowledgments==

Thank you to Anja Volk, Darrell Conklin, Srikanth Cherla, David Meredith, Matevz Pesek, and Gissel Velarde for discussions!

==References==
*Cherla, S., Weyde, T., Garcez, A., and Pearce, M. (2013). A distributed model for multiple-viewpoint melodic prediction. In In ''Proceedings of the International Society for Music Information Retrieval Conference'' (pp. 15-20). Curitiba, Brazil.

*Collins, T. (2011). "[http://oro.open.ac.uk/30103/ Improved methods for pattern discovery in music, with applications in automated stylistic composition]". PhD Thesis.

*Collins, T., Böck, S., Krebs, F., & Widmer, G. (2014). [http://tomcollinsresearch.net/pdf/collinsEtAlAES2014.pdf Bridging the audio-symbolic gap: The discovery of repeated note content directly from polyphonic music audio]. In ''Proceedings of the Audio Engineering Society's 53rd Conference on Semantic Audio''. London, UK.

*Collins, T., Tillmann, B., Barrett, F. S., Delbé, C., & Janata, P. (2014). [http://psycnet.apa.org/journals/rev/121/1/33/ A combined model of sensory and cognitive representations underlying tonal expectations in music: From audio signals to behavior]. ''Psychological Review, 121''(1), 33-65.

*Collins T., & Laney, R. (2017). [http://jcms.org.uk/issues/Vol1Issue2/computer-generated-stylistic-compositions/computer-generated-stylistic-compositions.html Computer-generated stylistic compositions with long-term repetitive and phrasal structure]. ''Journal of Creative Music Systems, 1''(2).

*Conklin, D., and Witten, I. H. (1995). Multiple viewpoint systems for music prediction. ''Journal of New Music Research, 24''(1), 51-73.

*Elmsley, A., Weyde, T., & Armstrong, N. (2017). Generating time: Rhythmic perception, prediction and production with recurrent neural networks. ''Journal of Creative Music Systems, 1''(2).

*Engel, J., Resnick, C., Roberts, A., Dieleman, S., Eck, D., Simonyan, K., & Norouzi, M. (2017). Neural audio synthesis of musical notes with WaveNet autoencoders. https://arxiv.org/abs/1704.01279

*Gjerdingen, R. O. (1989). Using connectionist models to explore complex musical patterns. ''Computer Music Journal, 13''(3), 67-75.

*Gjerdingen, R. (2007). Music in the galant style. New York, NY: Oxford University Press.

*Hadjeres, G., Pachet, F., & Nielsen, F. (2016). Deepbach: A steerable model for Bach chorales generation. arXiv preprint arXiv:1612.01010.

*Huron, D. (2006). Sweet Anticipation. Music and the Psychology of Expectation. Cambridge, MA: MIT Press.

*Janssen, B., Burgoyne, J. A., & Honing, H. (2017). Predicting variation of folk songs: A corpus analysis study on the memorability of melodies. ''Frontiers in Psychology, 8'', 621.

*Janssen, B., van Kranenburg, P., & Volk, A. (2017). Finding occurrences of melodic segments in folk songs employing symbolic similarity measures. ''Journal of New Music Research, 46''(2), 118-134.

*Koelsch, S., Gunter, T. C., Wittfoth, M., & Sammler, D. (2005). Interaction between syntax processing in language and in music: an ERP study. ''Journal of Cognitive Neuroscience, 17''(10), 1565-1577.

*Lerdahl, F., and Jackendoff, R. (1983). "A generative theory of tonal music. Cambridge, MA: MIT Press.

*Margulis, E. H. (2014). ''On repeat: How music plays the mind''. New York, NY: Oxford University Press.

*Meredith, D. (1999). The computational representation of octave equivalence in the Western staff notation system. In ''Proceedings of the Cambridge Music Processing Colloquium''. Cambridge, UK.

*Meredith, D. (2013). COSIATEC and SIATECCompress: Pattern discovery by geometric compression. In ''Proceedings of the 10th Annual Music Information Retrieval Evaluation eXchange (MIREX'13)''. Curitiba, Brazil.

*Morgan, E., Fogel, A., Nair, A., & Patel, A. D. (2019). Statistical learning and Gestalt-like principles predict melodic expectations. ''Cognition, 189'', 23-34.

*Pardo, B., & Birmingham, W. P. (2002). Algorithms for chordal analysis. ''Computer Music Journal, 26''(2), 27-49.

*Pearce, M. T., & Wiggins, G. A. (2006). Melody: The influence of context and learning. ''Music Perception, 23''(5), 377–405.

*Raffel, C. (2016). "Learning-based methods for comparing sequences, with applications to audio-to-MIDI alignment and matching". PhD Thesis.

*Ren, I.Y., Koops, H.V, Volk, A., Swierstra, W. (2017). In search of the consensus among musical pattern discovery algorithms. In ''Proceedings of the International Society for Music Information Retrieval Conference'' (pp. 671-678). Suzhou, China.

*Roberts, A., Engel, J., Raffel, C., Hawthorne, C., & Eck, D. (2018). A hierarchical latent vector model for learning long-term structure in music. In ''Proceedings of the International Conference on Machine Learning'' (pp. 4361-4370). Stockholm, Sweden.

*Rohrmeier, M., & Pearce, M. (2018). Musical syntax I: theoretical perspectives. In ''Springer Handbook of Systematic Musicology'' (pp. 473-486). Berlin, Germany: Springer.

*Schellenberg, E. G. (1997). Simplifying the implication-realization model of melodic expectancy. ''Music Perception, 14''(3), 295-318.

*Schmuckler, M. A. (1989). Expectation in music: Investigation of melodic and harmonic processes. ''Music Perception, 7''(2), 109-149.

*Sturm, B.L., Santos, J.F., Ben-Tal., O., & Korshunova, I. (2016), Music transcription modelling and composition using deep learning. In ''Proceedings of the International Conference on Computer Simulation of Musical Creativity''. Huddersfield, UK.

*Temperley, D. (2007). ''Music and probability''. Cambridge, MA: MIT Press.

*Widmer, G. (2017). Getting closer to the essence of music: The con espressione manifesto. ''ACM Transactions on Intelligent Systems and Technology (TIST), 8''(2), 19.

2021:Music Detection

2021-09-10T19:56:13Z

Djevans: Created page with "==Description== Music detection refers to the task of finding music segments in an audio file. The two main applications of music detection algorithms are (1) the automatic i..."

==Description==

Music detection refers to the task of finding music segments in an audio file. The two main applications of music detection algorithms are (1) the automatic indexing and retrieving of auditory information based on its audio content, and (2) the monitoring of music for copyright management. Additionally, the detection of music can be applied as an intermediate step to improve the performance of algorithms designed for other purposes.

Regarding the application of music detection algorithms to the copyright management, the industry is lately becoming more and more interested in not only detecting the presence of music but also estimating if it appears in the foreground (as the main focus of attention) or in the background. In this scenario, the music detection task falls short as we need to estimate the loudness of music in relation to other simultaneous non-music sounds, i.e., its relative loudness. This is why we propose a second task that we name Music Relative Loudness Estimation. We define this second task as the task of finding music segments in and audio file and classifying them into foreground or background music.

==Tasks==

===Music Detection===

The music detection sub-task consists in finding segments of music in a signal. This task applies to complete recordings from archives. No assumptions are made about the number of segments present in each archive or about their duration.

classes: music (and non-music)

===Music Relative Loudness Estimation===

The music relative loudness estimation sub-task consists in finding segments of one of the following two classes: foreground music and background music. This task applies to complete recordings from archives. No assumptions are made about the number of segments present in each archive or about their duration.

classes: fg-music, bg-music (and non-music)

==Datasets==

===Available Training Datasets===

These resources may be a good starting point for participants.

GTZAN Speech and Music Dataset
http://opihi.cs.uvic.ca/sound/music_speech.tar.gz

Scheirer & Slaney Music Speech Corpus
http://www.ee.columbia.edu/~dpwe/sounds/musp/scheislan.html

MUSAN Corpus
http://www.openslr.org/17/

Muspeak Speech and Music Detection Dataset
http://mirg.city.ac.uk/datasets/muspeak/muspeak-mirex2015-detection-examples.zip

Music detection dataset:
www.seyerlehner.info/download/music_detection_dataset_dafx_07.zip
(Ask the author for the password)

Open Broadcast Media Audio from TV:
https://zenodo.org/record/3381249

===Evaluation Dataset===

====Content====

The evaluation dataset consists of 2987 1-minute, stereo excerpts at 22050 Hz extracted from programs from France (753), Germany (760), Spain (723) and the United States (751).

====Annotation====

The evaluation dataset has been cross-annotated by 3 annotators using a 6-class taxonomy: ''Music'', ''Foreground Music'', ''Similar'', ''Background Music'', ''Low Background Music'', and ''No Music'' as done in the OpenBMAT dataset, which can be used for training.

==Evaluation==

In the literature we find two ways of measuring the performance of an algorithm depending on the way we compare the ground truth with an algorithm's estimation: the segment-level evaluation and the event-level evaluation. We will report the statistics for each of these evaluations by file and for the whole dataset. We will do that for each algorithm and dataset.

===Segment-level evaluation:===

In the segment-level evaluation, we compare the estimation (est) produced by the algorithms with the reference (ref) in segments of 10 ms. We first compute the intermediate statistics for each class C, which include:
* True Positives (TPc): ref segment’s class = C & est segment’s class = C
* False Positives (FPc): ref segment’s class != C & est segment’s class = C
* True Negatives (TNc): ref segment’s class != C & est segment’s class != C
* False Negatives (FNc): ref segment’s class = C & est segment’s class != C

Then we report class-wise Precision, Recall and F-measure.
* Precision (Pc) = TPc / (TPc + FPc)
* Recall (Rc) = TPc / (TPc + FNc)
* F-measure (Fc) = 2 * Pc * Rc / (Pc + Rc)

As well as the overall Accuracy:
* Accuracy = (TP + TN) / (TP + TN + FP + FN)

Where:
* TP = sum(TPc), for every class c
* FP = sum(FPc), for every class c
* TN = sum(TNc), for every class c
* FN = sum(FNc), for every class c

===Event-level evaluation:===

In the event-level evaluation, we compare the estimation (est) produced by the algorithms with the reference (ref) in terms of events. Each annotated segment of the ground truth is considered and event. We first compute the intermediate statistics for the onsets and offsets of each class C, which include:
* True Positives (TPc): an est event of class = C that starts and ends at the same temporal positions as a ref event of class = C, taking into account a tolerance time-window.
* False Positives (FPc): an est event of class = C that starts and ends at temporal positions where no ref event of class = C does, taking into account a tolerance time-window.
* False Negatives (FNc): a ref event of class = C that starts and ends at temporal positions where no est event of class = C does, taking into account a tolerance time-window.

Then we report class-wise Precision, Recall, F-measure, Deletion Rate, Insertion Rate and Error Rate.
* Precision (Pc) = TPc / (TPc + FPc)
* Recall (Rc) = TPc / (TPc + FNc)
* F-measure (Fc) = 2 * Pc * Rc / (Pc + Rc)
* Deletion Rate (Dc) = FNc / Nc
* Insertion Rate (Ic) = FPc / Nc
* Error Rate (Ec) = Dc + Ic

Where:
* Nc is the number of ref events of class = C.

We also report the overall version of these statistics:
* Precision (P) = TP / (TP + FP)
* Recall (R) = TP / (TP + FN)
* F-measure (F) = 2 * P * R / (P + R)
* Deletion Rate (D) = FN / N
* Insertion Rate (I) = FP / N
* Error Rate (E) = D + I

Where:
* TP = sum(TPc), for every class c
* FP = sum(FPc), for every class c
* TN = sum(TNc), for every class c
* FN = sum(FNc), for every class c
* N is the number of ref events.

Different tolerance time-windows will be used: +/- 1000 ms, +/- 500 ms, +/- 200 ms, +/- 100 ms.

===Other evaluated features===

The execution time of each algorithm will also be reported.

==Submission Format==

===Command line calling format===

Submissions have to conform to the specified format below:

Music Detection: ''doMusicDetection path/to/file.wav path/to/output/file.mud ''

Music Relative Loudness Estimation: ''doMusicRelLoudEstimation path/to/file.wav path/to/output/file.mrle ''

where:
* path/to/file.wav: Path to the input audio file.
* path/to/output/file.*: Path to the output file.

Programs can use their working directory if they need to keep temporary cache files or internal debugging info. Stdout and stderr will be logged.

===I/O format===

For each detected segment, the file should include a row containing the onset (seconds), offset (seconds) and the class separated by a tab. Rows should be ordered by onset time:

''onset1 offset1 class1''
''onset2 offset2 class2''
''... ... ...''

(note that events in the case of music and speech detection can overlap)

===Packaging submissions===

All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed) and include a README file including the following the information:
* Command line calling format for all executables and an example formatted set of commands
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Any required environments (and versions), e.g. python, java, bash, matlab.

== Potential Participants ==
name/email

Blai Meléndez-Catalán, bmelendez … bmat.com
----

==Time and hardware limits==

Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions are specified.
A hard limit of 72 hours will be imposed on runs. Submissions that exceed this runtime may not receive a result.

==Submission closing date==

September 30th 2019

==Task specific mailing list==

All discussions on this task will take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have a question or comment, simply include the task name in the subject heading.

2021:Multiple Fundamental Frequency Estimation & Tracking

2021-09-10T19:53:17Z

Djevans: Created page with "==Description== That a complex music signal can be represented by the F0 contours of its constituent sources is a very useful concept for most music information retrieval sys..."

==Description==

That a complex music signal can be represented by the F0 contours of its constituent sources is a very useful concept for most music information retrieval systems. There have been many attempts at multiple (aka polyphonic) F0 estimation and melody extraction, a related area. The goal of multiple F0 estimation and tracking is to identify the active F0s in each time frame and to track notes and timbres continuously in a complex music signal. In this task, we would like to evaluate state-of-the-art multiple-F0 estimation and tracking algorithms. Since F0 tracking of all sources in a complex audio mixture can be very hard, we are restricting the problem to 3 cases:

# Estimate active fundamental frequencies on a frame-by-frame basis.
# Track note contours on a continuous time basis. (as in audio-to-midi). This task will also include a piano transcription sub task.
# Track timbre on a continous time basis.

=== Task specific mailing list ===
In the past we have use a specific mailing list for the discussion of this task and related tasks. This year, however, we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.

==Data==
=== MIREX Dataset ===
The 2009 Multi-F0 dataset will be reused, which is composed of:
* A woodwind quintet transcription of the fifth variation from L. van Beethoven's Variations for String Quartet Op.18 No. 5. Each part (flute, oboe, clarinet, horn, or bassoon) was recorded separately while the performer listened to the other parts (recorded previously) through headphones. Later the parts were mixed to a monaural 44.1kHz/16bits file.
* Synthesized pieces using RWC MIDI and RWC samples. Includes pieces from Classical and Jazz collections. Polyphony changes from 1 to 4 sources.
* Polyphonic piano recordings generated using a disklavier playback piano.

There are:
* 6, 30-sec clips for each polyphony (2-3-4-5) for a total of 30 examples,
* 10 30-sec polyphonic piano clips.

=== Development Dataset ===
A development dataset can be found at:
[https://www.music-ir.org/evaluation/MIREX/data/2007/multiF0/index.htm Development Set for MIREX 2007 MultiF0 Estimation Tracking Task].

In order to get the development set, we request you fill out the [https://goo.gl/forms/seCGIPYlwBMNBeg63 request form] here to write a short statement about the data being used only for research. After filling out the form, you will receive an email from IMIRSEL with username and password included.

=== Su Dataset ===
Last year a newly annotated polyphonic dataset was proposed. This dataset contains a wider range of real-world music in comparison to the old dataset used from 2009. Specifically, the new dataset contains 3 clips of piano solo, 3 clips of string quartet, 2 clips of piano quintet, and 2 clips of violin sonata (violin with piano accompaniment), all of which are selected from real-world recordings. The length of each clip is between 20 and 30 seconds. The dataset is annotated by the method described in the following paper:

Li Su and Yi-Hsuan Yang, "Escaping from the Abyss of Manual Annotation: New Methodology of Building Polyphonic Datasets for Automatic Music Transcription," in Int. Symp. Computer Music Multidisciplinary Research (CMMR), June 2015.

As also mentioned in the paper, we tried our best to calibrate the errors (mostly the mismatch between onset and offset time stamps) in the preliminary annotation by human labor. Since there are still potential errors of annotation that we didn’t find, we decide to make the data and the annotation publicly available after the announcement of MIREX result this year. Specifically, we encourage every participant to help us check the annotation. The result of each competing algorithm will be updated based on the revised annotation. We hope that this can let the participants get more detailed information about the behaviors of the algorithm performing on the dataset. Moreover, in this way we can join our efforts to create a better dataset for the research on multiple-F0 estimation and tracking.

==Evaluation==

This year, We would like to discuss different evaluation methods. From last year`s result, it can be seen that on note tracking, algorithms performed poorly when evaluated using note offsets. Below is the evaluation methods we used last year:

For Task 1 (frame level evaluation), systems will report the number of active pitches every 10ms. Precision (the portion of correct retrieved pitches for all pitches retrieved for each frame) and Recall (the ratio of correct pitches to all ground truth pitches for each frame) will be reported. A Returned Pitch is assumed to be correct if it is within a half semitone (+ - 3%) of a ground-truth pitch for that frame. Only one ground-truth pitch can be associated with each Returned Pitch.
Also as suggested, an error score as described in [http://www.hindawi.com/GetArticle.aspx?doi=10.1155/2007/48317 Poliner and Ellis p.g. 5 ] will be calculated.
The frame level ground truth will be calculated by [http://www.ircam.fr/pcm/cheveign/sw/yin.zip YIN] and hand corrected.

For Task 2 (note tracking), again Precision (the ratio of correctly transcribed ground truth notes to the number of ground truth notes for that input clip) and Recall (ratio of correctly transcribed ground truth notes to the number of transcribed notes) will be reported. A ground truth note is assumed to be correctly transcribed if the system returns a note that is within a half semitone (+ - 3%) of that note AND the returned note`s onset is within a 100ms range( + - 50ms) of the onset of the ground truth note, and its offset is within 20% range of the ground truth note`s offset. Again, one ground truth note can only be associated with one transcribed note.

The ground truth for this task will be annotated by hand. An amplitude threshold relative to the file/instrument will be determined. Note onset is going to be set to the time where its amplitude rises higher than the threshold and the offset is going to be set to the the time where the note`s amplitude decays lower than the threshold. The ground truth is going to be set as the average F0 between the onset and the offset of the note.
In the case of legato, the onset/offset is going to be set to the time where the F0 deviates more than 3% of the average F0 through out the the note up to that point. There is not going to be any vibrato larger than a half semitone in the test data.

Different statistics can also be reported if agreed by the participants.

== Submission Format ==

=== Audio Format ===
The audio files are encoded as 44.1kHz / 16 bit WAV files.

=== Command line calling format ===
Submissions have to conform to the specified format below:

''doMultiF0 "path/to/file.wav" "path/to/output/file.F0" ''

where:
* path/to/file.wav: Path to the input audio file.
* path/to/output/file.F0: The output file.

Programs can use their working directory if they need to keep temporary cache files or internal debuggin info. Stdout and stderr will be logged.

=== I/O format ===
For each task, the format of the output file is going to be different:

For the first task, F0-estimation on frame basis, the output will be a file where each row has a time stamp and a number of active F0s in that frame, separated by a tab for every 10ms increments.

Example :
''time F01 F02 F03 ''
''time F01 F02 F03 F04''
''time ... ... ... ...''

which might look like:

''0.78 146.83 220.00 349.23''
''0.79 349.23 146.83 369.99 220.00 ''
''0.80 ... ... ... ...''

For the second task, for each row, the file should contain the onset, offset and the F0 of each note event separated by a tab, ordered in terms of onset times:

onset offset F01
onset offset F02
... ... ...

which might look like:

0.68 1.20 349.23
0.72 1.02 220.00
... ... ...

=== Packaging submissions ===
All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guarenteed).

All submissions should include a README file including the following the information:

* Command line calling format for all executables and an example formatted set of commands
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Any required environments (and versions), e.g. python, java, bash, matlab.

== Time and hardware limits ==
Due to the potentially high number of particpants in this and other audio tasks,
hard limits on the runtime of submissions are specified.

A hard limit of 24 hours will be imposed on runs. Submissions that exceed this runtime may not receive a result.

== Submission opening date ==

== Submission closing date ==

== Potential Participants ==

2021:Drum Transcription

2021-09-10T19:52:45Z

Djevans: Created page with "==Description== Drum transcription is the task of detecting the positions in time and labeling the drum class of drum instrument onsets in polyphonic music. This information..."

==Description==

Drum transcription is the task of detecting the positions in time and labeling the drum class of drum instrument onsets in polyphonic music.
This information is a prerequisite for several applications and can also be used for other high-level MIR tasks.
Due to several new approaches recently being presented we propose to reintroduce this task.
We will mainly stick to the mode used in the first edition in 2005, but new datasets will be used.
Only the three main drum instruments of drum kits for western pop music are considered.
These are: bass drum, snare drum, and hi-hat (in all variations like open, closed, pedal, etc.).

==Data==

For evaluation 6 (5) different datasets will be used.
By the time the evaluation is run, we hope to have the three datasets from the 2005 drum detection MIREX task as a baseline.
Currently we only have the set provided by Christian Dittmar and Koen Tanghe.

* CD set
* KT set
* (GM set)

Additionally three new datasets will be used. They contain polyphonic music of different genres, as well as drum only tracks, and some tracks without drums:
* RBMA set (35 full length, polyphonic tracks, electronically produced and recorded, manually annotated and double-checked)
* MEDLEY set (23 full length tracks, recorded, manually annotated and double-checked)
* GEN set (synthesized MIDI drum tracks and loops without accompaniment)

===Audio Format===

The input for this task is a set of sound files adhering to the format and content requirements mentioned below.

* All audio is 44100 Hz, 16-bit mono, WAV PCM
* All available sound files will be used in their entirety (which can be short excerpts of 30s or full length music tracks of up to 7m)
* Some sound files will be recorded polyphonic music with drums (might be live performances or studio recordings)
* Some sound files will be rendered audio of MIDI files
* Some sound files may not contain any drums
* Both drums mixed with music and solo drums, will be part of the set
* Tracks with only the three drum instruments (or less) as well as tracks with full drum kits (with instruments not expected to be transcribed) will be part of the set
* Drum kit sounds will have a broad range: from natural recorded kits, live kits to sampled drums as well as electronic synthesizers

===Training Data===

* A representative random subset of the data will be made available to all participants in advance of the evaluation - please contact the task captains!
* Training data can be used by the participants as they please.
* Training data will not be used again during the evaluation.
* Usage of additional training data is discouraged. If additional training data is used, please note so in the submission.

==I/O format==

The input will be a directory containing audio files in the audio format specified above.
There might be other files in the directory, so make sure to filter for ‘*.wav’ files.

The output will also be a directory.
The algorithm is expected to process every file and generate an individual *.txt output file for every wav file with the same name.
e.g.:
input:
audio_file_10.wav
output:
audio_file_10.txt

For transcription three drum instrument types are considered:
BD 0 bass drum
SD 1 snare drum
HH 2 hi-hat (any hi-hat like open, half-open, closed, ...)

Drum types are strictly these types only (so: no ride cymbals in the HH, no toms in the BD, no claps nor side sticks/rim shots in the SD, etc...)
This involves the following remapping from other labels to these 3 base labels:

name midi label code
bass drum 36 KD 0
snare drum 38 SD 1
closed hi-hat 42 HH 2
open hi-hat 46 HH 2
pedal hi-hat 44 HH 2
cowbell 56
ride bell 53
low floor tom 41
high floor tom 43
low tom 45
low-mid tom 47
high-mid tom 48
high tom 50
side stick 37
hand clap 39
ride cymbal 51
crash cymbal 49
splash cymbal 55
chinese cymbal 52
shaker, maracas 70
tambourine 54
claves, sticks 75

All annotations are remapped to these three labels in advance (no looking back to the broader labels afterwards).

The annotation files as well as the expected output of the algorithms will have the following format:
A text file (UTF-8 encoding) with no header and footer, one line represents an instrument onset with the following format:
<TTT.TTT> \t <LL> \n
Where <TTT.TTT> is a floating point number with 3 decimals (ms accuracy), followed by a tab and <LL> the label for drum instrument onset as defined above (either number, or string), followed by a newline.
If multiple onsets occur at the exact same time, two separate lines with the same timestamp are expected.

Example of the content of a output file:

[test_file_0.txt]
<start-of-file>
0.125 0
0.125 2
0.250 2
0.375 1
0.375 2
0.500 2
0.625 0
0.625 2
0.750 2
0.875 1
0.875 2
1.000 2
<end-of-file>

Annotation files for the public subset will have the same format

==Evaluation==

* F-measure (harmonic mean of the recall rate and the precision rate, beta parameter 1, so equal importance to prec. and recall) is calculated for each of three drum types (BD, SD, and HH), resulting in three F-measure scores.
* Additionally a total F-measure score for all onsets over all instrument classes will be calculated.
* Calculation time measure: the time it takes to do the complete run from the moment your algorithm starts until the moment it stops will be reported

Evaluation parameters:
* The limit of onset-deviation errors in calculating the above F-measure is 30 ms (so a range of [-30 ms, +30 ms] around the true times)
* Any parameter adaptation (e.g. for peak picking) must be done on public data, i.e. in advance.

Conditions:
* The actual drum sounds (sound samples) used in any of the input audio are not public and not used for training.
* Participants are encouraged to only use the provided training data for training and parameter optimization.
If this is not possible, it should explicitly be stated that and which additional data was used. In this case it would be favorable to submit two versions: one trained with the public data only, and one trained using additional data.

==Packaging submissions==

* Participants only send in the application part of their algorithm, not the training part (if there is one)
* Algorithms must adhere to the specifications on the MIREX web page

===Command line calling format===

Python:

python <your_script_name.py> -i <inputfolder> -o <outputfolder>

Matlab:

<path_to_matlab>\matlab.exe" -nodisplay -nosplash -nodesktop -r "try, <your_script_name>(<inputfolder>, <outputfolder>), catch me, fprintf('%s / %s\n',me.identifier,me.message), end, exit"

Sonic Annotator:

[TODO]

===Time, Software and Hardware limits===

Max runtime: [TODO]

Software: Preferred Python. May be Matlab, Sonic Annotator.

==Submission closing date==
TBD