2016:Multiple Fundamental Frequency Estimation & Tracking Results - MIREX Dataset

2019-06-17T16:50:16Z

Daylin Troxel:

==Introduction==
These are the results for the 2016 running of the Multiple Fundamental Frequency Estimation and Tracking task on MIREX dataset. For background information about this task set please refer to the [[2015:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! KB1 (with bugs)
| Conv_Piano_Transcriptor_2016 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/KB1.pdf PDF] || [http://www.cp.jku.at Rainer Kelz], [http://www.cp.jku.at Sebastian Böck]
|-
! MM1
| Sonic || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/MM1.pdf PDF] || [http://lgm.fri.uni-lj.si Matija Marolt]
|-
! CB1
| Silvet || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/CB1.pdf PDF] || [http://www.eecs.qmul.ac.uk/people/view/3111/chris-cannam Chris Cannam], [http://www.soi.city.ac.uk/~sbbj660/ Emmanouil Benetos]
|-
! CB2
| Silvet Live || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/CB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/people/view/3111/chris-cannam Chris Cannam], [http://www.soi.city.ac.uk/~sbbj660/ Emmanouil Benetos]
|-
|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

====Detailed Results====

<csv>2016/mf0/est_mirex/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2016/mf0/est_mirex/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''MM1'''= [https://music-ir.org/mirex/results/2016/mf0/est_mirex/MM1.tar.gz Matija Marolt] 
'''CB1'''= [https://music-ir.org/mirex/results/2016/mf0/est_mirex/CB1.tar.gz Chris Cannam, Emmanouil Benetos] 
'''CB2'''= [https://music-ir.org/mirex/results/2016/mf0/est_mirex/CB2.tar.gz Chris Cannam, Emmanouil Benetos] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2016/mf0/est_mirex/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2016_Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2016/mf0/nt_mirex/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2016/mf0/nt_mirex/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2016/mf0/nt_mirex/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2016/mf0/nt_mirex/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2016/mf0/nt_mirex/summary/task2.onsetOnly.chroma.results.csv</csv>

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2016/mf0/nt_mirex/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2016_Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_Mean_Ranks.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2016/mf0/nt_mirex/summary_piano_subtask/task2_piano.overall.results.csv</csv>

*Submissions marked by asterisk are with bugs

====Detailed Results====

<csv>2016/mf0/nt_mirex/summary_piano_subtask/task2_piano.results.csv</csv>

*Submissions marked by asterisk are with bugs

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2016/mf0/nt_mirex/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

*Submissions marked by asterisk are with bugs

====Results Based on Onset Only====

<csv>2016/mf0/nt_mirex/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

*Submissions marked by asterisk are with bugs

====Chroma Results Based on Onset Only====

<csv>2016/mf0/nt_mirex/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

*Submissions marked by asterisk are with bugs

===Individual Results Files for Task 2===
'''KB1'''= [https://music-ir.org/mirex/results/2016/mf0/nt_mirex/KB1.tar.gz Rainer Kelz, Sebastian Böck] 
'''MM1'''= [https://music-ir.org/mirex/results/2016/mf0/nt_mirex/MM1.tar.gz Matija Marolt] 
'''CB1'''= [https://music-ir.org/mirex/results/2016/mf0/nt_mirex/CB1.tar.gz Chris Cannam, Emmanouil Benetos] 
'''CB2'''= [https://music-ir.org/mirex/results/2016/mf0/nt_mirex/CB2.tar.gz Chris Cannam, Emmanouil Benetos]

MIREX Wiki - User contributions [en]

2016:Multiple Fundamental Frequency Estimation & Tracking Results - MIREX Dataset