MIREX Wiki - User contributions [en]

2013:Multiple Fundamental Frequency Estimation & Tracking Results

2013-10-30T06:59:09Z

MertBay: /* Tukey-Kramer HSD Multi-Comparison for Task2 */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2012:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! CPG1
| Zhuo&DAn_MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG1.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG2
| CPG2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG2.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG3
| CPG3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG3.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://perso.telecom-paristech.fr/~grichard/ Graham Grindlay]
|-
! FBR1
| FBR1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR1.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FBR2
| FBR || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR2.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FT1
| TAVTZAN || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FT1.pdf PDF] || [http://www.dca.fee.unicamp.br/~tavares Tiago Fernandes Tavares], [http://www.cs.uvic.ca/~gtzan George Tzanetakis]
|-
! KD1
| multif0 2012a || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD2
| multi f0 2012b || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD2.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD3
| multif0 2012c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD3.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! SB5
| PianoTranscriptor || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SB5.pdf PDF] || [http://www.cp.jku.at Sebastian Böck ]
|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2013/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2013/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/FBR1.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''KD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''KD2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD2.tar.gz Karin Dressler ] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

<csv>2013/mf0/est/runtimes_mf0_2011.csv</csv>

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2013/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2013_Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2013/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2013/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2013/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2013/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Run Times====

TBD

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2013/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2013_Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_Mean_Ranks.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/FBR2.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''FT1'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/FT1.tar.gz Candidate Tiago Fernandes, George Tzanetakis] 
'''KD3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/KD3.tar.gz Karin Dressler ] 
'''SB5'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/SB5.tar.gz Sebastian Böck ]

2013:Multiple Fundamental Frequency Estimation & Tracking Results

2013-10-30T06:58:03Z

MertBay: /* Tukey-Kramer HSD Multi-Comparison for Task2 */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2012:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! CPG1
| Zhuo&DAn_MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG1.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG2
| CPG2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG2.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG3
| CPG3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG3.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://perso.telecom-paristech.fr/~grichard/ Graham Grindlay]
|-
! FBR1
| FBR1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR1.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FBR2
| FBR || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR2.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FT1
| TAVTZAN || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FT1.pdf PDF] || [http://www.dca.fee.unicamp.br/~tavares Tiago Fernandes Tavares], [http://www.cs.uvic.ca/~gtzan George Tzanetakis]
|-
! KD1
| multif0 2012a || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD2
| multi f0 2012b || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD2.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD3
| multif0 2012c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD3.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! SB5
| PianoTranscriptor || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SB5.pdf PDF] || [http://www.cp.jku.at Sebastian Böck ]
|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2013/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2013/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/FBR1.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''KD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''KD2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD2.tar.gz Karin Dressler ] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

<csv>2013/mf0/est/runtimes_mf0_2011.csv</csv>

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2013/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2013_Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2013/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2013/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2013/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2013/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Run Times====

TBD

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2013/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2013_Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_Mean_Ranks.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/FBR2.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''FT1'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/FT1.tar.gz Candidate Tiago Fernandes, George Tzanetakis] 
'''KD3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/KD3.tar.gz Karin Dressler ] 
'''SB5'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/SB5.tar.gz Sebastian Böck ]

2013:Multiple Fundamental Frequency Estimation & Tracking Results

2013-10-30T06:56:24Z

MertBay: /* Tukey-Kramer HSD Multi-Comparison */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2012:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! CPG1
| Zhuo&DAn_MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG1.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG2
| CPG2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG2.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG3
| CPG3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG3.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://perso.telecom-paristech.fr/~grichard/ Graham Grindlay]
|-
! FBR1
| FBR1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR1.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FBR2
| FBR || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR2.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FT1
| TAVTZAN || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FT1.pdf PDF] || [http://www.dca.fee.unicamp.br/~tavares Tiago Fernandes Tavares], [http://www.cs.uvic.ca/~gtzan George Tzanetakis]
|-
! KD1
| multif0 2012a || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD2
| multi f0 2012b || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD2.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD3
| multif0 2012c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD3.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! SB5
| PianoTranscriptor || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SB5.pdf PDF] || [http://www.cp.jku.at Sebastian Böck ]
|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2013/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2013/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/FBR1.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''KD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''KD2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD2.tar.gz Karin Dressler ] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

<csv>2013/mf0/est/runtimes_mf0_2011.csv</csv>

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2013/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2013_Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2013/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2013/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2013/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2013/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Run Times====

TBD

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2013/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2013_Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_Mean_Ranks.png.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/FBR2.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''FT1'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/FT1.tar.gz Candidate Tiago Fernandes, George Tzanetakis] 
'''KD3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/KD3.tar.gz Karin Dressler ] 
'''SB5'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/SB5.tar.gz Sebastian Böck ]

2013:Multiple Fundamental Frequency Estimation & Tracking Results

2013-10-30T06:52:21Z

MertBay: /* Tukey-Kramer HSD Multi-Comparison */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2012:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! CPG1
| Zhuo&DAn_MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG1.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG2
| CPG2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG2.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG3
| CPG3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG3.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://perso.telecom-paristech.fr/~grichard/ Graham Grindlay]
|-
! FBR1
| FBR1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR1.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FBR2
| FBR || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR2.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FT1
| TAVTZAN || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FT1.pdf PDF] || [http://www.dca.fee.unicamp.br/~tavares Tiago Fernandes Tavares], [http://www.cs.uvic.ca/~gtzan George Tzanetakis]
|-
! KD1
| multif0 2012a || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD2
| multi f0 2012b || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD2.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD3
| multif0 2012c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD3.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! SB5
| PianoTranscriptor || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SB5.pdf PDF] || [http://www.cp.jku.at Sebastian Böck ]
|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2013/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2013/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/FBR1.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''KD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''KD2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD2.tar.gz Karin Dressler ] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

<csv>2013/mf0/est/runtimes_mf0_2011.csv</csv>

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2013/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2013_Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2013/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2013/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2013/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2013/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Run Times====

TBD

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2013/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2013_Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_Mean_Ranks.png.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/FBR2.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''FT1'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/FT1.tar.gz Candidate Tiago Fernandes, George Tzanetakis] 
'''KD3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/KD3.tar.gz Karin Dressler ] 
'''SB5'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/SB5.tar.gz Sebastian Böck ]

File:2013 Accuracy Per Song Friedman Mean Rankstask2.onsetOnly.friedman.Friedman Mean Ranks.png

2013-10-30T06:51:11Z

MertBay:

2013:Multiple Fundamental Frequency Estimation & Tracking Results

2013-10-30T06:50:32Z

MertBay: /* Tukey-Kramer HSD Multi-Comparison for Task2 */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2012:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! CPG1
| Zhuo&DAn_MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG1.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG2
| CPG2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG2.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG3
| CPG3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG3.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://perso.telecom-paristech.fr/~grichard/ Graham Grindlay]
|-
! FBR1
| FBR1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR1.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FBR2
| FBR || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR2.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FT1
| TAVTZAN || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FT1.pdf PDF] || [http://www.dca.fee.unicamp.br/~tavares Tiago Fernandes Tavares], [http://www.cs.uvic.ca/~gtzan George Tzanetakis]
|-
! KD1
| multif0 2012a || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD2
| multi f0 2012b || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD2.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD3
| multif0 2012c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD3.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! SB5
| PianoTranscriptor || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SB5.pdf PDF] || [http://www.cp.jku.at Sebastian Böck ]
|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2013/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2013/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/FBR1.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''KD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''KD2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD2.tar.gz Karin Dressler ] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

<csv>2013/mf0/est/runtimes_mf0_2011.csv</csv>

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2013/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2013_Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2013/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2013/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2013/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2013/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Run Times====

TBD

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2013/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2013_Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_Mean_Ranks.png.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/FBR2.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''FT1'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/FT1.tar.gz Candidate Tiago Fernandes, George Tzanetakis] 
'''KD3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/KD3.tar.gz Karin Dressler ] 
'''SB5'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/SB5.tar.gz Sebastian Böck ]

2013:Multiple Fundamental Frequency Estimation & Tracking Results

2013-10-30T06:46:07Z

MertBay: /* Task 2:Note Tracking (NT) */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2012:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! CPG1
| Zhuo&DAn_MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG1.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG2
| CPG2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG2.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG3
| CPG3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG3.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://perso.telecom-paristech.fr/~grichard/ Graham Grindlay]
|-
! FBR1
| FBR1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR1.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FBR2
| FBR || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR2.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FT1
| TAVTZAN || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FT1.pdf PDF] || [http://www.dca.fee.unicamp.br/~tavares Tiago Fernandes Tavares], [http://www.cs.uvic.ca/~gtzan George Tzanetakis]
|-
! KD1
| multif0 2012a || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD2
| multi f0 2012b || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD2.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD3
| multif0 2012c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD3.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! SB5
| PianoTranscriptor || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SB5.pdf PDF] || [http://www.cp.jku.at Sebastian Böck ]
|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2013/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2013/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/FBR1.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''KD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''KD2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD2.tar.gz Karin Dressler ] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

<csv>2013/mf0/est/runtimes_mf0_2011.csv</csv>

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2013/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2013_Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2013/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2013/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2013/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2013/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Run Times====

TBD

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2013/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2012Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_Mean_Ranks.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2013/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR2'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/FBR2.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''FT1'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/FT1.tar.gz Candidate Tiago Fernandes, George Tzanetakis] 
'''KD3'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/KD3.tar.gz Karin Dressler ] 
'''SB5'''= [https://music-ir.org/mirex/results/2013/mf0/nt/tars/SB5.tar.gz Sebastian Böck ]

File:2013 Accuracy Per Song Friedman Mean Rankstask1.friedman.Friedman Mean Ranks.png

2013-10-30T06:33:21Z

MertBay:

2013:Multiple Fundamental Frequency Estimation & Tracking Results

2013-10-30T06:32:03Z

MertBay: /* Tukey-Kramer HSD Multi-Comparison */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2012:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! CPG1
| Zhuo&DAn_MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG1.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG2
| CPG2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG2.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG3
| CPG3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG3.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://perso.telecom-paristech.fr/~grichard/ Graham Grindlay]
|-
! FBR1
| FBR1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR1.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FBR2
| FBR || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR2.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FT1
| TAVTZAN || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FT1.pdf PDF] || [http://www.dca.fee.unicamp.br/~tavares Tiago Fernandes Tavares], [http://www.cs.uvic.ca/~gtzan George Tzanetakis]
|-
! KD1
| multif0 2012a || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD2
| multi f0 2012b || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD2.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD3
| multif0 2012c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD3.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! SB5
| PianoTranscriptor || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SB5.pdf PDF] || [http://www.cp.jku.at Sebastian Böck ]
|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2013/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2013/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/FBR1.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''KD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''KD2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD2.tar.gz Karin Dressler ] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

<csv>2013/mf0/est/runtimes_mf0_2011.csv</csv>

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2013/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2013_Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2012/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2012/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2012/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2012/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2012/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Run Times====

TBD

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2012/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2012Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_Mean_Ranks.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR2'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/FBR2.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''FT1'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/FT1.tar.gz Candidate Tiago Fernandes, George Tzanetakis] 
'''KD3'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/KD3.tar.gz Karin Dressler ] 
'''SB5'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/SB5.tar.gz Sebastian Böck ]

2013:Multiple Fundamental Frequency Estimation & Tracking Results

2013-10-30T06:14:37Z

MertBay: /* Task 1: Multiple Fundamental Frequency Estimation (MF0E) */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2012:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! CPG1
| Zhuo&DAn_MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG1.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG2
| CPG2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG2.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG3
| CPG3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG3.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://perso.telecom-paristech.fr/~grichard/ Graham Grindlay]
|-
! FBR1
| FBR1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR1.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FBR2
| FBR || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR2.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FT1
| TAVTZAN || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FT1.pdf PDF] || [http://www.dca.fee.unicamp.br/~tavares Tiago Fernandes Tavares], [http://www.cs.uvic.ca/~gtzan George Tzanetakis]
|-
! KD1
| multif0 2012a || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD2
| multi f0 2012b || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD2.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD3
| multif0 2012c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD3.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! SB5
| PianoTranscriptor || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SB5.pdf PDF] || [http://www.cp.jku.at Sebastian Böck ]
|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2013/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2013/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2013/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/FBR1.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''KD1'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''KD2'''= [https://music-ir.org/mirex/results/2013/mf0/est/tars/KD2.tar.gz Karin Dressler ] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

<csv>2013/mf0/est/runtimes_mf0_2011.csv</csv>

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2013/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2012/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2012/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2012/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2012/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2012/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Run Times====

TBD

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2012/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2012Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_Mean_Ranks.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR2'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/FBR2.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''FT1'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/FT1.tar.gz Candidate Tiago Fernandes, George Tzanetakis] 
'''KD3'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/KD3.tar.gz Karin Dressler ] 
'''SB5'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/SB5.tar.gz Sebastian Böck ]

2013:Multiple Fundamental Frequency Estimation & Tracking Results

2013-10-30T06:13:23Z

MertBay: Created page with "==Introduction== These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set ple..."

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2012:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! CPG1
| Zhuo&DAn_MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG1.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG2
| CPG2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG2.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://www.ee.columbia.edu/~grindlay/ Graham Grindlay]
|-
! CPG3
| CPG3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/CPG3.pdf PDF] || [http://labrosa.ee.columbia.edu/ Zhuo Chen], [http://www.ee.columbia.edu/~dpwe/ Danile P.W. Ellis] [http://perso.telecom-paristech.fr/~grichard/ Graham Grindlay]
|-
! FBR1
| FBR1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR1.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FBR2
| FBR || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FBR2.pdf PDF] || [http://perso.telecom-paristech.fr/~fuentes/ Benoit Fuentes], [http://perso.telecom-paristech.fr/~rbadeau/ Roland Badeau] [http://www.ee.columbia.edu/~grindlay/ Gael Richard]
|-
! FT1
| TAVTZAN || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/FT1.pdf PDF] || [http://www.dca.fee.unicamp.br/~tavares Tiago Fernandes Tavares], [http://www.cs.uvic.ca/~gtzan George Tzanetakis]
|-
! KD1
| multif0 2012a || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD2
| multi f0 2012b || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD2.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! KD3
| multif0 2012c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/KD3.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! SB5
| PianoTranscriptor || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SB5.pdf PDF] || [http://www.cp.jku.at Sebastian Böck ]
|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2012/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2012/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2012/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1'''= [https://music-ir.org/mirex/results/2012/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2012/mf0/est/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2012/mf0/est/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2012/mf0/est/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR1'''= [https://music-ir.org/mirex/results/2012/mf0/est/tars/FBR1.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''KD1'''= [https://music-ir.org/mirex/results/2012/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''KD2'''= [https://music-ir.org/mirex/results/2012/mf0/est/tars/KD2.tar.gz Karin Dressler ] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

<csv>2012/mf0/est/runtimes_mf0_2011.csv</csv>

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2012/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2012/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2012/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2012/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2012/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2012/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Run Times====

TBD

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2012/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2012Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_Mean_Ranks.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2012/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''CPG1'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/CPG1.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG2'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/CPG2.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''CPG3'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/CPG3.tar.gz Zhuo Chen, Danile P.W. Ellis Graham Grindlay] 
'''FBR2'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/FBR2.tar.gz Benoit Fuentes, Roland Badeau Gael Richard] 
'''FT1'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/FT1.tar.gz Candidate Tiago Fernandes, George Tzanetakis] 
'''KD3'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/KD3.tar.gz Karin Dressler ] 
'''SB5'''= [https://music-ir.org/mirex/results/2012/mf0/nt/tars/SB5.tar.gz Sebastian Böck ]

2013:Task Captains

2013-06-13T12:46:39Z

MertBay:

In response to discussions at ISMIR 2012, we are prepared to improve the distribution of tasks for the upcoming MIREX 2013. To do so, we really need leaders to help us organize and run each task.

To volunteer to lead one or more tasks, please add your name in the "Captains" column.

What does it mean to lead a task?
* Update wiki pages as needed
* Communicate with submitters and troubleshooting submissions
* Execution and evaluation of submissions
* Publishing final results

Due to the proprietary nature of much of the data, the submission system, evaluation framework, and most of the datasets will continue to be hosted by IMIRSEL. However, we are prepared to provide access to task organizers to manage and run submissions on the IMIRSEL systems.

{| class="wikitable" style="margin-left: 20px"
!ID !! Task !! Captain(s)
|-
|abt
|[[2013:Audio Beat Tracking]]
|
|-
|ace
|[[2013:Audio Chord Estimation]]
|
|-
|act
|[[2013:Audio Classification (Train/Test) Tasks]]
|IMIRSEL
|-
|acs
|[[2013:Audio Cover Song Identification]]
|
|-
|akd
|[[2013:Audio Key Detection]]
|
|-
|ame
|[[2013:Audio Melody Extraction]]
|KETI
|-
|ams
|[[2013:Audio Music Similarity and Retrieval]]
|IMIRSEL
|-
|aod
|[[2013:Audio Onset Detection]]
|
|-
|ate
|[[2013:Audio Tempo Estimation]]
|
|-
|atg
|[[2013:Audio Tag Classification]]
|
|-
|mf0
|[[2013:Multiple Fundamental Frequency Estimation & Tracking]]
|Mert Bay
|-
|qbsh
|[[2013:Query by Singing/Humming]]
|KETI
|-
|qbt
|[[2013:Query by Tapping]]
|KETI
|-
|scofo
|[[2013:Real-time Audio to Score Alignment (a.k.a Score Following)]]
|
|-
|sms
|[[2013:Symbolic Melodic Similarity]]
|IMIRSEL
|-
|struct
|[[2013:Structural Segmentation]]
|
|-
|drts
|[[2013:Discovery of Repeated Themes & Sections]]
|Tom Collins?
|}

2012:Audio Chord Estimation

2012-08-09T07:12:31Z

MertBay: /* Time and hardware limits */

[[The Utrecht Agreement on Chord Evaluation]]

===Evaluation of Chord Transcriptions===

Before the final description of the chord evaluation goes live here, please see the discussion based on the [[The Utrecht Agreement on Chord Evaluation]].

== Description ==
This task requires participants to extract or transcribe a sequence of chords from an audio music recording. For many applications in music information retrieval, extracting the harmonic structure of an audio track is very desirable, for example for segmenting pieces into characteristic segments, for finding similar pieces, or for semantic analysis of music.

The extraction of the harmonic structure requires the detection of as many chords as possible in a piece. That includes the characterisation of chords with a key and type as well as a chronological order with onset and duration of the chords.

Although some publications are available on this topic [1,2,3,4,5], comparison of the results is difficult, because different measures are used to assess the performance. To overcome this problem an accurately defined methodology is needed. This includes a repertory of the findable chords, a defined test set along with ground truth and unambiguous calculation rules to measure the performance.

== Data ==
Two datasets are used to evaluate chord transcription accuracy:

=== Beatles dataset ===
Christopher Harte`s Beatles dataset consisting of annotations of 12 Beatles albums.

The text annotation procedure of musical chords that was used to produce this dataset is presented in [6].

=== Queen and Zweieck dataset ===
Matthias Mauch's Queen and Zweieck dataset consisting of 38 songs from Queen and Zweieck.

===Example ground-truth file ===
The ground-truth files take the form:

...
41.2631021 44.2456460 B
44.2456460 45.7201230 E
45.7201230 47.2061900 E:7/3
47.2061900 48.6922670 A
48.6922670 50.1551240 A:min/b3
...

== Evaluation ==

=== Segmentation Score ===

The segmentation score will be calculated using directional hamming distance as described in [8]. An over-segmentation value (m) and an under-segmentation value (f) will be calculated and the final segmentation score will be calculated using the worst case from these two i.e:

segmentation score = 1 - max(m,f)

m and f are not independent of each other so combining them this way ensures that a good score in one does not hide a bad score in the other. The combined segmentation score can take values between 0 and 1 with 0 being the worst and 1 being the best result.-- Chrish 17:05, 9 September 2009 (UTC)

=== Frame-based recall ===

For recall evaluation, we may define a different chord dictionary for each level of evaluation (dyads, triads, tetrads etc). Each dictionary is a text file containing chord shorthands / interval lists of the chords that will be considered in that evaluation. The following dictionaries are proposed:

For dyad comparison of major/minor chords only:

N 
X:maj 
X:min 

For comparison of standard triad chords:

N 
X:maj 
X:min 
X:aug 
X:dim 
X:sus2 
X:sus4 

For comparison of tetrad (quad) chords:

N 
X:maj 
X:min 
X:aug 
X:dim 
X:sus2 
X:sus4 
X:maj7 
X:7 
X:maj(9) 
X:aug(7) 
X:min(7) 
X:min7 
X:min(9) 
X:dim(7) 
X:hdim7 
X:sus4(7) 
X:sus4(b7) 
X:dim7 

For each evaluation level, the ground truth annotation is compared against the dictionary. Any chord label not belonging to the current dictionary will be replaced with an "X" in a local copy of the annotation and will not be included in the recall calculation.

Note that the level of comparison in terms of intervals can be varied. For example, in a triad evaluation we can consider the first three component intervals in the chord so that a major (1,3,5) and a major7 (1,3,5,7) will be considered the same chord. For a tetrad (quad) evaluation, we would consider the first 4 intervals so major and major7 would then be considered to be different chords.

For the maj/min evaluation (using the first example dictionary), using an interval comparison of 2 (dyad) will compare only the first two intervals of each chord label. This would map augmented and diminished chords to major and minor respectively (and any other symbols that had a major 3rd or minor 3rd as their first interval). Using an interval comparison of 3 with the same dictionary would keep only those chords that have major and minor triads as their first 3 intervals so augmented and diminished chords would be removed from the evaluation.

After the annotation has been "filtered" using a given dictionary, it can be compared against the machine generated estimates output by the algorithm under test. The chord sequences described in the annotation and estimate text files are sampled at a given frame rate (in this case 10ms per frame) to give two sequences of chord frames which may be compared directly with each other. For calculating a hit or a miss, the chord labels from the current frame in each sequence will be compared. Chord comparison is done by converting each chord label into an ordered list of pitch classes then comparing the two lists element by element. If the lists match to the required number of intervals then a hit is recorded, otherwise the estimate is considered a miss. It should be noted that, by converting to pitch classes in the comparison, this evaluation ignores enharmonic pitch and interval spellings so the following chords (slightly silly example just for illustration) will all evaluate as identical:

C:maj = Dbb:maj = C#:(b1,b3,#4)

Basic recall calculation algorithm:

1) filter annotated transcription using chord dictionary for a defined number of intervals

2) sample annotated transcription and machine estimated transcription at 10ms intervals to create a sequence of annotation frames and estimate frames

3) start at the first frame

4) get chord label for current annotation frame and estimate frame

5) check annotation label: 

IF symbol is 'X' (i.e. non-dictionary) 

THEN ignore frame (record number of ignored frames) 

ELSE compare annotated/estimated chords for the predefined number of intervals 
increment hit count if chords match 

ENDIF

6) increment frame count

7) go back to 4 until final chord frame
--[[User:Chrish|Chrish]] 17:05, 9 September 2009 (UTC)

== Submission Format ==

=== Audio Format ===
Audio tracks will be encoded as 44.1 kHz 16bit mono WAV files.

=== I/O Format ===
The expected output chord transcription file for participating algorithms is that proposed by Christopher Harte [6].

Hence, algorithms should output text files with a similar format to that used in the ground truth transcriptions. That is to say, they should be flat text files with chord segment labels and times arranged thus:

start_time end_time chord_label

with elements separated by white spaces, times given in seconds, chord labels corresponding to the syntax described in [6] and one chord segment per line.

The chord root is given as a natural (A|B|C|D|E|F|G) followed by optional sharp or flat modifiers (#|b). For the evaluation process we may assume enharmonic equivalence for chord roots. For a given chord type on root X, the chord labels can be given as a list of intervals or as a shorthand notation as shown in the following table:

{|border="1" cellpadding="5" cellspacing="0" align="center"
|-
!NAME
!INTERVALS
!SHORTHAND
|-
|-*Triads:
|-
|-
|major
|X:(1,3,5)
|X or X:maj
|-
|-
|minor
|X:(1,b3,5)
|X:min
|-
|-
|diminished
|X:(1,b3,b5)
|X:dim
|-
|-
|augmented
|X:(1,3,#5)
|X:aug
|-
|-
|suspended4
|X:(1,4,5)
|X:sus4
|-
|-
|possible 6th triad:
|
|
|-
|-
|suspended2
|X:(1,2,5)
|X:sus2
|-
|-
|*Quads:
|
|
|-
|-
|major-major7
|X:(1,3,5,7)
|X:maj7
|-
|-
|major-minor7
|X:(1,3,5,b7)
|X:7
|-
|-
|major-add9
|X:(1,3,5,9)
|X:maj(9)
|-
|-
|major-major7-#5
|X:(1,3,#5,7)
|X:aug(7)
|-
|-
|minor-major7
|X:(1,b3,5,7)
|X:min(7)
|-
|-
|minor-minor7
|X:(1,b3,5,b7)
|X:min7
|-
|-
|minor-add9
|X:(1,b3,5,9)
|X:min(9)
|-
|-
|minor 7/b5 (ambiguous - could be either of the following)
|
|
|-
|-
|minor-major7-b5
|X:(1,b3,b5,7)
|X:dim(7)
|-
|-
|minor-minor7-b5 (a half diminished-7th)
|X:(1,b3,b5,b7)
|X:hdim7
|-
|-
|sus4-major7
|X:(1,4,5,7)
|X:sus4(7)
|-
|-
|sus4-minor7
|X:(1,4,5,b7)
|X:sus4(b7)
|-
|-
|omitted from list on wiki:
|
|
|-
|-
|diminished7
|X:(1,b3,b5,bb7)
|X:dim7
|-
|-
|No Chord
|N
|
|}

Please note that two things have changed in the syntax since it was originally described in [6]. The first change is that the root is no longer implied as a voiced element of a chord so a C major chord (notes C, E and G) should be written C:(1,3,5) instead of just C:(3,5) if using the interval list representation. As before, the labels C and C:maj are equivalent to C:(1,3,5). The second change is that the shorthand label "sus2" (intervals 1,2,5) has been added to the available shorthand list.--[[User:Chrish|Chrish]] 17:05, 9 September 2009 (UTC)

We still accept participants who would only like to be evaluated on major/minor chords and want to use the number format which is an integer chord id on range 0-24, where values 0-11 denote the C major, C# major, ..., B major and 12-23 denote the C minor, C# minor, ..., B minor and 24 denotes silence or no-chord segments. '''Please note that the format is still the same'''

start_time end_time chord_number

Systems are supposed to print out the onset-offset times as opposed to MIREX 2008 chord output format where only onset were used.

=== Command line calling format ===

Submissions have to conform to the specified format below:

''extractFeaturesAndTrain "/path/to/trainFileList.txt" "/path/to/scratch/dir" ''

Where fileList.txt has the paths to each wav file. The features extracted on this stage can be stored under "/path/to/scratch/dir"
The ground truth files for the supervised learning will be in the same path with a ".txt" extension at the end. For example for "/path/to/trainFile1.wav", there will be a corresponding ground truth file called "/path/to/trainFile1.wav.txt" .

For testing:

''doChordID.sh "/path/to/testFileList.txt" "/path/to/scratch/dir" "/path/to/results/dir" ''

If there is no training, you can ignore the second argument here. In the results directory, there should be one file for each testfile with same name as the test file + .txt .

Programs can use their working directory if they need to keep temporary cache files or internal debuggin info. Stdout and stderr will be logged.

=== Packaging submissions ===
All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed).

All submissions should include a README file including the following information:

* Command line calling format for all executables and an example formatted set of commands
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Any required environments (and versions), e.g. python, java, bash, matlab.

== Time and hardware limits ==
Due to the potentially high number of particpants in this and other audio tasks, hard limits on the runtime of submissions are specified.

A hard limit of 24 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.

== Discussion ==
Please write your comments below with your name and date.

== Potential Participants ==
name / email

== Bibliography ==

1. Harte,C.A. and Sandler,M.B.(2005). '''Automatic chord identification using a quantised chromagram.''' Proceedings of 118th Audio Engineering Society's Convention.

2. Sailer,C. and Rosenbauer K.(2006). '''A bottom-up approach to chord detection.''' Proceedings of International Computer Music Conference 2006.

3. Shenoy,A. and Wang,Y.(2005). '''Key, chord, and rythm tracking of popular music recordings.''' Computer Music Journal 29(3), 75-86.

4. Sheh,A. and Ellis,D.P.W.(2003). '''Chord segmentation and recognition using em-trained hidden markov models.''' Proceedings of 4th International Conference on Music Information Retrieval.

5. Yoshioka,T. et al.(2004). '''Automatic Chord Transcription with concurrent recognition of chord symbols and boundaries.''' Proceedings of 5th International Conference on Music Information Retrieval.

6. Harte,C. and Sandler,M. and Abdallah,S. and G├│mez,E.(2005). '''Symbolic representation of musical chords: a proposed syntax for text annotations.''' Proceedings of 6th International Conference on Music Information Retrieval.

7. Papadopoulos,H. and Peeters,G.(2007). '''Large-scale study of chord estimation algorithms based on chroma representation and HMM.''' Proceedings of 5th International Conference on Content-Based Multimedia Indexing.

8. Samer Abdallah, Katy Noland, Mark Sandler, Michael Casey & Christophe Rhodes: '''Theory and Evaluation of a Bayesian Music Structure Extractor''' (pp. 420-425) Proc. 6th International Conference on Music Information Retrieval, ISMIR 2005.

2012:MIREX Home

2012-06-08T16:33:14Z

MertBay: /* MIREX 2012 Task Participation Poll */

==Welcome to MIREX 2012==

This is the main page for the eighth running of the Music Information Retrieval Evaluation eXchange (MIREX 2012). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2012.

The MIREX 2012 community will hold its annual meeting as part of [http://ismir2012.ismir.net/ The 13th International Conference on Music Information Retrieval], ISMIR 2012, which will be held in Porto, Portugal, the 8-12 October, 2012. The MIREX plenary (working lunch) and poster sessions will be held Thursday, October 11 during the conference.

J. Stephen Downie 
Director, IMIRSEL 

==MIREX 2012 Deadline Dates==

We have two sets of deadlines for submissions. We have to stagger the deadlines because of runtime and human evaluation considerations. The submission system will open July 30, 2012.

===Tasks with a '''20 August 2012''' deadline:===
# Audio Classification (Train/Test) Tasks
# Audio Music Similarity and Retrieval
# Symbolic Melodic Similarity

===Tasks with a '''27 August 2012''' deadline:===

# All remaining MIREX 2012 tasks.

Nota Bene: In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.

Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.

PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.

==MIREX 2012 Submission Instructions==
* Be sure to read through the rest of this page
* Be sure to read though the task pages for which you are submitting
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[2012:MIREX 2012 Submission Instructions | MIREX 2012 Submission Instructions ]] including both the tutorial video and the text
* The submission system will open Monday, July 30, 2012

==MIREX 2012 Task Participation Poll==

Please answer [https://docs.google.com/spreadsheet/viewform?formkey=dFlrZlZIWUpGX3o0S0NCb3huVHBfeUE6MA#gid=0 MIREX 2012 Task Participation Poll]
on your likelihood of participation in each task.
Poll is going to close on Sunday July 29th 2012.

==MIREX 2012 Possible Evaluation Tasks==

* [[2012:Audio Classification (Train/Test) Tasks]], incorporating:
** Audio US Pop Genre Classification
** Audio Latin Genre Classification
** Audio Music Mood Classification
** Audio Classical Composer Identification
* [[2012:Audio Cover Song Identification]]
* [[2012:Audio Tag Classification]]
* [[2012:Audio Music Similarity and Retrieval]]
* [[2012:Symbolic Melodic Similarity]]
* [[2012:Audio Onset Detection]]
* [[2012:Audio Key Detection]]
* [[2012:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2012:Query by Singing/Humming]]
* [[2012:Audio Melody Extraction]]
* [[2012:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2012:Audio Chord Estimation]]
* [[2012:Query by Tapping]]
* [[2012:Audio Beat Tracking]]
* [[2012:Structural Segmentation]]
* [[2012:Audio Tempo Estimation]]

===Note to New Participants===
Please take the time to read the following review article that explains the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

Downie, J. Stephen, Andreas F. Ehmann, Mert Bay and M. Cameron Jones. (2010). 
The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. 
Advances in Music Information Retrieval Vol. 274, pp. 93-115 
Available at: [http://www.google.com/url?sa=t&rct=j&q=the%20music%20information%20retrieval%20evaluation%20exchange%3A%20some%20observations%20and%20insights&source=web&cd=1&ved=0CFQQFjAA&url=https%3A%2F%2Fkaggle2.blob.core.windows.net%2Fcompetitions%2Fkaggle%2F2799%2Fmedia%2FThe%2520Music%2520Information%2520Retrieval%2520Evaluation%2520eXchange-%2520Some%2520Observations%2520and%2520Insights.pdf&ei=UhvST4__Cq2H0QGm7qSnAw&usg=AFQjCNG0hMxcHm98GTu0gRX2X0AjwYYuow here]

===Note to All Participants===
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2012 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)
# present a poster at the MIREX 2012 poster session at ISMIR 2012

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://docs.google.com/spreadsheet/viewform?formkey=dFpmNF9PUGdvd1o1OHVhMkZ4cXZvdkE6MA#gid=0 dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you.

Due to the high volume of submissions expected at MIREX 2012, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2012==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2012 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2012 tasks. Subscription information at:
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central].

If you are participating in MIREX 2012, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2012 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2012 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
Please create an account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2011 Wikis==
Content from MIREX 2005 - 2011 are available at:

'''[[2010:Main_Page|MIREX 2011]]'''
'''[[2010:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

2012:MIREX Home

2012-06-08T16:32:24Z

MertBay: /* Software Dependency Requests */

==Welcome to MIREX 2012==

This is the main page for the eighth running of the Music Information Retrieval Evaluation eXchange (MIREX 2012). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2012.

The MIREX 2012 community will hold its annual meeting as part of [http://ismir2012.ismir.net/ The 13th International Conference on Music Information Retrieval], ISMIR 2012, which will be held in Porto, Portugal, the 8-12 October, 2012. The MIREX plenary (working lunch) and poster sessions will be held Thursday, October 11 during the conference.

J. Stephen Downie 
Director, IMIRSEL 

==MIREX 2012 Deadline Dates==

We have two sets of deadlines for submissions. We have to stagger the deadlines because of runtime and human evaluation considerations. The submission system will open July 30, 2012.

===Tasks with a '''20 August 2012''' deadline:===
# Audio Classification (Train/Test) Tasks
# Audio Music Similarity and Retrieval
# Symbolic Melodic Similarity

===Tasks with a '''27 August 2012''' deadline:===

# All remaining MIREX 2012 tasks.

Nota Bene: In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.

Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.

PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.

==MIREX 2012 Submission Instructions==
* Be sure to read through the rest of this page
* Be sure to read though the task pages for which you are submitting
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[2012:MIREX 2012 Submission Instructions | MIREX 2012 Submission Instructions ]] including both the tutorial video and the text
* The submission system will open Monday, July 30, 2012

==MIREX 2012 Task Participation Poll==

Please answer [https://spreadsheets.google.com/spreadsheet/viewform?authkey=CI-w_csM&hl=en_US&formkey=dG4ybGt1OUYtNHpnRF9RVm1GWjBxckE6MA#gid=0 MIREX 2012 Task Participation Poll]
on your likelihood of participation in each task.
Poll is going to close on Sunday July 29th 2012.

==MIREX 2012 Possible Evaluation Tasks==

* [[2012:Audio Classification (Train/Test) Tasks]], incorporating:
** Audio US Pop Genre Classification
** Audio Latin Genre Classification
** Audio Music Mood Classification
** Audio Classical Composer Identification
* [[2012:Audio Cover Song Identification]]
* [[2012:Audio Tag Classification]]
* [[2012:Audio Music Similarity and Retrieval]]
* [[2012:Symbolic Melodic Similarity]]
* [[2012:Audio Onset Detection]]
* [[2012:Audio Key Detection]]
* [[2012:Real-time Audio to Score Alignment (a.k.a Score Following)]]
* [[2012:Query by Singing/Humming]]
* [[2012:Audio Melody Extraction]]
* [[2012:Multiple Fundamental Frequency Estimation & Tracking]]
* [[2012:Audio Chord Estimation]]
* [[2012:Query by Tapping]]
* [[2012:Audio Beat Tracking]]
* [[2012:Structural Segmentation]]
* [[2012:Audio Tempo Estimation]]

===Note to New Participants===
Please take the time to read the following review article that explains the history and structure of MIREX.

Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007): 
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. 
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]

Downie, J. Stephen, Andreas F. Ehmann, Mert Bay and M. Cameron Jones. (2010). 
The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. 
Advances in Music Information Retrieval Vol. 274, pp. 93-115 
Available at: [http://www.google.com/url?sa=t&rct=j&q=the%20music%20information%20retrieval%20evaluation%20exchange%3A%20some%20observations%20and%20insights&source=web&cd=1&ved=0CFQQFjAA&url=https%3A%2F%2Fkaggle2.blob.core.windows.net%2Fcompetitions%2Fkaggle%2F2799%2Fmedia%2FThe%2520Music%2520Information%2520Retrieval%2520Evaluation%2520eXchange-%2520Some%2520Observations%2520and%2520Insights.pdf&ei=UhvST4__Cq2H0QGm7qSnAw&usg=AFQjCNG0hMxcHm98GTu0gRX2X0AjwYYuow here]

===Note to All Participants===
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:

# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2012 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)
# present a poster at the MIREX 2012 poster session at ISMIR 2012

===Software Dependency Requests===
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://docs.google.com/spreadsheet/viewform?formkey=dFpmNF9PUGdvd1o1OHVhMkZ4cXZvdkE6MA#gid=0 dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you.

Due to the high volume of submissions expected at MIREX 2012, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.

Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.

==Getting Involved in MIREX 2012==
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2012 the best yet.

===Mailing List Participation===
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2012 tasks. Subscription information at:
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central].

If you are participating in MIREX 2012, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2012 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here.

Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2012 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.

===Wiki Participation===
Please create an account via: [[Special:Userlogin]].

Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).

==MIREX 2005 - 2011 Wikis==
Content from MIREX 2005 - 2011 are available at:

'''[[2010:Main_Page|MIREX 2011]]'''
'''[[2010:Main_Page|MIREX 2010]]'''
'''[[2009:Main_Page|MIREX 2009]]'''
'''[[2008:Main_Page|MIREX 2008]]'''
'''[[2007:Main_Page|MIREX 2007]]'''
'''[[2006:Main_Page|MIREX 2006]]'''
'''[[2005:Main_Page|MIREX 2005]]'''

2011:MIREX2011 Results

2012-05-18T22:21:56Z

MertBay: /* Train-Test Task Set */

==OVERALL RESULTS POSTERS ==

[https://www.music-ir.org/mirex/results/2011/mirex_2011_poster.pdf MIREX 2011 Overall Results Posters (PDF)]

==Results by Task ==
This year we ran many MIREX 2011 Tasks using the new [http://nema.lis.uiuc.edu/drupal/?q=nema/architecture NEMA MIREX DIY] infrastructure. Task results with "(DIY)" appended are those generated using the NEMA MIREX DIY system. Where appropriate, do explore the various new outputs that help visualize both individual and task-wide comparative performances. A demonstration [https://www.music-ir.org/diy-demo/ video] of the NEMA MIREX DIY system can be found is also available.

===Train-Test Task Set===
* [https://www.music-ir.org/nema_out/mirex2011/results/act/composer_report/ Audio Classical Composer Identification Results ]   (DIY)
* [https://www.music-ir.org/nema_out/mirex2011/results/act/latin_report/ Audio Latin Genre Classification Results ]   (DIY)
* [https://www.music-ir.org/nema_out/mirex2011/results/act/mood_report/index.html Audio Music Mood Classification Results ]   (DIY)
* [https://www.music-ir.org/nema_out/mirex2011/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]   (DIY)

===Other Tasks===

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/abt/mck/ MCK Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/abt/maz/ MAZ Dataset]  (DIY)
* [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ace/ Audio Chord Detection]  (DIY)
* [[2011:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]]
* [https://nema.lis.illinois.edu/nema_out/mirex2011/results/akd/ Audio Key Detection Results]  (DIY)
* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/adc04/ ADC04 Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/mirex05/ MIREX05 Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/indian08/ INDIAN08 Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/mirex09_0dB/ MIREX09 0dB Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset]  (DIY)
* [[2011:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]
* [https://nema.lis.illinois.edu/nema_out/mirex2011/results/aod/ Audio Onset Detection Results]  (DIY)
* Audio Tag Classification Results
** Major Miner Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)]  (DIY)
*** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/atg/subtask1_report/aff/ Affinity estimation evaluation]  (DIY)
** Mood Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)]  (DIY)
*** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/atg/subtask2_report/aff/ Affinity estimation evaluation]  (DIY)
* [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ate/ Audio Tempo Estimation Results]  (DIY)
* [[2011:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]
* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/struct/mirex09/ MIREX09 dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/struct/mirex10/ MIREX10 dataset]  (DIY)
* [[2011:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]]
* [[2011:Query-by-Tapping_Results| Query-by-Tapping Results]]
*[[2011:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]
* [[2011:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]

== Machine Specifications ==
TBA

== Runtime for Submissions Run by NEMA DIY ==

TBA

[[Category:Results]]

2012:Audio Onset Detection

2012-05-18T19:28:32Z

MertBay: /* Submission closing date */

== Description ==

Audio Onset Detection concerns itself with finding the time-locations of all sonic events in a piece of audio. This task was originally proposed in 2005 by Paul Brossier and Pierre Leveau . It has subsequently been run in 2005, 2006, 2007, 2009, 2010, 2011.

== Data ==
=== Collections ===
The dataset will be the same as in 2005/2006/2007/2009 unless new or updated datasets are made available. The current dataset is subdivided into classes, because onset detection is sometimes performed in applications dedicated to a single type of signal (ex: segmentation of a single track in a mix, drum transcription, complex mixes databases segmentation...). The performance of each algorithm will be assessed on the whole dataset but also on each class separately.

The dataset contains 85 files from 5 classes annotated as follows:

* 30 solo drum excerpts cross-annotated by 3 people
* 30 solo monophonic pitched instruments excerpts cross-annotated by 3 people
* 10 solo polyphonic pitched instruments excerpts cross-annotated by 3 people
* 15 complex mixes cross-annotated by 5 people

Moreover the monophonic pitched instruments class is divided into 6 sub-classes: brass (2 excerpts), winds (4), sustained strings (6), plucked strings (9), bars and bells (4), singing voice (5).

=== Audio Formats ===

The data are monophonic sound files, with the associated onset times and data about the annotation robustness.

* CD-quality (PCM, 16-bit, 44100 Hz)
* single channel (mono)
* file length between 2 and 36 seconds (total time: 14 minutes)

== Evaluation Procedures ==

The detected onset times will be compared with the ground-truth ones. For a given ground-truth onset time, if there is a detection in a tolerance time-window around it, it is considered as a correct detection (CD). If not, there is a false negative (FN). The detections outside all the tolerance windows are counted as false positives (FP). Doubled onsets (two detections for one ground-truth onset) and merged onsets (one detection for two ground-truth onsets) will be taken into account in the evaluation. Doubled onsets are a subset of the FP onsets, and merged onsets a subset of FN onsets.

We define:

*'''Precision''' P = Ocd / (Ocd +Ofp)
*'''Recall''' R = Ocd / (Ocd + Ofn)
*'''F-measure''' F = 2*P*R/(P+R)

with these notations:

*'''Ocd''' number of correctly detected onsets (CD)
*'''Ofn''' number of missed onsets (FN)
*'''Om''' number of merged onsets
*'''Ofp''' number of false positive onsets (FP)
*'''Od''' number of double onsets

Other indicative measurements:

*'''FP rate''' FP = 100. * (Ofp) / (Ocd+Ofp)
*'''Doubled Onset rate in FP''' D = 100 * Od / Ofp
*'''Merged Onset rate in FN''' M = 100 * Om / Ofn

Because files are cross-annotated, the mean Precision and Recall rates are defined by averaging Precision and Recall rates computed for each annotation.

To establish a ranking, we will use the F-measure, widely used in string comparisons. This criterion is arbitrary, but gives an indication of performance. It must be remembered that onset detection is a preprocessing step, so the real cost of an error of each type (false positive or false negative) depends on the application following this task.

=== Evaluation measures ===

* percentage of correct detections / false positives (can also be expressed as precision/recall)
* time precision (tolerance from +/- 50 ms to less). For certain file, we can't be much more accurate than 50 ms because of the weak annotation precision. This must be taken into account.
* separate scoring for different instrument types (percussive, strings, winds, etc)
* percentage of doubled detections
* speed measurements of the algorithms

== Submission Format ==
Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.

=== Input Data ===
Participating algorithms will have to read audio in the following format:

* Sample rate: 44.1 KHz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

=== Output Data ===

The onset detection algorithms will return onset times in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.

==== Output File Format (Audio Onset Detection) ====

The Audio Onset Detection output file format is an ASCII text format. Each onset time is specified, in seconds, on its own line. Specifically,

<onset time(in seconds)>\n

where \n denotes the end of line. The < and > characters are not included. An example output file would look something like:

0.243
1.476
1.987
2.449
3.224

=== Algorithm Calling Format ===

The submitted algorithm must take as arguments a SINGLE .wav file to perform the onset detection on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:

foobar %input %output
foobar -i %input -o %output

Moreover, if your submission takes additional parameters, such as a detection threshold, foobar could be called like:

foobar .1 %input %output
foobar -param1 .1 -i %input -o %output

If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example:

foobar('%input','%output')
foobar(.1,'%input','%output')

=== Parameter Sweeps ===
In past iterations of MIREX, submitters have been allowed to specify a parameter sweep so as to generate a precision-recall operator characteristic to better evaluate and understand the algorithm. If you wish to do so, please specify TEN different settings for your sweepable parameter. There are no guarantees that all ten will be tested and evaluated, however, as the time-constraints for MIREX are getting ever smaller as the number of submissions are getting ever larger. Therefore, please also specify the ONE single parameterization you feel is best in the README. If the whole parameter sweep cannot be evaluated, this single parameterization will be used.

=== Packaging submissions ===

* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2012 Submission Instructions]]

All submissions should include a README file including the following the information:

* Command line calling format for all executables including examples
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Approximately how much scratch disk space will the submission need to store any feature/cache files?
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.
* Any special notice regarding to running your algorithm

Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.

==== README File ====

A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.

For instance, to test the program foobar with different values for parameters param1, the README file would look like:

foobar -param1 .1 -i %input -o %output
foobar -param1 .15 -i %input -o %output
foobar -param1 .2 -i %input -o %output
foobar -param1 .25 -i %input -o %output
foobar -param1 .3 -i %input -o %output
...

For a submission using MATLAB, the README file could look like:

matlab -r "foobar(.1,'%input','%output');quit;"
matlab -r "foobar(.15,'%input','%output');quit;"
matlab -r "foobar(.2,'%input','%output');quit;"
matlab -r "foobar(.25,'%input','%output');quit;"
matlab -r "foobar(.3,'%input','%output');quit;"
...

The different command lines to evaluate the performance of each parameter set over the whole database will be generated automatically from each line in the README file containing both '%input' and '%output' strings.

== Time and hardware limits ==
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.

A hard limit of 6 hours will be imposed on analysis times.

== Submission opening date ==

Friday 5th August 2012

== Submission closing date ==
September 2nd August 2012

2012:Audio Onset Detection

2012-05-18T19:28:19Z

MertBay: /* Description */

2012:Audio Onset Detection

2012-05-18T19:27:17Z

MertBay: /* Submission closing date */

== Description ==

Audio Onset Detection concerns itself with finding the time-locations of all sonic events in a piece of audio. This task was originally proposed in 2005 by Paul Brossier and Pierre Leveau . It has subsequently been run in 2005, 2006, 2007, 2009.

== Data ==
=== Collections ===
The dataset will be the same as in 2005/2006/2007/2009 unless new or updated datasets are made available. The current dataset is subdivided into classes, because onset detection is sometimes performed in applications dedicated to a single type of signal (ex: segmentation of a single track in a mix, drum transcription, complex mixes databases segmentation...). The performance of each algorithm will be assessed on the whole dataset but also on each class separately.

The dataset contains 85 files from 5 classes annotated as follows:

* 30 solo drum excerpts cross-annotated by 3 people
* 30 solo monophonic pitched instruments excerpts cross-annotated by 3 people
* 10 solo polyphonic pitched instruments excerpts cross-annotated by 3 people
* 15 complex mixes cross-annotated by 5 people

Moreover the monophonic pitched instruments class is divided into 6 sub-classes: brass (2 excerpts), winds (4), sustained strings (6), plucked strings (9), bars and bells (4), singing voice (5).

=== Audio Formats ===

The data are monophonic sound files, with the associated onset times and data about the annotation robustness.

* CD-quality (PCM, 16-bit, 44100 Hz)
* single channel (mono)
* file length between 2 and 36 seconds (total time: 14 minutes)

== Evaluation Procedures ==

The detected onset times will be compared with the ground-truth ones. For a given ground-truth onset time, if there is a detection in a tolerance time-window around it, it is considered as a correct detection (CD). If not, there is a false negative (FN). The detections outside all the tolerance windows are counted as false positives (FP). Doubled onsets (two detections for one ground-truth onset) and merged onsets (one detection for two ground-truth onsets) will be taken into account in the evaluation. Doubled onsets are a subset of the FP onsets, and merged onsets a subset of FN onsets.

We define:

*'''Precision''' P = Ocd / (Ocd +Ofp)
*'''Recall''' R = Ocd / (Ocd + Ofn)
*'''F-measure''' F = 2*P*R/(P+R)

with these notations:

*'''Ocd''' number of correctly detected onsets (CD)
*'''Ofn''' number of missed onsets (FN)
*'''Om''' number of merged onsets
*'''Ofp''' number of false positive onsets (FP)
*'''Od''' number of double onsets

Other indicative measurements:

*'''FP rate''' FP = 100. * (Ofp) / (Ocd+Ofp)
*'''Doubled Onset rate in FP''' D = 100 * Od / Ofp
*'''Merged Onset rate in FN''' M = 100 * Om / Ofn

Because files are cross-annotated, the mean Precision and Recall rates are defined by averaging Precision and Recall rates computed for each annotation.

To establish a ranking, we will use the F-measure, widely used in string comparisons. This criterion is arbitrary, but gives an indication of performance. It must be remembered that onset detection is a preprocessing step, so the real cost of an error of each type (false positive or false negative) depends on the application following this task.

=== Evaluation measures ===

* percentage of correct detections / false positives (can also be expressed as precision/recall)
* time precision (tolerance from +/- 50 ms to less). For certain file, we can't be much more accurate than 50 ms because of the weak annotation precision. This must be taken into account.
* separate scoring for different instrument types (percussive, strings, winds, etc)
* percentage of doubled detections
* speed measurements of the algorithms

== Submission Format ==
Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.

=== Input Data ===
Participating algorithms will have to read audio in the following format:

* Sample rate: 44.1 KHz
* Sample size: 16 bit
* Number of channels: 1 (mono)
* Encoding: WAV

=== Output Data ===

The onset detection algorithms will return onset times in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.

==== Output File Format (Audio Onset Detection) ====

The Audio Onset Detection output file format is an ASCII text format. Each onset time is specified, in seconds, on its own line. Specifically,

<onset time(in seconds)>\n

where \n denotes the end of line. The < and > characters are not included. An example output file would look something like:

0.243
1.476
1.987
2.449
3.224

=== Algorithm Calling Format ===

The submitted algorithm must take as arguments a SINGLE .wav file to perform the onset detection on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:

foobar %input %output
foobar -i %input -o %output

Moreover, if your submission takes additional parameters, such as a detection threshold, foobar could be called like:

foobar .1 %input %output
foobar -param1 .1 -i %input -o %output

If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example:

foobar('%input','%output')
foobar(.1,'%input','%output')

=== Parameter Sweeps ===
In past iterations of MIREX, submitters have been allowed to specify a parameter sweep so as to generate a precision-recall operator characteristic to better evaluate and understand the algorithm. If you wish to do so, please specify TEN different settings for your sweepable parameter. There are no guarantees that all ten will be tested and evaluated, however, as the time-constraints for MIREX are getting ever smaller as the number of submissions are getting ever larger. Therefore, please also specify the ONE single parameterization you feel is best in the README. If the whole parameter sweep cannot be evaluated, this single parameterization will be used.

=== Packaging submissions ===

* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
* Be sure to follow the [[MIREX 2012 Submission Instructions]]

All submissions should include a README file including the following the information:

* Command line calling format for all executables including examples
* Number of threads/cores used or whether this should be specified on the command line
* Expected memory footprint
* Expected runtime
* Approximately how much scratch disk space will the submission need to store any feature/cache files?
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.
* Any special notice regarding to running your algorithm

Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.

==== README File ====

A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.

For instance, to test the program foobar with different values for parameters param1, the README file would look like:

foobar -param1 .1 -i %input -o %output
foobar -param1 .15 -i %input -o %output
foobar -param1 .2 -i %input -o %output
foobar -param1 .25 -i %input -o %output
foobar -param1 .3 -i %input -o %output
...

For a submission using MATLAB, the README file could look like:

matlab -r "foobar(.1,'%input','%output');quit;"
matlab -r "foobar(.15,'%input','%output');quit;"
matlab -r "foobar(.2,'%input','%output');quit;"
matlab -r "foobar(.25,'%input','%output');quit;"
matlab -r "foobar(.3,'%input','%output');quit;"
...

The different command lines to evaluate the performance of each parameter set over the whole database will be generated automatically from each line in the README file containing both '%input' and '%output' strings.

== Time and hardware limits ==
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.

A hard limit of 6 hours will be imposed on analysis times.

== Submission opening date ==

Friday 5th August 2012

== Submission closing date ==
Friday 26th August 2012

2006:MIREX2006 Results

2012-01-15T14:09:39Z

MertBay: /* OVERALL RESULTS POSTER */

The MIREX 2006 Results are now available for all tasks!

=OVERALL RESULTS POSTER=
[[https://www.music-ir.org/mirex/bak/old/2006/images/b/bd/MIREX2006_poster_final.pdf MIREX 2006 Results Poster (PDF)]]

== Results by Task ==
* [[2006:Audio_Beat_Tracking_Results | Audio Beat Tracking Results]]
* [[2006:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]]
* [[2006:Audio_Melody_Extraction_Results | Audio Melody Extraction Results]]
* [[2006:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]
* [[2006:Audio_Onset_Detection_Results | Audio Onset Detection Results]]
* [[2006:Audio_Tempo_Extraction_Results | Audio Tempo Extraction Results]]
* [[2006:QBSH:_Query-by-Singing/Humming_Results | QBSH: Query-by-Singing/Humming Results]]
* [[2006:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]
* [[2006:Score_Following_Results | Score Following Results]]

== Machine Specifications ==

<csv>2006/mirex06_machine_specs.csv</csv>

2006:MIREX2006 Results

2012-01-15T14:09:15Z

MertBay: /* OVERALL RESULTS POSTER */

The MIREX 2006 Results are now available for all tasks!

=OVERALL RESULTS POSTER=
[[https://www.music-ir.org/mirex/bak/old/2006/images/b/bd/MIREX2006_poster_final.pdfmirex/bak/old/2006/images/b/bd/MIREX2006_poster_final.pdf MIREX 2006 Results Poster (PDF)]]

== Results by Task ==
* [[2006:Audio_Beat_Tracking_Results | Audio Beat Tracking Results]]
* [[2006:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]]
* [[2006:Audio_Melody_Extraction_Results | Audio Melody Extraction Results]]
* [[2006:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]
* [[2006:Audio_Onset_Detection_Results | Audio Onset Detection Results]]
* [[2006:Audio_Tempo_Extraction_Results | Audio Tempo Extraction Results]]
* [[2006:QBSH:_Query-by-Singing/Humming_Results | QBSH: Query-by-Singing/Humming Results]]
* [[2006:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]
* [[2006:Score_Following_Results | Score Following Results]]

== Machine Specifications ==

<csv>2006/mirex06_machine_specs.csv</csv>

2011:Multiple Fundamental Frequency Estimation & Tracking Results

2011-11-15T20:03:31Z

MertBay: /* Run Times */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2011:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! KD1
| Karin Dressler || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! LYC1
| LYC || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/LYC1.pdf PDF] || [http://mpac.ee.ntu.edu.tw/~ader888/ Cheng-Te Lee], [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yi-Hsuan Yang],[http://mpac.ee.ntu.edu.tw/people.php Homer Chen]
|-
! RFF1
| AlgorithmUsingPianoSamples || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/RFF1.pdf PDF] || [http://www.estg.ipleiria.pt/~gustavo.reis/ Gustavo Reis],[http:///sites.google.com/site/fcofdezdevegaweb/ Francisco Fernandéz ],[http://http://paginas.fe.up.pt/~ajf/ Aníbal Ferreira ]
|-
! RFF2
| MultiTimbralInternalSamples || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/RFF2.pdf PDF] || [http://www.estg.ipleiria.pt/~gustavo.reis/ Gustavo Reis],[http:///sites.google.com/site/fcofdezdevegaweb/ Francisco Fernandéz ],[http://http://paginas.fe.up.pt/~ajf/ Aníbal Ferreira ]
|-
! YR1
| MultiF0_Yeh_IRCAM f0FrmFile || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]
|-
! YR2
| MultiF0_Yeh_IRCAM f0TrkFile || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]
|-
! YR3
| MultiF0_Yeh_IRCAM f0FrmFile -c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]

|-
! YR4
| MultiF0_Yeh_IRCAM f0TrkFile -c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]

|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2011/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2011/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''KD1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''LYC1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/LYC1.tar.gz Cheng-Te Lee, Yi-Hsuan Yang,Homer Chen ] 
'''RFF1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/RFF1.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''RFF2''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/RFF2.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''YR1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR1.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR2''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR2.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR3''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR3.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR4''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR4.tar.gz Chunghsin YEH,Axel Roebel] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

<csv>2011/mf0/est/runtimes_mf0_2011.csv</csv>

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2011/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2011MF0task1.friedman.Friedman Mean Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2011/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2011/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2011/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2011/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Run Times====

<csv>2011/mf0/nt/runtimes_NT_2011.csv</csv>

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2011/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2011MF0.Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_Mean_Ranks.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''LYC1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/LYC1.tar.gz Cheng-Te Lee, Yi-Hsuan Yang,Homer Chen ] 
'''RFF1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/RFF1.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''RFF2''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/RFF2.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''YR1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/YR1.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR3''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/YR3.tar.gz Chunghsin YEH,Axel Roebel]

2011:Multiple Fundamental Frequency Estimation & Tracking Results

2011-11-15T20:02:22Z

MertBay: /* Friedman Tests for Note Tracking */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2011:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! KD1
| Karin Dressler || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! LYC1
| LYC || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/LYC1.pdf PDF] || [http://mpac.ee.ntu.edu.tw/~ader888/ Cheng-Te Lee], [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yi-Hsuan Yang],[http://mpac.ee.ntu.edu.tw/people.php Homer Chen]
|-
! RFF1
| AlgorithmUsingPianoSamples || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/RFF1.pdf PDF] || [http://www.estg.ipleiria.pt/~gustavo.reis/ Gustavo Reis],[http:///sites.google.com/site/fcofdezdevegaweb/ Francisco Fernandéz ],[http://http://paginas.fe.up.pt/~ajf/ Aníbal Ferreira ]
|-
! RFF2
| MultiTimbralInternalSamples || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/RFF2.pdf PDF] || [http://www.estg.ipleiria.pt/~gustavo.reis/ Gustavo Reis],[http:///sites.google.com/site/fcofdezdevegaweb/ Francisco Fernandéz ],[http://http://paginas.fe.up.pt/~ajf/ Aníbal Ferreira ]
|-
! YR1
| MultiF0_Yeh_IRCAM f0FrmFile || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]
|-
! YR2
| MultiF0_Yeh_IRCAM f0TrkFile || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]
|-
! YR3
| MultiF0_Yeh_IRCAM f0FrmFile -c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]

|-
! YR4
| MultiF0_Yeh_IRCAM f0TrkFile -c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]

|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2011/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2011/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''KD1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''LYC1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/LYC1.tar.gz Cheng-Te Lee, Yi-Hsuan Yang,Homer Chen ] 
'''RFF1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/RFF1.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''RFF2''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/RFF2.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''YR1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR1.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR2''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR2.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR3''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR3.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR4''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR4.tar.gz Chunghsin YEH,Axel Roebel] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

<csv>2011/mf0/est/runtimes_mf0_2011.csv</csv>

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2011/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2011MF0task1.friedman.Friedman Mean Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2011/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2011/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2011/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2011/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

===Run Times===

<csv>2011/mf0/nt/runtimes_NT_2011.csv</csv>

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2011/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2011MF0.Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_Mean_Ranks.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''LYC1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/LYC1.tar.gz Cheng-Te Lee, Yi-Hsuan Yang,Homer Chen ] 
'''RFF1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/RFF1.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''RFF2''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/RFF2.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''YR1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/YR1.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR3''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/YR3.tar.gz Chunghsin YEH,Axel Roebel]

2011:Multiple Fundamental Frequency Estimation & Tracking Results

2011-11-15T19:55:24Z

MertBay: /* Run Times */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2011:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! KD1
| Karin Dressler || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! LYC1
| LYC || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/LYC1.pdf PDF] || [http://mpac.ee.ntu.edu.tw/~ader888/ Cheng-Te Lee], [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yi-Hsuan Yang],[http://mpac.ee.ntu.edu.tw/people.php Homer Chen]
|-
! RFF1
| AlgorithmUsingPianoSamples || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/RFF1.pdf PDF] || [http://www.estg.ipleiria.pt/~gustavo.reis/ Gustavo Reis],[http:///sites.google.com/site/fcofdezdevegaweb/ Francisco Fernandéz ],[http://http://paginas.fe.up.pt/~ajf/ Aníbal Ferreira ]
|-
! RFF2
| MultiTimbralInternalSamples || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/RFF2.pdf PDF] || [http://www.estg.ipleiria.pt/~gustavo.reis/ Gustavo Reis],[http:///sites.google.com/site/fcofdezdevegaweb/ Francisco Fernandéz ],[http://http://paginas.fe.up.pt/~ajf/ Aníbal Ferreira ]
|-
! YR1
| MultiF0_Yeh_IRCAM f0FrmFile || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]
|-
! YR2
| MultiF0_Yeh_IRCAM f0TrkFile || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]
|-
! YR3
| MultiF0_Yeh_IRCAM f0FrmFile -c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]

|-
! YR4
| MultiF0_Yeh_IRCAM f0TrkFile -c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]

|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2011/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2011/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''KD1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''LYC1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/LYC1.tar.gz Cheng-Te Lee, Yi-Hsuan Yang,Homer Chen ] 
'''RFF1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/RFF1.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''RFF2''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/RFF2.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''YR1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR1.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR2''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR2.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR3''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR3.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR4''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR4.tar.gz Chunghsin YEH,Axel Roebel] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

<csv>2011/mf0/est/runtimes_mf0_2011.csv</csv>

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2011/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2011MF0task1.friedman.Friedman Mean Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2011/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2011/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2011/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2011/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2011/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2011MF0.Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_Mean_Ranks.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''LYC1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/LYC1.tar.gz Cheng-Te Lee, Yi-Hsuan Yang,Homer Chen ] 
'''RFF1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/RFF1.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''RFF2''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/RFF2.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''YR1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/YR1.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR3''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/YR3.tar.gz Chunghsin YEH,Axel Roebel]

2011:Multiple Fundamental Frequency Estimation & Tracking Results

2011-11-15T19:55:11Z

MertBay: /* Runtimes */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2011:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! KD1
| Karin Dressler || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! LYC1
| LYC || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/LYC1.pdf PDF] || [http://mpac.ee.ntu.edu.tw/~ader888/ Cheng-Te Lee], [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yi-Hsuan Yang],[http://mpac.ee.ntu.edu.tw/people.php Homer Chen]
|-
! RFF1
| AlgorithmUsingPianoSamples || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/RFF1.pdf PDF] || [http://www.estg.ipleiria.pt/~gustavo.reis/ Gustavo Reis],[http:///sites.google.com/site/fcofdezdevegaweb/ Francisco Fernandéz ],[http://http://paginas.fe.up.pt/~ajf/ Aníbal Ferreira ]
|-
! RFF2
| MultiTimbralInternalSamples || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/RFF2.pdf PDF] || [http://www.estg.ipleiria.pt/~gustavo.reis/ Gustavo Reis],[http:///sites.google.com/site/fcofdezdevegaweb/ Francisco Fernandéz ],[http://http://paginas.fe.up.pt/~ajf/ Aníbal Ferreira ]
|-
! YR1
| MultiF0_Yeh_IRCAM f0FrmFile || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]
|-
! YR2
| MultiF0_Yeh_IRCAM f0TrkFile || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]
|-
! YR3
| MultiF0_Yeh_IRCAM f0FrmFile -c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]

|-
! YR4
| MultiF0_Yeh_IRCAM f0TrkFile -c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]

|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2011/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2011/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''KD1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''LYC1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/LYC1.tar.gz Cheng-Te Lee, Yi-Hsuan Yang,Homer Chen ] 
'''RFF1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/RFF1.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''RFF2''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/RFF2.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''YR1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR1.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR2''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR2.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR3''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR3.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR4''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR4.tar.gz Chunghsin YEH,Axel Roebel] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

TBA

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2011/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2011MF0task1.friedman.Friedman Mean Ranks.png|500px]]

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2011/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2011/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2011/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2011/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2011/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2011MF0.Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_Mean_Ranks.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''LYC1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/LYC1.tar.gz Cheng-Te Lee, Yi-Hsuan Yang,Homer Chen ] 
'''RFF1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/RFF1.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''RFF2''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/RFF2.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''YR1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/YR1.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR3''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/YR3.tar.gz Chunghsin YEH,Axel Roebel]

2011:Multiple Fundamental Frequency Estimation & Tracking Results

2011-11-15T19:54:35Z

MertBay: /* Runtimes */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2011:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! KD1
| Karin Dressler || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! LYC1
| LYC || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/LYC1.pdf PDF] || [http://mpac.ee.ntu.edu.tw/~ader888/ Cheng-Te Lee], [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yi-Hsuan Yang],[http://mpac.ee.ntu.edu.tw/people.php Homer Chen]
|-
! RFF1
| AlgorithmUsingPianoSamples || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/RFF1.pdf PDF] || [http://www.estg.ipleiria.pt/~gustavo.reis/ Gustavo Reis],[http:///sites.google.com/site/fcofdezdevegaweb/ Francisco Fernandéz ],[http://http://paginas.fe.up.pt/~ajf/ Aníbal Ferreira ]
|-
! RFF2
| MultiTimbralInternalSamples || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/RFF2.pdf PDF] || [http://www.estg.ipleiria.pt/~gustavo.reis/ Gustavo Reis],[http:///sites.google.com/site/fcofdezdevegaweb/ Francisco Fernandéz ],[http://http://paginas.fe.up.pt/~ajf/ Aníbal Ferreira ]
|-
! YR1
| MultiF0_Yeh_IRCAM f0FrmFile || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]
|-
! YR2
| MultiF0_Yeh_IRCAM f0TrkFile || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]
|-
! YR3
| MultiF0_Yeh_IRCAM f0FrmFile -c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]

|-
! YR4
| MultiF0_Yeh_IRCAM f0TrkFile -c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]

|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2011/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2011/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''KD1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''LYC1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/LYC1.tar.gz Cheng-Te Lee, Yi-Hsuan Yang,Homer Chen ] 
'''RFF1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/RFF1.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''RFF2''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/RFF2.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''YR1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR1.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR2''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR2.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR3''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR3.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR4''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR4.tar.gz Chunghsin YEH,Axel Roebel] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

TBA

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2011/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2011MF0task1.friedman.Friedman Mean Ranks.png|500px]]

====Runtimes====

<csv>2011/mf0/est/runtimes_mf0_2011.csv</csv>

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2011/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2011/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2011/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2011/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2011/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2011MF0.Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_Mean_Ranks.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''LYC1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/LYC1.tar.gz Cheng-Te Lee, Yi-Hsuan Yang,Homer Chen ] 
'''RFF1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/RFF1.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''RFF2''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/RFF2.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''YR1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/YR1.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR3''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/YR3.tar.gz Chunghsin YEH,Axel Roebel]

2011:Multiple Fundamental Frequency Estimation & Tracking Results

2011-11-15T19:54:13Z

MertBay: /* Task 1: Multiple Fundamental Frequency Estimation (MF0E) */

==Introduction==
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2011:Multiple Fundamental Frequency Estimation & Tracking]] page.

==General Legend==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! BD1
| BenetosDixon MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD1.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD2
| BenetosDixon NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD2.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! BD3
| BenetosDixon NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/BD3.pdf PDF] || [http://www.dcs.qmul.ac.uk/~emmanouilb Emmanouil Benetos], [http://www.eecs.qmul.ac.uk/~simond Simon Dixon]
|-
! KD1
| Karin Dressler || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/KD1.pdf PDF] || [http://www.idmt.fraunhofer.de/ Karin Dressler ]
|-
! LYC1
| LYC || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/LYC1.pdf PDF] || [http://mpac.ee.ntu.edu.tw/~ader888/ Cheng-Te Lee], [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yi-Hsuan Yang],[http://mpac.ee.ntu.edu.tw/people.php Homer Chen]
|-
! RFF1
| AlgorithmUsingPianoSamples || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/RFF1.pdf PDF] || [http://www.estg.ipleiria.pt/~gustavo.reis/ Gustavo Reis],[http:///sites.google.com/site/fcofdezdevegaweb/ Francisco Fernandéz ],[http://http://paginas.fe.up.pt/~ajf/ Aníbal Ferreira ]
|-
! RFF2
| MultiTimbralInternalSamples || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/RFF2.pdf PDF] || [http://www.estg.ipleiria.pt/~gustavo.reis/ Gustavo Reis],[http:///sites.google.com/site/fcofdezdevegaweb/ Francisco Fernandéz ],[http://http://paginas.fe.up.pt/~ajf/ Aníbal Ferreira ]
|-
! YR1
| MultiF0_Yeh_IRCAM f0FrmFile || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]
|-
! YR2
| MultiF0_Yeh_IRCAM f0TrkFile || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]
|-
! YR3
| MultiF0_Yeh_IRCAM f0FrmFile -c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]

|-
! YR4
| MultiF0_Yeh_IRCAM f0TrkFile -c || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/YR1.pdf PDF] || [http://anasynth.ircam.fr/home/english/people/chung... Chunghsin YEH],[http://anasynth.ircam.fr/home/english/people/axelr... Axel Roebel ]

|}

==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

===MF0E Overall Summary Results===
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

<csv>2011/mf0/est/summary/task1.overall.csv</csv>

====Detailed Results====

<csv>2011/mf0/est/summary/task1.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/est/summary/task1.chroma.results.csv</csv>

===Individual Results Files for Task 1===
'''BD1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/BD1.tar.gz Emmanouil Benetos, Simon Dixon] 
'''KD1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/KD1.tar.gz Karin Dressler ] 
'''LYC1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/LYC1.tar.gz Cheng-Te Lee, Yi-Hsuan Yang,Homer Chen ] 
'''RFF1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/RFF1.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''RFF2''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/RFF2.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''YR1''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR1.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR2''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR2.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR3''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR3.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR4''' = [https://music-ir.org/mirex/results/2011/mf0/est/tars/YR4.tar.gz Chunghsin YEH,Axel Roebel] 

=====Info about the filenames=====
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

'''bs''' = bassoon,
'''cl''' = clarinet,
'''fl''' = flute,
'''hn''' = horn,
'''ob''' = oboe,
'''vl''' = violin,
'''cel''' = cello,
'''gtr''' = guitar,
'''sax''' = saxophone,
'''bass''' = electric bass guitar

===Run Times===

TBA

===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)===
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

====Tukey-Kramer HSD Multi-Comparison====

<csv>2011/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2011MF0task1.friedman.Friedman Mean Ranks.png|500px]]

====Runtimes====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/est/summary/runtimes_mf0_2011.csv</csv>

==Task 2:Note Tracking (NT)==
===NT Mixed Set Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

<csv>2011/mf0/nt/summary/task2.overall.results.csv</csv>

====Detailed Results====

<csv>2011/mf0/nt/summary/task2.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/nt/summary/task2.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2011/mf0/nt/summary/task2.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2011/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv>

====Friedman Tests for Note Tracking====
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

=====Tukey-Kramer HSD Multi-Comparison for Task2=====

<csv>2011/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv</csv>

[[File:2011MF0.Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_Mean_Ranks.png|500px]]

===NT Piano-Only Overall Summary Results===
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
6 piano recordings are evaluated separately for this subtask.

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv>

====Detailed Results====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv>

====Detailed Chroma Results====
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv>

====Results Based on Onset Only====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv>

====Chroma Results Based on Onset Only====

<csv>2011/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv>

===Individual Results Files for Task 2===
'''BD2''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/BD2.tar.gz Emmanouil Benetos, Simon Dixon] 
'''BD3''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/BD3.tar.gz Emmanouil Benetos, Simon Dixon] 
'''LYC1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/LYC1.tar.gz Cheng-Te Lee, Yi-Hsuan Yang,Homer Chen ] 
'''RFF1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/RFF1.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''RFF2''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/RFF2.tar.gz Gustavo Reis,Francisco Fernandéz ,Aníbal Ferreira ] 
'''YR1''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/YR1.tar.gz Chunghsin YEH,Axel Roebel] 
'''YR3''' = [https://music-ir.org/mirex/results/2011/mf0/nt/tars/YR3.tar.gz Chunghsin YEH,Axel Roebel]

2011:Audio Cover Song Identification Results

2011-11-06T05:25:20Z

MertBay: /* Friedman's Test for Significant Differences */

== Introduction ==

This year, we ran Audio Cover Song (ACS) Identfication with two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! ALL1
| tonalspacecoversys.m || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/ALL1.pdf PDF] || [http://www.cs.helsinki.fi Teppo E. Ahonen], [http://www.cs.helsinki.fi Kjell Lemström], [http://www.cs.helsinki.fi Simo Linkola]
|-
! CWWJ1
| CWWJ || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/CWWJ1.pdf PDF] || [http://sovideo.iis.sinica.edu.tw/SLG/index.html Chuan-Yau Chan], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang],[http://www.ee.ntu.edu.tw Shyh-Kang Jeng]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2011/cover/coversong.mixed.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mixed.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mixed.precision.groups.csv</csv>

====Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=2>2011/cover/Coversong_Mixed_Ave_Performance_Per_Query_Group_friedman_tukeyKramerHSD.csv</csv>

[[Image:2011_coversong_mixed_ave_performance_per_query_group_friedman_mean_ranks.png]]

====Run Times====
TBA

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2011//cover/coversong.mazurka.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mazurka.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mazurka.precision.groups.csv</csv>

====Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=2>2011/cover/Coversong_Maz_Ave_Performance_Per_Query_Group_friedman_tukeyKramerHSD.csv</csv>

[[Image:2011_coversong_maz_ave_performance_per_query_group_friedman_mean_ranks.png]]

====Run Times====
TBA

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 
[[Category: Results]]

2011:Audio Cover Song Identification Results

2011-11-06T05:24:58Z

MertBay: /* Friedman's Test for Significant Differences */

== Introduction ==

This year, we ran Audio Cover Song (ACS) Identfication with two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! ALL1
| tonalspacecoversys.m || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/ALL1.pdf PDF] || [http://www.cs.helsinki.fi Teppo E. Ahonen], [http://www.cs.helsinki.fi Kjell Lemström], [http://www.cs.helsinki.fi Simo Linkola]
|-
! CWWJ1
| CWWJ || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/CWWJ1.pdf PDF] || [http://sovideo.iis.sinica.edu.tw/SLG/index.html Chuan-Yau Chan], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang],[http://www.ee.ntu.edu.tw Shyh-Kang Jeng]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2011/cover/coversong.mixed.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mixed.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mixed.precision.groups.csv</csv>

====Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=2>2011/cover/Coversong_Mixed_Ave_Performance_Per_Query_Group_friedman_tukeyKramerHSD.csv</csv>

[[Image:2011_coversong_maz_ave_performance_per_query_group_friedman_mean_ranks.png]]

====Run Times====
TBA

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2011//cover/coversong.mazurka.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mazurka.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mazurka.precision.groups.csv</csv>

====Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=2>2011/cover/Coversong_Maz_Ave_Performance_Per_Query_Group_friedman_tukeyKramerHSD.csv</csv>

[[Image:2011_coversong_maz_ave_performance_per_query_group_friedman_mean_ranks.png]]

====Run Times====
TBA

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 
[[Category: Results]]

File:2011 coversong maz ave performance per query group friedman mean ranks.png

2011-11-06T05:23:54Z

MertBay:

2011:Audio Cover Song Identification Results

2011-11-06T05:23:34Z

MertBay: /* Friedman's Test for Significant Differences */

== Introduction ==

This year, we ran Audio Cover Song (ACS) Identfication with two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! ALL1
| tonalspacecoversys.m || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/ALL1.pdf PDF] || [http://www.cs.helsinki.fi Teppo E. Ahonen], [http://www.cs.helsinki.fi Kjell Lemström], [http://www.cs.helsinki.fi Simo Linkola]
|-
! CWWJ1
| CWWJ || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/CWWJ1.pdf PDF] || [http://sovideo.iis.sinica.edu.tw/SLG/index.html Chuan-Yau Chan], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang],[http://www.ee.ntu.edu.tw Shyh-Kang Jeng]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2011/cover/coversong.mixed.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mixed.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mixed.precision.groups.csv</csv>

====Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=2>2011/cover/Coversong_Mixed_Ave_Performance_Per_Query_Group_friedman_tukeyKramerHSD.csv</csv>

[[Image:2011_coversong_maz_ave_performance_per_query_group_friedman_mean_ranks.png]]

====Run Times====
TBA

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2011//cover/coversong.mazurka.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mazurka.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mazurka.precision.groups.csv</csv>

====Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=2>2011/cover/Coversong_Maz_Ave_Performance_Per_Query_Group_friedman_tukeyKramerHSD.csv</csv>

[[Image:2011_coversong_mixed_ave_performance_per_query_group_friedman_mean_ranks.png]]

====Run Times====
TBA

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 
[[Category: Results]]

2011:Audio Cover Song Identification Results

2011-11-06T05:22:37Z

MertBay: /* Average Performance per Query Group */

== Introduction ==

This year, we ran Audio Cover Song (ACS) Identfication with two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! ALL1
| tonalspacecoversys.m || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/ALL1.pdf PDF] || [http://www.cs.helsinki.fi Teppo E. Ahonen], [http://www.cs.helsinki.fi Kjell Lemström], [http://www.cs.helsinki.fi Simo Linkola]
|-
! CWWJ1
| CWWJ || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/CWWJ1.pdf PDF] || [http://sovideo.iis.sinica.edu.tw/SLG/index.html Chuan-Yau Chan], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang],[http://www.ee.ntu.edu.tw Shyh-Kang Jeng]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2011/cover/coversong.mixed.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mixed.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mixed.precision.groups.csv</csv>

====Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=2>2011/cover/Coversong_Mixed_Ave_Performance_Per_Query_Group_friedman_tukeyKramerHSD.csv</csv>

[[Image:2011_coversong_mixed_ave_performance_per_query_group_friedman_mean_ranks.png]]

====Run Times====
TBA

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2011//cover/coversong.mazurka.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mazurka.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mazurka.precision.groups.csv</csv>

====Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=2>2011/cover/Coversong_Maz_Ave_Performance_Per_Query_Group_friedman_tukeyKramerHSD.csv</csv>

[[Image:2011_coversong_mixed_ave_performance_per_query_group_friedman_mean_ranks.png]]

====Run Times====
TBA

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 
[[Category: Results]]

File:2011 coversong mixed ave performance per query group friedman mean ranks.png

2011-11-06T05:20:37Z

MertBay:

2011:Audio Cover Song Identification Results

2011-11-06T05:20:07Z

MertBay: /* Friedman's Test for Significant Differences */

== Introduction ==

This year, we ran Audio Cover Song (ACS) Identfication with two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! ALL1
| tonalspacecoversys.m || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/ALL1.pdf PDF] || [http://www.cs.helsinki.fi Teppo E. Ahonen], [http://www.cs.helsinki.fi Kjell Lemström], [http://www.cs.helsinki.fi Simo Linkola]
|-
! CWWJ1
| CWWJ || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/CWWJ1.pdf PDF] || [http://sovideo.iis.sinica.edu.tw/SLG/index.html Chuan-Yau Chan], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang],[http://www.ee.ntu.edu.tw Shyh-Kang Jeng]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2011/cover/coversong.mixed.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mixed.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mixed.precision.groups.csv</csv>

====Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=2>2011/cover/Coversong_Mixed_Ave_Performance_Per_Query_Group_friedman_tukeyKramerHSD.csv</csv>

[[Image:2011_coversong_mixed_ave_performance_per_query_group_friedman_mean_ranks.png]]

====Run Times====
TBA

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2011//cover/coversong.mazurka.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mazurka.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mazurka.precision.groups.csv</csv>

====Run Times====
TBA

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 
[[Category: Results]]

2011:Audio Cover Song Identification Results

2011-11-06T05:18:58Z

MertBay: /* Friedman's Test for Significant Differences */

== Introduction ==

This year, we ran Audio Cover Song (ACS) Identfication with two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! ALL1
| tonalspacecoversys.m || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/ALL1.pdf PDF] || [http://www.cs.helsinki.fi Teppo E. Ahonen], [http://www.cs.helsinki.fi Kjell Lemström], [http://www.cs.helsinki.fi Simo Linkola]
|-
! CWWJ1
| CWWJ || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/CWWJ1.pdf PDF] || [http://sovideo.iis.sinica.edu.tw/SLG/index.html Chuan-Yau Chan], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang],[http://www.ee.ntu.edu.tw Shyh-Kang Jeng]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2011/cover/coversong.mixed.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mixed.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mixed.precision.groups.csv</csv>

====Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=2>2011/cover/Coversong_Mixed_Ave_Performance_Per_Query_Group_friedman_tukeyKramerHSD.csv</csv>

[[Image:2009_coversong_mixed_ave_performance_per_query_group_friedman_mean_ranks.png]]

====Run Times====
TBA

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2011//cover/coversong.mazurka.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mazurka.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mazurka.precision.groups.csv</csv>

====Run Times====
TBA

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 
[[Category: Results]]

2011:Audio Cover Song Identification Results

2011-11-06T05:18:11Z

MertBay: /* Average Performance per Query Group */

== Introduction ==

This year, we ran Audio Cover Song (ACS) Identfication with two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! ALL1
| tonalspacecoversys.m || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/ALL1.pdf PDF] || [http://www.cs.helsinki.fi Teppo E. Ahonen], [http://www.cs.helsinki.fi Kjell Lemström], [http://www.cs.helsinki.fi Simo Linkola]
|-
! CWWJ1
| CWWJ || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/CWWJ1.pdf PDF] || [http://sovideo.iis.sinica.edu.tw/SLG/index.html Chuan-Yau Chan], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang],[http://www.ee.ntu.edu.tw Shyh-Kang Jeng]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2011/cover/coversong.mixed.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mixed.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mixed.precision.groups.csv</csv>

====Friedman's Test for Significant Differences====
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv p=2>2009//covertests/Coversong_Mixed_Ave_Performance_Per_Query_Group_friedman_tukeyKramerHSD.csv</csv>

[[Image:2009_coversong_mixed_ave_performance_per_query_group_friedman_mean_ranks.png]]

====Run Times====
TBA

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2011//cover/coversong.mazurka.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mazurka.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mazurka.precision.groups.csv</csv>

====Run Times====
TBA

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 
[[Category: Results]]

2011:Audio Cover Song Identification Results

2011-11-06T04:45:44Z

MertBay: /* Individual Results Files */

== Introduction ==

This year, we ran Audio Cover Song (ACS) Identfication with two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! ALL1
| tonalspacecoversys.m || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/ALL1.pdf PDF] || [http://www.cs.helsinki.fi Teppo E. Ahonen], [http://www.cs.helsinki.fi Kjell Lemström], [http://www.cs.helsinki.fi Simo Linkola]
|-
! CWWJ1
| CWWJ || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/CWWJ1.pdf PDF] || [http://sovideo.iis.sinica.edu.tw/SLG/index.html Chuan-Yau Chan], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang],[http://www.ee.ntu.edu.tw Shyh-Kang Jeng]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2011/cover/coversong.mixed.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mixed.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mixed.precision.groups.csv</csv>

====Run Times====
TBA

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2011//cover/coversong.mazurka.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mazurka.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mazurka.precision.groups.csv</csv>

====Run Times====
TBA

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.mazurkas.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 
[[Category: Results]]

2011:Audio Cover Song Identification Results

2011-11-06T04:43:45Z

MertBay: /* Mixed Collection */

== Introduction ==

This year, we ran Audio Cover Song (ACS) Identfication with two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! ALL1
| tonalspacecoversys.m || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/ALL1.pdf PDF] || [http://www.cs.helsinki.fi Teppo E. Ahonen], [http://www.cs.helsinki.fi Kjell Lemström], [http://www.cs.helsinki.fi Simo Linkola]
|-
! CWWJ1
| CWWJ || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/CWWJ1.pdf PDF] || [http://sovideo.iis.sinica.edu.tw/SLG/index.html Chuan-Yau Chan], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang],[http://www.ee.ntu.edu.tw Shyh-Kang Jeng]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2011/cover/coversong.mixed.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mixed.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mixed.precision.groups.csv</csv>

====Run Times====
TBA

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2011//cover/coversong.mazurka.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mazurka.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mazurka.precision.groups.csv</csv>

====Run Times====
TBA

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.avgprec.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.ranklist.txt Chuan-Yau Chan, Ju-Chiang Wang, Hsin-Min Wang,Shyh-Kang Jeng] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 
[[Category: Results]]

2011:Audio Cover Song Identification Results

2011-11-06T04:42:19Z

MertBay: /* Individual Results Files */

== Introduction ==

This year, we ran Audio Cover Song (ACS) Identfication with two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! ALL1
| tonalspacecoversys.m || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/ALL1.pdf PDF] || [http://www.cs.helsinki.fi Teppo E. Ahonen], [http://www.cs.helsinki.fi Kjell Lemström], [http://www.cs.helsinki.fi Simo Linkola]
|-
! CWWJ1
| CWWJ || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/CWWJ1.pdf PDF] || [http://sovideo.iis.sinica.edu.tw/SLG/index.html Chuan-Yau Chan], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang],[http://www.ee.ntu.edu.tw Shyh-Kang Jeng]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2011/cover/coversong.mixed.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mixed.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mixed.precision.groups.csv</csv>

====Run Times====
TBA

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2011//cover/coversong.mazurka.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2011/cover/coversong.mazurka.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2011/cover/coversong.mazurka.precision.groups.csv</csv>

====Run Times====
TBA

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/CWWJ1.1000.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''CWWJ1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.1000.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.avgprec.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 

'''Rank Lists'''

'''ALL1''' : [https://music-ir.org/mirex/results/2011/cover/ALL1.maz.ranklist.txt Teppo E. Ahonen, Kjell Lemström, Simo Linkola] 
[[Category: Results]]

2011:Audio Cover Song Identification Results

2011-11-06T04:41:08Z

MertBay: /* General Legend */

2011:Audio Cover Song Identification Results

2011-11-06T04:40:46Z

MertBay: /* General Legend */

2011:Audio Cover Song Identification Results

2011-11-06T04:40:32Z

MertBay: /* General Legend */

2011:Audio Cover Song Identification Results

2011-11-06T04:38:04Z

MertBay: /* General Legend */

2011:Audio Cover Song Identification Results

2011-11-06T04:35:45Z

MertBay: /* General Legend */

2011:Symbolic Melodic Similarity Results

2011-10-24T22:45:28Z

MertBay: /* Friedman Test with Multiple Comparisons Results (p=0.05) */

==Introduction==
These are the results for the 2011 running of the Symbolic Melodic Similarity task set. For background information about this task set please refer to the [[2011:Symbolic Melodic Similarity]] page.

Each system was given a query and returned the 10 most melodically similar songs from those taken from the Essen Collection (5274 pieces in the MIDI format; see [http://www.esac-data.org/ ESAC Data Homepage] for more information). For each query, we made four classes of error-mutations, thus the set comprises the following query classes:

* 0. No errors
* 1. One note deleted
* 2. One note inserted
* 3. One interval enlarged
* 4. One interval compressed

For each query (and its 4 mutations), the returned results (candidates) from all systems were then grouped together (query set) for evaluation by the human graders. The graders were provide with only heard perfect version against which to evaluate the candidates and did not know whether the candidates came from a perfect or mutated query. Each query/candidate set was evaluated by 1 individual grader. Using the Evalutron 6000 system, the graders gave each query/candidate pair two types of scores. Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 100).

====Evalutron 6000 Summary Data====
'''Number of evaluators''' = 6 
'''Number of evaluations per query/candidate pair''' = 1 
'''Number of queries per grader''' = 1 
'''Total number of candidates returned''' = 3900 
'''Total number of unique query/candidate pairs graded''' = 895 
'''Average number of query/candidate pairs evaluated per grader: 149 
'''Number of queries''' = 6 (perfect) with each perfect query error-mutated 4 different ways = 30 

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="540" | Contributors
|-

! LJY1
| LEE1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/LJY1.pdf PDF] || [http://slsp.kaist.ac.kr Juwan Lee],[http://slsp.kaist.ac.kr Seokhwan Jo],[http://slsp.kaist.ac.kr Chang D. Yoo]
|-

! LJY2
| LEE2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/LJY2.pdf PDF] || [http://slsp.kaist.ac.kr Juwan Lee],[http://slsp.kaist.ac.kr Seokhwan Jo],[http://slsp.kaist.ac.kr Chang D. Yoo]
|-

! UL1
| Shape || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/UL1.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns],[https://sites.google.com/site/jorgemorato/ Jorge Morato]
|-

! UL2
| Pitch || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/UL2.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns],[https://sites.google.com/site/jorgemorato/ Jorge Morato]
|-

! UL3
| Time || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/UL3.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns],[https://sites.google.com/site/jorgemorato/ Jorge Morato]
|-

! WK1
| Tir'a'Mir - binary || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK1.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK2
| Tir'a'Mir - melodic cosine || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK2.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK3
| Tir'a'Mir - cosine combine || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK3.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK4
| Tir'a'Mir - tfidf || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK4.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK5
| Tir'a'Mir - melodic bm25 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK5.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK6
| Tir'a'Mir - bm25 combine || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK6.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

|}

====Broad Categories====
'''NS''' = Not Similar 
'''SS''' = Somewhat Similar 
'''VS''' = Very Similar 

====Table Headings ====
'''ADR''' = Average Dynamic Recall 
'''NRGB''' = Normalize Recall at Group Boundaries 
'''AP''' = Average Precision (non-interpolated) 
'''PND''' = Precision at N Documents 

===Calculating Summary Measures===
'''Fine'''(1) = Sum of fine-grained human similarity decisions (0-100). 
'''PSum'''(1) = Sum of human broad similarity decisions: NS=0, SS=1, VS=2. 
'''WCsum'''(1) = 'World Cup' scoring: NS=0, SS=1, VS=3 (rewards Very Similar). 
'''SDsum'''(1) = 'Stephen Downie' scoring: NS=0, SS=1, VS=4 (strongly rewards Very Similar). 
'''Greater0'''(1) = NS=0, SS=1, VS=1 (binary relevance judgment). 
'''Greater1'''(1) = NS=0, SS=0, VS=1 (binary relevance judgment using only Very Similar). 

(1)Normalized to the range 0 to 1.

==Summary Results==
===Overall Scores (Includes Perfect and Error Candidates)===
<csv>2011/sms/Overall.csv</csv>

===Scores by Query Error Types===

<csv>2011/sms/No_Errors.csv</csv>

<csv>2011/sms/Deleted.csv</csv>

<csv>2011/sms/Inserted.csv</csv>

<csv>2011/sms/Enlarged.csv</csv>

<csv>2011/sms/Compressed.csv</csv>

===Friedman Test with Multiple Comparisons Results (p=0.05)===
The Friedman test was run in MATLAB against the Fine summary data over the 30 queries. 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
<csv>2011/sms/sum_friedman_fine.csv</csv>

<csv>2011/sms/sms_fine_scores_friedman.csv</csv>

[[Image:2011 sms fine scores friedmans.png]]

TBA

==Raw Scores==
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2011:Symbolic Melodic Similarity Raw Data]] page.

===Run Times===
TBA

[[Category: Results]]

File:2011 sms fine scores friedmans.png

2011-10-24T22:42:25Z

MertBay:

2011:Symbolic Melodic Similarity Results

2011-10-24T22:41:58Z

MertBay: /* Friedman Test with Multiple Comparisons Results (p=0.05) */

==Introduction==
These are the results for the 2011 running of the Symbolic Melodic Similarity task set. For background information about this task set please refer to the [[2011:Symbolic Melodic Similarity]] page.

Each system was given a query and returned the 10 most melodically similar songs from those taken from the Essen Collection (5274 pieces in the MIDI format; see [http://www.esac-data.org/ ESAC Data Homepage] for more information). For each query, we made four classes of error-mutations, thus the set comprises the following query classes:

* 0. No errors
* 1. One note deleted
* 2. One note inserted
* 3. One interval enlarged
* 4. One interval compressed

For each query (and its 4 mutations), the returned results (candidates) from all systems were then grouped together (query set) for evaluation by the human graders. The graders were provide with only heard perfect version against which to evaluate the candidates and did not know whether the candidates came from a perfect or mutated query. Each query/candidate set was evaluated by 1 individual grader. Using the Evalutron 6000 system, the graders gave each query/candidate pair two types of scores. Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 100).

====Evalutron 6000 Summary Data====
'''Number of evaluators''' = 6 
'''Number of evaluations per query/candidate pair''' = 1 
'''Number of queries per grader''' = 1 
'''Total number of candidates returned''' = 3900 
'''Total number of unique query/candidate pairs graded''' = 895 
'''Average number of query/candidate pairs evaluated per grader: 149 
'''Number of queries''' = 6 (perfect) with each perfect query error-mutated 4 different ways = 30 

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="540" | Contributors
|-

! LJY1
| LEE1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/LJY1.pdf PDF] || [http://slsp.kaist.ac.kr Juwan Lee],[http://slsp.kaist.ac.kr Seokhwan Jo],[http://slsp.kaist.ac.kr Chang D. Yoo]
|-

! LJY2
| LEE2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/LJY2.pdf PDF] || [http://slsp.kaist.ac.kr Juwan Lee],[http://slsp.kaist.ac.kr Seokhwan Jo],[http://slsp.kaist.ac.kr Chang D. Yoo]
|-

! UL1
| Shape || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/UL1.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns],[https://sites.google.com/site/jorgemorato/ Jorge Morato]
|-

! UL2
| Pitch || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/UL2.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns],[https://sites.google.com/site/jorgemorato/ Jorge Morato]
|-

! UL3
| Time || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/UL3.pdf PDF] || [http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es Juan Lloréns],[https://sites.google.com/site/jorgemorato/ Jorge Morato]
|-

! WK1
| Tir'a'Mir - binary || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK1.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK2
| Tir'a'Mir - melodic cosine || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK2.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK3
| Tir'a'Mir - cosine combine || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK3.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK4
| Tir'a'Mir - tfidf || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK4.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK5
| Tir'a'Mir - melodic bm25 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK5.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

! WK6
| Tir'a'Mir - bm25 combine || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/WK6.pdf PDF] || [http://dnlp.ca Jacek Wolkowicz], [http://dnlp.ca Vlado Keselj]
|-

|}

====Broad Categories====
'''NS''' = Not Similar 
'''SS''' = Somewhat Similar 
'''VS''' = Very Similar 

====Table Headings ====
'''ADR''' = Average Dynamic Recall 
'''NRGB''' = Normalize Recall at Group Boundaries 
'''AP''' = Average Precision (non-interpolated) 
'''PND''' = Precision at N Documents 

===Calculating Summary Measures===
'''Fine'''(1) = Sum of fine-grained human similarity decisions (0-100). 
'''PSum'''(1) = Sum of human broad similarity decisions: NS=0, SS=1, VS=2. 
'''WCsum'''(1) = 'World Cup' scoring: NS=0, SS=1, VS=3 (rewards Very Similar). 
'''SDsum'''(1) = 'Stephen Downie' scoring: NS=0, SS=1, VS=4 (strongly rewards Very Similar). 
'''Greater0'''(1) = NS=0, SS=1, VS=1 (binary relevance judgment). 
'''Greater1'''(1) = NS=0, SS=0, VS=1 (binary relevance judgment using only Very Similar). 

(1)Normalized to the range 0 to 1.

==Summary Results==
===Overall Scores (Includes Perfect and Error Candidates)===
<csv>2011/sms/Overall.csv</csv>

===Scores by Query Error Types===

<csv>2011/sms/No_Errors.csv</csv>

<csv>2011/sms/Deleted.csv</csv>

<csv>2011/sms/Inserted.csv</csv>

<csv>2011/sms/Enlarged.csv</csv>

<csv>2011/sms/Compressed.csv</csv>

===Friedman Test with Multiple Comparisons Results (p=0.05)===



<csv>2011/sms/sms_fine_scores_friedman.csv</csv>

[[Image:2011 sms fine scores friedmans.png]]

TBA

==Raw Scores==
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2011:Symbolic Melodic Similarity Raw Data]] page.

===Run Times===
TBA

[[Category: Results]]

2011:MIREX2011 Results

2011-10-22T04:02:01Z

MertBay: /* Other Tasks */

==OVERALL RESULTS POSTERS (First Version: Will need updating as last runs are completed)==
[https://www.music-ir.org/mirex/results/2011/mirex_2011_poster.pdf MIREX 2011 Overall Results Posters (PDF)]

==Results by Task ==
This year we ran many MIREX 2011 Tasks using the new [http://nema.lis.uiuc.edu/drupal/?q=nema/architecture NEMA MIREX DIY] infrastructure. Task results with "(DIY)" appended are those generated using the NEMA MIREX DIY system. Where appropriate, do explore the various new outputs that help visualize both individual and task-wide comparative performances. A demonstration [https://www.music-ir.org/diy-demo/ video] of the NEMA MIREX DIY system can be found is also available.

===Train-Test Task Set===
* [Audio Classical Composer Identification Results ]   (DIY)
* [ Audio Latin Genre Classification Results ]   (DIY)
* [ Audio Music Mood Classification Results ]   (DIY)
* [ Audio Mixed Popular Genre Classification Results ]   (DIY)

===Other Tasks===

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/abt/mck/ MCK Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/abt/maz/ MAZ Dataset]  (DIY)
* [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ace/ Audio Chord Detection]  (DIY)
* [[2011:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]]
* [https://nema.lis.illinois.edu/nema_out/mirex2011/results/akd/ Audio Key Detection Results]  (DIY)
* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/adc04/ ADC04 Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/mirex05/ MIREX05 Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/indian08/ INDIAN08 Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/mirex09_0dB/ MIREX09 0dB Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset]  (DIY)
* [[2011:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]
* [https://nema.lis.illinois.edu/nema_out/mirex2011/results/aod/ Audio Onset Detection Results]  (DIY)
* Audio Tag Classification Results
** Major Miner Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)]  (DIY)
*** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/atg/subtask1_report/aff/ Affinity estimation evaluation]  (DIY)
** Mood Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)]  (DIY)
*** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/atg/subtask2_report/aff/ Affinity estimation evaluation]  (DIY)
* [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ate/ Audio Tempo Estimation Results]  (DIY)
* [[2011:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]
* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/struct/mirex09/ MIREX09 dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/struct/mirex10/ MIREX10 dataset]  (DIY)
* [[2011:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]]
* Query-by-Tapping Results
*[[2011:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]
* [[2011:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]

== Machine Specifications ==

== Runtime for Submissions Run by NEMA DIY ==

* [[2011:Runtime | Runtime]]

[[Category:Results]]

2011:MIREX2011 Results

2011-10-22T04:01:38Z

MertBay: /* Other Tasks */

==OVERALL RESULTS POSTERS (First Version: Will need updating as last runs are completed)==
[https://www.music-ir.org/mirex/results/2011/mirex_2011_poster.pdf MIREX 2011 Overall Results Posters (PDF)]

==Results by Task ==
This year we ran many MIREX 2011 Tasks using the new [http://nema.lis.uiuc.edu/drupal/?q=nema/architecture NEMA MIREX DIY] infrastructure. Task results with "(DIY)" appended are those generated using the NEMA MIREX DIY system. Where appropriate, do explore the various new outputs that help visualize both individual and task-wide comparative performances. A demonstration [https://www.music-ir.org/diy-demo/ video] of the NEMA MIREX DIY system can be found is also available.

===Train-Test Task Set===
* [Audio Classical Composer Identification Results ]   (DIY)
* [ Audio Latin Genre Classification Results ]   (DIY)
* [ Audio Music Mood Classification Results ]   (DIY)
* [ Audio Mixed Popular Genre Classification Results ]   (DIY)

===Other Tasks===

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/abt/mck/ MCK Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/abt/maz/ MAZ Dataset]  (DIY)
* [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ace/ Audio Chord Detection]  (DIY)
* [[2011:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]]
* [https://nema.lis.illinois.edu/nema_out/mirex2011/results/akd/ Audio Key Detection Results]  (DIY)
* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/adc04/ ADC04 Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/mirex05/ MIREX05 Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/indian08/ INDIAN08 Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/mirex09_0dB/ MIREX09 0dB Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset]  (DIY)
* [[2011:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]
* [https://nema.lis.illinois.edu/nema_out/mirex2011/results/aod/ Audio Onset Detection Results]  (DIY)
* Audio Tag Classification Results
** Major Miner Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)]  (DIY)
*** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/atg/subtask1_report/aff/ Affinity estimation evaluation]  (DIY)
** Mood Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)]  (DIY)
*** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/atg/subtask2_report/aff/ Affinity estimation evaluation]  (DIY)
* [https://nema.lis.illinois.edu/nema_out/mirex2011/results/ate/ Audio Tempo Estimation Results]  (DIY)
* [[2011:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]
* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/struct/mirex09/ MIREX09 dataset]  (DIY)
** [https://nema.lis.illinois.edu/nema_out/mirex2011/results/struct/mirex10/ MIREX10 dataset]  (DIY)
* [[2011:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]]
* [[ Query-by-Tapping Results]]
*[[2011:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]
* [[2011:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]

== Machine Specifications ==

== Runtime for Submissions Run by NEMA DIY ==

* [[2011:Runtime | Runtime]]

[[Category:Results]]

2011:Query-by-Singing/Humming Results

2011-10-22T03:18:33Z

MertBay: /* Runtime Results */

==Introduction==
These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the [[2011:Query by Singing/Humming]] page.

===Task Descriptions===

'''Task 1 [[#Task 1 Results|Goto Task 1 Results]]''': The first subtask is the same as last year. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:

*[[#Task 1a, Jang's dataset Results|Jang's Dataset Results]] Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBSH-corpus.rar MIR-QBSH corpus] with 48 songs as ground truth + 2000 Essen Collection MIDI noise files. See [http://www.esac-data.org/ ESAC Data Homepage] for more information about the Essen Collection. The queries consists of 4431 humming. All queries are from the beginning of references
*[[#Task 1b, ThinkIT's dataset Results|ThinkIT's Dataset Results]] [http://mirlab.org/dataSet/public/IOACAS_QBH_Coprus1.rar IOACAS corpus 1] data set with 106 songs as ground truth + 2000 Essen Collection MIDI noise files. See [http://www.esac-data.org/ ESAC Data Homepage] for more information about the Essen Collection. The queries consists of 355 humming. There are no "singing from beginning" gurantee.
*[[#Task 1c, IOACAS2's dataset Results|IOACAS's 2nd Dataset Results]] [http://mirlab.org/dataSet/public/IOACAS_QBH_Coprus2.rar IOACAS corpus 2] data set with 192 songs as ground truth + 2000 Essen Collection MIDI noise files. See [http://www.esac-data.org/ ESAC Data Homepage] for more information about the Essen Collection. The queries consists of 404 humming. There are no "singing from beginning" gurantee.

'''Task 2 [[#Task 2 Results|Goto Task 2 Results]]''': The second subtask is the query against other humming. In the second subtask, Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBSH-corpus.rar MIR-QBSH corpus] has been divided into two groups (2040 as queries and 2391 as database). The query is performed against the other humming database and the top 10 closed are returned. The score is simple count of how many returns belong to the same ground truth song.

===General Legend===
== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! TY1
| QBSH algorithm 1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/TY1.pdf PDF] || [http://mirlab.org/users/kenshincs/index.htm Tzu-Chun Yeh]
|-
! TY2
| QBSH algorithm 2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/TY2.pdf PDF] || [http://mirlab.org/users/kenshincs/index.htm Tzu-Chun Yeh]
|-
! JSSLP1
| KETI || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/JSSLP1.pdf PDF] || [http://www.keti.re.kr/digitalmedia/ Dalwon Jang], [http://www.keti.re.kr SUNG JOO PARK],[http://www.keti.re.kr Chai-Jong Song],[http://www.keti.re.kr Saim Shin],[http://www.keti.re.kr SEI-JIN JANG],[http://www.keti.re.kr Jong-Seol Lee],[http://www.keti.re.kr/digitalmedia/ Seok-Pil Lee],[http://www.keti.re.kr/home/rnd/sub02_04.asp Kyeung Hak Seo]
|}

===Task 1 Results===

====Task 1a, Jang's dataset Results====

=====Task 1a Overall Results=====
<csv>2011/qbsh/QbshFinalTask1a.csv</csv>

=====Task 1a Friedman's Test for Significant Differences=====
The Friedman test was run in MATLAB against the QBSH Task 1a Simple/MRR data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:
<csv>2011/qbsh/friedman/QbshTask1aSimpleByGroup.friedman.tukeyKramerHSD.csv</csv>

[[Image:2011QbshTask1aSimpleByGroup.friedman.tukeyKramerHSD.png|500px]]

MRR Method:
<csv>2011/qbsh/friedman/QbshTask1aMrrByGroup.friedman.tukeyKramerHSD.csv</csv>

[[Image:2011QbshTask1aMrrByGroup.friedman.tukeyKramerHSD.png|500px]]

=====Task 1a Summary Results by Query Group=====
Simple Hit/Miss Count
<csv>2011/qbsh/QbshTask1aSimpleByGroup.csv</csv>

MRR Method
<csv>2011/qbsh/QbshTask1aMrrByGroup.csv</csv>

====Task 1a Summary Results by Query ====
Simple Hit/Miss Counting
[https://www.music-ir.org/mirex/results/2011/qbsh/QbshTask1aSimpleByQuery.csv]

MRR Method
[https://www.music-ir.org/mirex/results/2011/qbsh/QbshTask1aMrrByQuery.csv]

====Task 1b, ThinkIT's dataset Results====

=====Task 1b Overall Results=====
<csv>2011/qbsh/QbshFinalTask1b.csv</csv>

=====Task 1b Friedman's Test for Significant Differences=====
The Friedman test was run in MATLAB against the QBSH Task 1b Simple/MRR data over the 106 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:
<csv>2011/qbsh/friedman/QbshTask1bSimpleByGroup.friedman.tukeyKramerHSD.csv</csv>

[[Image:2011QbshTask1bSimpleByGroup.friedman.tukeyKramerHSD.png|500px]]

MRR Method:
<csv>2011/qbsh/friedman/QbshTask1bMrrByGroup.friedman.tukeyKramerHSD.csv</csv>

[[Image:2011QbshTask1bMrrByGroup.friedman.tukeyKramerHSD.png|500px]]
=====Task 1b Summary Results by Query Group=====
Simple Hit/Miss Count
<csv>2011/qbsh/QbshTask1bSimpleByGroup.csv</csv>

MRR Method
<csv>2011/qbsh/QbshTask1bMrrByGroup.csv</csv>

====Task 1b Summary Results by Query ====
Simple Hit/Miss Counting
[https://www.music-ir.org/mirex/results/2011/qbsh/QbshTask1bSimpleByQuery.csv]

MRR Method
[https://www.music-ir.org/mirex/results/2011/qbsh/QbshTask1bMrrByQuery.csv]

====Task 1c, IOACAS2's dataset Results====
As argued by Jang, there are some abnormal in the IOACAS2 data set. Therefore, we just provide the raw result without any further analysis.

=====Task 1c Summary Results by Query Group=====
Simple Hit/Miss Count
<csv>2011/qbsh/QbshTask1cSimpleByGroup.csv</csv>

MRR Method
<csv>2011/qbsh/QbshTask1cMrrByGroup.csv</csv>

====Task 1c Summary Results by Query ====
Simple Hit/Miss Counting
[https://www.music-ir.org/mirex/results/2011/qbsh/QbshTask1cSimpleByQuery.csv]

MRR Method
[https://www.music-ir.org/mirex/results/2011/qbsh/QbshTask1cMrrByQuery.csv]

===Task 2 Results===

=====Task 2 Overall Results=====
<csv>2011/qbsh/QbshFinalTask2.csv</csv>

=====Task 2 Friedman's Test for Significant Differences=====
The Friedman test was run in MATLAB against the QBSH Task 2 data over the 48 ground truth song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

<csv>2011/qbsh/friedman/QbshTask2Group.friedman.tukeyKramerHSD.csv</csv>

[[Image:2011QbshTask2Group.friedman.tukeyKramerHSD.png|500px]]

=====Task 2 Summary Results by Query Group=====
<csv>2011/qbsh/QbshTask2Group.csv</csv>

====Task 2 Summary Results by Query ====
[https://www.music-ir.org/mirex/results/2011/qbsh/QbshTask2Query.csv]

===Runtime Results===
TBA
<csv>2011/qbsh/QbshRunTime.csv</csv>

[[Category: Results]]