Difference between revisions of "2014:Multiple Fundamental Frequency Estimation & Tracking Results"
(Created page with "==Introduction== These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set ple...") |
(→Introduction) |
||
(6 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
==Introduction== | ==Introduction== | ||
− | These are the results for the | + | These are the results for the 2014 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[2013:Multiple Fundamental Frequency Estimation & Tracking]] page. |
==General Legend== | ==General Legend== | ||
Line 10: | Line 10: | ||
! width="440" | Contributors | ! width="440" | Contributors | ||
|- | |- | ||
− | ! | + | ! BW1 |
− | | | + | | MSSIPLCA_VQT_MultiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/BW1.pdf PDF] || [http://www.soi.city.ac.uk/~sbbj660/ Emmanouil Benetos], [http://www.soi.city.ac.uk/~sa746/ Tillman Weyde] |
|- | |- | ||
− | ! | + | ! BW2 |
− | | | + | | MSSIPLCA_VQT_NoteTracking1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/BW2.pdf PDF] || [http://www.soi.city.ac.uk/~sbbj660/ Emmanouil Benetos], [http://www.soi.city.ac.uk/~sa746/ Tillman Weyde] |
|- | |- | ||
− | ! | + | ! BW3 |
− | | | + | | MSSIPLCA_VQT_NoteTracking2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/BW3.pdf PDF] || [http://www.soi.city.ac.uk/~sbbj660/ Emmanouil Benetos], [http://www.soi.city.ac.uk/~sa746/ Tillman Weyde] |
|- | |- | ||
− | ! | + | ! CB1 |
− | | | + | | Silvet || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB1.pdf PDF] || [http://www.eecs.qmul.ac.uk/people/view/3111/chris-cannam Chris Cannam], [http://www.soi.city.ac.uk/~sbbj660/ Emmanouil Benetos] |
+ | |- | ||
+ | ! CD3 | ||
+ | | Pianoamt || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CD3.pdf PDF] || Andrea Cogliati, [http://www.ece.rochester.edu/~zduan/ Zhiyao Duan] | ||
+ | |- | ||
+ | ! DT1 | ||
+ | | DT1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/DT1.pdf PDF] || [http://www.ece.rochester.edu/~zduan/ Zhiyao Duan], [http://www.esm.rochester.edu/faculty/temperley_david/ David Temperley] | ||
+ | |- | ||
+ | ! DT2 | ||
+ | | DT2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/DT2.pdf PDF] || [http://www.ece.rochester.edu/~zduan/ Zhiyao Duan], [http://www.esm.rochester.edu/faculty/temperley_david/ David Temperley] | ||
+ | |- | ||
+ | ! DT3 | ||
+ | | DT3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/DT3.pdf PDF] || [http://www.ece.rochester.edu/~zduan/ Zhiyao Duan], [http://www.esm.rochester.edu/faculty/temperley_david/ David Temperley] | ||
+ | |- | ||
+ | ! EF1 | ||
+ | | AEAF1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/EF1.pdf PDF] || Anders Elowsson, Anders Friberg | ||
+ | |- | ||
+ | ! KD2 | ||
+ | | KD multiF0 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/KD2.pdf PDF] || Karin Dressle | ||
+ | |- | ||
+ | ! RM1 | ||
+ | | Constrained NMF || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/RM1.pdf PDF] || [http://www.Daniel Recoskie, Richard Mann] | ||
+ | |- | ||
+ | ! SB5 | ||
+ | | PianoTranscriptor.2014 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/SB5.pdf PDF] || [http://www.cp.jku.at/people/boeck/ Sebastian Böck] | ||
+ | |- | ||
+ | ! SY1 | ||
+ | | MPE1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/SY1.pdf PDF] || Li Su, [http://mac.citi.sinica.edu.tw/~yang/ Yi-Hsuan Yang] | ||
+ | |- | ||
+ | ! SY2 | ||
+ | | MPE2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/SY2.pdf PDF] || Li Su, [http://mac.citi.sinica.edu.tw/~yang/ Yi-Hsuan Yang] | ||
|- | |- | ||
− | ! | + | ! SY3 |
− | | | + | | MPE3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/SY3.pdf PDF] || Li Su, [http://mac.citi.sinica.edu.tw/~yang/ Yi-Hsuan Yang] |
|- | |- | ||
− | ! | + | ! SY4 |
− | | | + | | MPE4 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/SY4.pdf PDF] || Li Su, [http://mac.citi.sinica.edu.tw/~yang/ Yi-Hsuan Yang] |
|- | |- | ||
|} | |} | ||
Line 35: | Line 65: | ||
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony. | Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony. | ||
− | <csv> | + | <csv>2014/mf0/est/summary/task1.overall.csv</csv> |
====Detailed Results==== | ====Detailed Results==== | ||
− | <csv> | + | <csv>2014/mf0/est/summary/task1.results.csv</csv> |
====Detailed Chroma Results==== | ====Detailed Chroma Results==== | ||
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating) | Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating) | ||
− | <csv> | + | <csv>2014/mf0/est/summary/task1.chroma.results.csv</csv> |
===Individual Results Files for Task 1=== | ===Individual Results Files for Task 1=== | ||
− | '''BW1'''= [https://music-ir.org/mirex/results/ | + | '''BW1'''= [https://music-ir.org/mirex/results/2014/mf0/est/BW1.results.tar Emmanouil Benetos, Tillman Weyde]<br /> |
− | ''' | + | '''EF1'''= [https://music-ir.org/mirex/results/2014/mf0/est/EF1.results.tar Anders Elowsson, Anders Friberg]<br /> |
− | ''' | + | '''KD2'''= [https://music-ir.org/mirex/results/2014/mf0/est/KD2.results.tar Karin Dressler]<br /> |
− | + | '''RM1'''= [https://music-ir.org/mirex/results/2014/mf0/est/RM1.results.tar Daniel Recoskie, Richard Mann]<br /> | |
+ | '''SY1'''= [https://music-ir.org/mirex/results/2014/mf0/est/SY1.results.tar Li Su, Yi-Hsuan Yang]<br /> | ||
+ | '''SY2'''= [https://music-ir.org/mirex/results/2014/mf0/est/SY2.results.tar Li Su, Yi-Hsuan Yang]<br /> | ||
+ | '''SY3'''= [https://music-ir.org/mirex/results/2014/mf0/est/SY3.results.tar Li Su, Yi-Hsuan Yang]<br /> | ||
=====Info about the filenames===== | =====Info about the filenames===== | ||
Line 68: | Line 101: | ||
===Run Times=== | ===Run Times=== | ||
− | <csv> | + | <csv>2014/mf0/est/runtimes_mf0_2014.csv</csv> |
===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)=== | ===Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)=== | ||
Line 75: | Line 108: | ||
====Tukey-Kramer HSD Multi-Comparison==== | ====Tukey-Kramer HSD Multi-Comparison==== | ||
− | <csv> | + | <csv>2014/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv> |
− | [[File: | + | [[File:2014_Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png|500px]] |
==Task 2:Note Tracking (NT)== | ==Task 2:Note Tracking (NT)== | ||
Line 85: | Line 118: | ||
A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano. | A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano. | ||
− | <csv> | + | <csv>2014/mf0/nt/summary/task2.overall.results.csv</csv> |
====Detailed Results==== | ====Detailed Results==== | ||
− | <csv> | + | <csv>2014/mf0/nt/summary/task2.results.csv</csv> |
====Detailed Chroma Results==== | ====Detailed Chroma Results==== | ||
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating) | Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating) | ||
− | <csv> | + | <csv>2014/mf0/nt/summary/task2.chroma.results.csv</csv> |
====Results Based on Onset Only==== | ====Results Based on Onset Only==== | ||
− | <csv> | + | <csv>2014/mf0/nt/summary/task2.onsetOnly.results.csv</csv> |
====Chroma Results Based on Onset Only==== | ====Chroma Results Based on Onset Only==== | ||
− | <csv> | + | <csv>2014/mf0/nt/summary/task2.onsetOnly.chroma.results.csv</csv> |
====Run Times==== | ====Run Times==== | ||
− | <csv> | + | <csv>2014/mf0/nt/runtimes_NT_2014.csv</csv> |
====Friedman Tests for Note Tracking==== | ====Friedman Tests for Note Tracking==== | ||
Line 114: | Line 147: | ||
=====Tukey-Kramer HSD Multi-Comparison for Task2===== | =====Tukey-Kramer HSD Multi-Comparison for Task2===== | ||
− | <csv> | + | <csv>2014/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_TukeyKramerHSD.csv</csv> |
− | [[File: | + | [[File:2014_Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_Mean_Ranks.png|500px]] |
===NT Piano-Only Overall Summary Results=== | ===NT Piano-Only Overall Summary Results=== | ||
Line 122: | Line 155: | ||
6 piano recordings are evaluated separately for this subtask. | 6 piano recordings are evaluated separately for this subtask. | ||
− | <csv> | + | <csv>2014/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv</csv> |
====Detailed Results==== | ====Detailed Results==== | ||
− | <csv> | + | <csv>2014/mf0/nt/summary_piano_subtask/task2_piano.results.csv</csv> |
====Detailed Chroma Results==== | ====Detailed Chroma Results==== | ||
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating) | Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating) | ||
− | <csv> | + | <csv>2014/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv</csv> |
====Results Based on Onset Only==== | ====Results Based on Onset Only==== | ||
− | <csv> | + | <csv>2014/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv</csv> |
====Chroma Results Based on Onset Only==== | ====Chroma Results Based on Onset Only==== | ||
− | <csv> | + | <csv>2014/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv</csv> |
===Individual Results Files for Task 2=== | ===Individual Results Files for Task 2=== | ||
− | '''BW2'''= [https://music-ir.org/mirex/results/ | + | '''BW2'''= [https://music-ir.org/mirex/results/2014/mf0/nt/BW2.results.tar.gz Emmanouil Benetos, Tillman Weyde]<br /> |
− | '''BW3'''= [https://music-ir.org/mirex/results/ | + | '''BW3'''= [https://music-ir.org/mirex/results/2014/mf0/nt/BW3.results.tar.gz Emmanouil Benetos, Tillman Weyde]<br /> |
− | ''' | + | '''CB1'''= [https://music-ir.org/mirex/results/2014/mf0/nt/CB1.results.tar Chris Cannam, Emmanouil Benetos]<br /> |
+ | '''CD3'''= [https://music-ir.org/mirex/results/2014/mf0/nt/CD3.results.tar Andrea Cogliati, Zhiyao Duan]<br /> | ||
+ | '''DT1'''= [https://music-ir.org/mirex/results/2014/mf0/nt/DT1.results.tar Zhiyao Duan, David Temperley]<br /> | ||
+ | '''DT2'''= [https://music-ir.org/mirex/results/2014/mf0/nt/DT2.results.tar Zhiyao Duan, David Temperley]<br /> | ||
+ | '''DT3'''= [https://music-ir.org/mirex/results/2014/mf0/nt/DT3.results.tar Zhiyao Duan, David Temperley]<br /> | ||
+ | '''EF1'''= [https://music-ir.org/mirex/results/2014/mf0/nt/EF1.results.tar Anders Elowsson, Anders Friberg]<br /> | ||
+ | '''KD2'''= [https://music-ir.org/mirex/results/2014/mf0/nt/KD2.results.tar Karin Dressler]<br /> | ||
+ | '''RM1'''= [https://music-ir.org/mirex/results/2014/mf0/nt/RM1.results.tar Daniel Recoskie, Richard Mann]<br /> | ||
+ | '''SB5'''= [https://music-ir.org/mirex/results/2014/mf0/nt/SB5.results.tar Sebastian Böck]<br /> | ||
+ | '''SY4'''= [https://music-ir.org/mirex/results/2014/mf0/nt/SY4.results.tar Li Su, Yi-Hsuan Yang]<br /> |
Latest revision as of 18:30, 21 October 2015
Contents
Introduction
These are the results for the 2014 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the 2013:Multiple Fundamental Frequency Estimation & Tracking page.
General Legend
Sub code | Submission name | Abstract | Contributors |
---|---|---|---|
BW1 | MSSIPLCA_VQT_MultiF0 | Emmanouil Benetos, Tillman Weyde | |
BW2 | MSSIPLCA_VQT_NoteTracking1 | Emmanouil Benetos, Tillman Weyde | |
BW3 | MSSIPLCA_VQT_NoteTracking2 | Emmanouil Benetos, Tillman Weyde | |
CB1 | Silvet | Chris Cannam, Emmanouil Benetos | |
CD3 | Pianoamt | Andrea Cogliati, Zhiyao Duan | |
DT1 | DT1 | Zhiyao Duan, David Temperley | |
DT2 | DT2 | Zhiyao Duan, David Temperley | |
DT3 | DT3 | Zhiyao Duan, David Temperley | |
EF1 | AEAF1 | Anders Elowsson, Anders Friberg | |
KD2 | KD multiF0 | Karin Dressle | |
RM1 | Constrained NMF | Recoskie, Richard Mann | |
SB5 | PianoTranscriptor.2014 | Sebastian Böck | |
SY1 | MPE1 | Li Su, Yi-Hsuan Yang | |
SY2 | MPE2 | Li Su, Yi-Hsuan Yang | |
SY3 | MPE3 | Li Su, Yi-Hsuan Yang | |
SY4 | MPE4 | Li Su, Yi-Hsuan Yang |
Task 1: Multiple Fundamental Frequency Estimation (MF0E)
MF0E Overall Summary Results
Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.
BW1 | EF1 | KD2 | RM1 | SY1 | SY2 | SY3 | |
---|---|---|---|---|---|---|---|
Accuracy | 0.657 | 0.723 | 0.68 | 0.411 | 0.634 | 0.635 | 0.609 |
Accuracy Chroma | 0.69 | 0.739 | 0.702 | 0.484 | 0.663 | 0.665 | 0.638 |
Detailed Results
Precision | Recall | Accuracy | Etot | Esubs | Emiss | Efa | ||
---|---|---|---|---|---|---|---|---|
BW1 | 0.751 | 0.747 | 0.657 | 0.412 | 0.112 | 0.140 | 0.159 | |
EF1 | 0.857 | 0.775 | 0.723 | 0.316 | 0.063 | 0.163 | 0.090 | |
KD2 | 0.774 | 0.773 | 0.680 | 0.374 | 0.094 | 0.133 | 0.147 | |
RM1 | 0.498 | 0.483 | 0.411 | 0.717 | 0.317 | 0.200 | 0.200 | |
SY1 | 0.701 | 0.767 | 0.634 | 0.474 | 0.126 | 0.108 | 0.241 | |
SY2 | 0.738 | 0.732 | 0.635 | 0.466 | 0.118 | 0.150 | 0.198 | |
SY3 | 0.690 | 0.743 | 0.609 | 0.545 | 0.128 | 0.129 | 0.288 |
Detailed Chroma Results
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)
Precision | Recall | Accuracy | Etot | Esubs | Emiss | Efa | ||
---|---|---|---|---|---|---|---|---|
BW1 | 0.789 | 0.789 | 0.690 | 0.370 | 0.071 | 0.140 | 0.159 | |
EF1 | 0.876 | 0.793 | 0.739 | 0.297 | 0.045 | 0.163 | 0.090 | |
KD2 | 0.801 | 0.798 | 0.702 | 0.349 | 0.069 | 0.133 | 0.147 | |
RM1 | 0.588 | 0.568 | 0.484 | 0.631 | 0.232 | 0.200 | 0.200 | |
SY1 | 0.736 | 0.802 | 0.663 | 0.439 | 0.091 | 0.108 | 0.241 | |
SY2 | 0.777 | 0.769 | 0.665 | 0.429 | 0.081 | 0.150 | 0.198 | |
SY3 | 0.726 | 0.780 | 0.638 | 0.508 | 0.090 | 0.129 | 0.288 |
Individual Results Files for Task 1
BW1= Emmanouil Benetos, Tillman Weyde
EF1= Anders Elowsson, Anders Friberg
KD2= Karin Dressler
RM1= Daniel Recoskie, Richard Mann
SY1= Li Su, Yi-Hsuan Yang
SY2= Li Su, Yi-Hsuan Yang
SY3= Li Su, Yi-Hsuan Yang
Info about the filenames
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:
bs = bassoon, cl = clarinet, fl = flute, hn = horn, ob = oboe, vl = violin, cel = cello, gtr = guitar, sax = saxophone, bass = electric bass guitar
Run Times
sub_ID | Runtime(sec) |
---|---|
BW1 | 2677 |
EF1 | 10800 |
KD2 | 180 |
RM1 | 7401 |
SY1 | ~600 |
SY2 | ~600 |
SY3 | ~600 |
Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.
Tukey-Kramer HSD Multi-Comparison
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
EF1 | KD2 | 0.4258 | 1.8500 | 3.2742 | TRUE |
EF1 | BW1 | 0.5758 | 2.0000 | 3.4242 | TRUE |
EF1 | SY2 | 0.7008 | 2.1250 | 3.5492 | TRUE |
EF1 | SY1 | 1.1008 | 2.5250 | 3.9492 | TRUE |
EF1 | SY3 | 2.1758 | 3.6000 | 5.0242 | TRUE |
EF1 | RM1 | 3.8008 | 5.2250 | 6.6492 | TRUE |
KD2 | BW1 | -1.2742 | 0.1500 | 1.5742 | FALSE |
KD2 | SY2 | -1.1492 | 0.2750 | 1.6992 | FALSE |
KD2 | SY1 | -0.7492 | 0.6750 | 2.0992 | FALSE |
KD2 | SY3 | 0.3258 | 1.7500 | 3.1742 | TRUE |
KD2 | RM1 | 1.9508 | 3.3750 | 4.7992 | TRUE |
BW1 | SY2 | -1.2992 | 0.1250 | 1.5492 | FALSE |
BW1 | SY1 | -0.8992 | 0.5250 | 1.9492 | FALSE |
BW1 | SY3 | 0.1758 | 1.6000 | 3.0242 | TRUE |
BW1 | RM1 | 1.8008 | 3.2250 | 4.6492 | TRUE |
SY2 | SY1 | -1.0242 | 0.4000 | 1.8242 | FALSE |
SY2 | SY3 | 0.0508 | 1.4750 | 2.8992 | TRUE |
SY2 | RM1 | 1.6758 | 3.1000 | 4.5242 | TRUE |
SY1 | SY3 | -0.3492 | 1.0750 | 2.4992 | FALSE |
SY1 | RM1 | 1.2758 | 2.7000 | 4.1242 | TRUE |
SY3 | RM1 | 0.2008 | 1.6250 | 3.0492 | TRUE |
Task 2:Note Tracking (NT)
NT Mixed Set Overall Summary Results
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.
BW2 | BW3 | CB1 | DT1 | DT2 | DT3 | EF1 | KD2 | RM1 | SB5 | SY4 | |
---|---|---|---|---|---|---|---|---|---|---|---|
Ave. F-Measure Onset-Offset | 0.3625 | 0.3254 | 0.2558 | 0.2376 | 0.2787 | 0.2820 | 0.5817 | 0.4380 | 0.2655 | 0.1439 | 0.2908 |
Ave. F-Measure Onset Only | 0.5845 | 0.5354 | 0.4828 | 0.3889 | 0.4475 | 0.4506 | 0.8213 | 0.6592 | 0.4367 | 0.5471 | 0.4602 |
Ave. F-Measure Chroma | 0.3825 | 0.3712 | 0.2683 | 0.2540 | 0.2957 | 0.2994 | 0.5821 | 0.4482 | 0.2830 | 0.1681 | 0.3133 |
Ave. F-Measure Onset Only Chroma | 0.6132 | 0.5965 | 0.5169 | 0.4224 | 0.4823 | 0.4855 | 0.8107 | 0.6650 | 0.4749 | 0.6101 | 0.5007 |
Detailed Results
Precision | Recall | Ave. F-measure | Ave. Overlap | |
---|---|---|---|---|
BW2 | 0.355 | 0.387 | 0.363 | 0.878 |
BW3 | 0.285 | 0.399 | 0.325 | 0.882 |
CB1 | 0.241 | 0.285 | 0.256 | 0.864 |
DT1 | 0.192 | 0.334 | 0.238 | 0.847 |
DT2 | 0.259 | 0.315 | 0.279 | 0.848 |
DT3 | 0.265 | 0.315 | 0.282 | 0.848 |
EF1 | 0.596 | 0.573 | 0.582 | 0.882 |
KD2 | 0.420 | 0.463 | 0.438 | 0.889 |
RM1 | 0.256 | 0.286 | 0.265 | 0.857 |
SB5 | 0.145 | 0.152 | 0.144 | 0.837 |
SY4 | 0.299 | 0.290 | 0.291 | 0.881 |
Detailed Chroma Results
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)
Precision | Recall | Ave. F-measure | Ave. Overlap | |
---|---|---|---|---|
BW2 | 0.376 | 0.408 | 0.382 | 0.875 |
BW3 | 0.322 | 0.461 | 0.371 | 0.881 |
CB1 | 0.253 | 0.299 | 0.268 | 0.862 |
DT1 | 0.204 | 0.358 | 0.254 | 0.843 |
DT2 | 0.275 | 0.335 | 0.296 | 0.847 |
DT3 | 0.281 | 0.335 | 0.299 | 0.846 |
EF1 | 0.597 | 0.573 | 0.582 | 0.881 |
KD2 | 0.430 | 0.473 | 0.448 | 0.887 |
RM1 | 0.272 | 0.305 | 0.283 | 0.856 |
SB5 | 0.168 | 0.179 | 0.168 | 0.860 |
SY4 | 0.323 | 0.313 | 0.313 | 0.880 |
Results Based on Onset Only
Precision | Recall | Ave. F-measure | Ave. Overlap | |
---|---|---|---|---|
BW2 | 0.597 | 0.599 | 0.584 | 0.741 |
BW3 | 0.473 | 0.645 | 0.535 | 0.735 |
CB1 | 0.465 | 0.523 | 0.483 | 0.683 |
DT1 | 0.317 | 0.540 | 0.389 | 0.686 |
DT2 | 0.421 | 0.501 | 0.447 | 0.695 |
DT3 | 0.428 | 0.500 | 0.451 | 0.695 |
EF1 | 0.843 | 0.807 | 0.821 | 0.774 |
KD2 | 0.638 | 0.690 | 0.659 | 0.788 |
RM1 | 0.425 | 0.469 | 0.437 | 0.703 |
SB5 | 0.529 | 0.598 | 0.547 | 0.579 |
SY4 | 0.488 | 0.448 | 0.460 | 0.692 |
Chroma Results Based on Onset Only
Precision | Recall | Ave. F-measure | Ave. Overlap | |
---|---|---|---|---|
BW2 | 0.626 | 0.630 | 0.613 | 0.727 |
BW3 | 0.524 | 0.726 | 0.596 | 0.715 |
CB1 | 0.497 | 0.562 | 0.517 | 0.659 |
DT1 | 0.344 | 0.588 | 0.422 | 0.630 |
DT2 | 0.454 | 0.541 | 0.482 | 0.665 |
DT3 | 0.461 | 0.540 | 0.486 | 0.664 |
EF1 | 0.832 | 0.796 | 0.811 | 0.767 |
KD2 | 0.644 | 0.696 | 0.665 | 0.777 |
RM1 | 0.462 | 0.511 | 0.475 | 0.684 |
SB5 | 0.587 | 0.670 | 0.610 | 0.582 |
SY4 | 0.529 | 0.489 | 0.501 | 0.637 |
Run Times
sub_ID | Runtime(sec) |
---|---|
BW2 | 3078 |
BW3 | 1593 |
CB1 | 7493 |
CD3 (piano only) | 28929 |
DT1 | ~72000 |
DT2 | 79458 |
DT3 | 77877 |
EF1 | 23400 |
KD2 | 180 |
RM1 | 3309 |
SB5 | 180 |
SY4 | ~600 |
Friedman Tests for Note Tracking
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.
Tukey-Kramer HSD Multi-Comparison for Task2
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
EF1 | KD2 | -0.8534 | 1.7353 | 4.3240 | FALSE |
EF1 | BW2 | 0.5583 | 3.1471 | 5.7358 | TRUE |
EF1 | SB5 | 1.8083 | 4.3971 | 6.9858 | TRUE |
EF1 | BW3 | 2.1613 | 4.7500 | 7.3387 | TRUE |
EF1 | CB1 | 3.5583 | 6.1471 | 8.7358 | TRUE |
EF1 | SY4 | 3.4995 | 6.0882 | 8.6770 | TRUE |
EF1 | DT3 | 3.3377 | 5.9265 | 8.5152 | TRUE |
EF1 | DT2 | 3.5730 | 6.1618 | 8.7505 | TRUE |
EF1 | RM1 | 3.4995 | 6.0882 | 8.6770 | TRUE |
EF1 | DT1 | 5.7054 | 8.2941 | 10.8829 | TRUE |
KD2 | BW2 | -1.1770 | 1.4118 | 4.0005 | FALSE |
KD2 | SB5 | 0.0730 | 2.6618 | 5.2505 | TRUE |
KD2 | BW3 | 0.4260 | 3.0147 | 5.6034 | TRUE |
KD2 | CB1 | 1.8230 | 4.4118 | 7.0005 | TRUE |
KD2 | SY4 | 1.7642 | 4.3529 | 6.9417 | TRUE |
KD2 | DT3 | 1.6024 | 4.1912 | 6.7799 | TRUE |
KD2 | DT2 | 1.8377 | 4.4265 | 7.0152 | TRUE |
KD2 | RM1 | 1.7642 | 4.3529 | 6.9417 | TRUE |
KD2 | DT1 | 3.9701 | 6.5588 | 9.1476 | TRUE |
BW2 | SB5 | -1.3387 | 1.2500 | 3.8387 | FALSE |
BW2 | BW3 | -0.9858 | 1.6029 | 4.1917 | FALSE |
BW2 | CB1 | 0.4113 | 3.0000 | 5.5887 | TRUE |
BW2 | SY4 | 0.3524 | 2.9412 | 5.5299 | TRUE |
BW2 | DT3 | 0.1907 | 2.7794 | 5.3681 | TRUE |
BW2 | DT2 | 0.4260 | 3.0147 | 5.6034 | TRUE |
BW2 | RM1 | 0.3524 | 2.9412 | 5.5299 | TRUE |
BW2 | DT1 | 2.5583 | 5.1471 | 7.7358 | TRUE |
SB5 | BW3 | -2.2358 | 0.3529 | 2.9417 | FALSE |
SB5 | CB1 | -0.8387 | 1.7500 | 4.3387 | FALSE |
SB5 | SY4 | -0.8976 | 1.6912 | 4.2799 | FALSE |
SB5 | DT3 | -1.0593 | 1.5294 | 4.1181 | FALSE |
SB5 | DT2 | -0.8240 | 1.7647 | 4.3534 | FALSE |
SB5 | RM1 | -0.8976 | 1.6912 | 4.2799 | FALSE |
SB5 | DT1 | 1.3083 | 3.8971 | 6.4858 | TRUE |
BW3 | CB1 | -1.1917 | 1.3971 | 3.9858 | FALSE |
BW3 | SY4 | -1.2505 | 1.3382 | 3.9270 | FALSE |
BW3 | DT3 | -1.4123 | 1.1765 | 3.7652 | FALSE |
BW3 | DT2 | -1.1770 | 1.4118 | 4.0005 | FALSE |
BW3 | RM1 | -1.2505 | 1.3382 | 3.9270 | FALSE |
BW3 | DT1 | 0.9554 | 3.5441 | 6.1329 | TRUE |
CB1 | SY4 | -2.6476 | -0.0588 | 2.5299 | FALSE |
CB1 | DT3 | -2.8093 | -0.2206 | 2.3681 | FALSE |
CB1 | DT2 | -2.5740 | 0.0147 | 2.6034 | FALSE |
CB1 | RM1 | -2.6476 | -0.0588 | 2.5299 | FALSE |
CB1 | DT1 | -0.4417 | 2.1471 | 4.7358 | FALSE |
SY4 | DT3 | -2.7505 | -0.1618 | 2.4270 | FALSE |
SY4 | DT2 | -2.5152 | 0.0735 | 2.6623 | FALSE |
SY4 | RM1 | -2.5887 | 0.0000 | 2.5887 | FALSE |
SY4 | DT1 | -0.3829 | 2.2059 | 4.7946 | FALSE |
DT3 | DT2 | -2.3534 | 0.2353 | 2.8240 | FALSE |
DT3 | RM1 | -2.4270 | 0.1618 | 2.7505 | FALSE |
DT3 | DT1 | -0.2211 | 2.3676 | 4.9564 | FALSE |
DT2 | RM1 | -2.6623 | -0.0735 | 2.5152 | FALSE |
DT2 | DT1 | -0.4564 | 2.1324 | 4.7211 | FALSE |
RM1 | DT1 | -0.3829 | 2.2059 | 4.7946 | FALSE |
NT Piano-Only Overall Summary Results
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger. 6 piano recordings are evaluated separately for this subtask.
BW2 | BW3 | CB1 | CD3 | DT1 | DT2 | DT3 | EF1 | KD2 | RM1 | SB5 | SY4 | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Ave. F-Measure Onset-Offset | 0.1537 | 0.2051 | 0.1850 | 0.1948 | 0.1505 | 0.1742 | 0.1745 | 0.2942 | 0.1719 | 0.0546 | 0.0423 | 0.1337 |
Ave. F-Measure Onset Only | 0.5588 | 0.6268 | 0.6635 | 0.4174 | 0.3201 | 0.3834 | 0.3813 | 0.8016 | 0.6778 | 0.2194 | 0.6802 | 0.4963 |
Ave. F-Measure Chroma | 0.1675 | 0.2176 | 0.2029 | 0.2341 | 0.1574 | 0.1749 | 0.1759 | 0.2737 | 0.1517 | 0.0624 | 0.0443 | 0.1413 |
Ave. F-Measure Onset Only Chroma | 0.5727 | 0.6393 | 0.6747 | 0.4940 | 0.3385 | 0.3914 | 0.3892 | 0.7347 | 0.6165 | 0.2391 | 0.6876 | 0.5233 |
Detailed Results
Precision | Recall | Ave. F-measure | Ave. Overlap | |
---|---|---|---|---|
BW2 | 0.163 | 0.146 | 0.154 | 0.837 |
BW3 | 0.198 | 0.213 | 0.205 | 0.842 |
CB1 | 0.203 | 0.171 | 0.185 | 0.813 |
CD3 | 0.187 | 0.207 | 0.195 | 0.728 |
DT1 | 0.121 | 0.222 | 0.150 | 0.752 |
DT2 | 0.165 | 0.199 | 0.174 | 0.760 |
DT3 | 0.163 | 0.201 | 0.175 | 0.759 |
EF1 | 0.313 | 0.278 | 0.294 | 0.835 |
KD2 | 0.166 | 0.180 | 0.172 | 0.838 |
RM1 | 0.050 | 0.062 | 0.055 | 0.799 |
SB5 | 0.041 | 0.044 | 0.042 | 0.773 |
SY4 | 0.155 | 0.119 | 0.134 | 0.834 |
Detailed Chroma Results
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)
Precision | Recall | Ave. F-measure | Ave. Overlap | |
---|---|---|---|---|
BW2 | 0.178 | 0.159 | 0.168 | 0.828 |
BW3 | 0.210 | 0.226 | 0.218 | 0.832 |
CB1 | 0.221 | 0.189 | 0.203 | 0.805 |
CD3 | 0.226 | 0.248 | 0.234 | 0.859 |
DT1 | 0.126 | 0.239 | 0.157 | 0.737 |
DT2 | 0.165 | 0.202 | 0.175 | 0.760 |
DT3 | 0.163 | 0.205 | 0.176 | 0.753 |
EF1 | 0.292 | 0.259 | 0.274 | 0.836 |
KD2 | 0.147 | 0.158 | 0.152 | 0.837 |
RM1 | 0.057 | 0.071 | 0.062 | 0.804 |
SB5 | 0.043 | 0.046 | 0.044 | 0.774 |
SY4 | 0.164 | 0.126 | 0.141 | 0.833 |
Results Based on Onset Only
Precision | Recall | Ave. F-measure | Ave. Overlap | |
---|---|---|---|---|
BW2 | 0.590 | 0.534 | 0.559 | 0.558 |
BW3 | 0.602 | 0.655 | 0.627 | 0.583 |
CB1 | 0.700 | 0.636 | 0.664 | 0.566 |
CD3 | 0.422 | 0.430 | 0.417 | 0.555 |
DT1 | 0.253 | 0.508 | 0.320 | 0.553 |
DT2 | 0.353 | 0.461 | 0.383 | 0.554 |
DT3 | 0.347 | 0.460 | 0.381 | 0.555 |
EF1 | 0.845 | 0.764 | 0.802 | 0.618 |
KD2 | 0.659 | 0.701 | 0.678 | 0.594 |
RM1 | 0.202 | 0.244 | 0.219 | 0.555 |
SB5 | 0.640 | 0.728 | 0.680 | 0.408 |
SY4 | 0.584 | 0.437 | 0.496 | 0.552 |
Chroma Results Based on Onset Only
Precision | Recall | Ave. F-measure | Ave. Overlap | |
---|---|---|---|---|
BW2 | 0.605 | 0.547 | 0.573 | 0.559 |
BW3 | 0.614 | 0.668 | 0.639 | 0.582 |
CB1 | 0.712 | 0.646 | 0.675 | 0.562 |
CD3 | 0.506 | 0.506 | 0.494 | 0.546 |
DT1 | 0.267 | 0.538 | 0.339 | 0.518 |
DT2 | 0.360 | 0.472 | 0.391 | 0.534 |
DT3 | 0.354 | 0.470 | 0.389 | 0.533 |
EF1 | 0.775 | 0.700 | 0.735 | 0.615 |
KD2 | 0.601 | 0.636 | 0.616 | 0.591 |
RM1 | 0.220 | 0.266 | 0.239 | 0.537 |
SB5 | 0.646 | 0.736 | 0.688 | 0.409 |
SY4 | 0.616 | 0.461 | 0.523 | 0.346 |
Individual Results Files for Task 2
BW2= Emmanouil Benetos, Tillman Weyde
BW3= Emmanouil Benetos, Tillman Weyde
CB1= Chris Cannam, Emmanouil Benetos
CD3= Andrea Cogliati, Zhiyao Duan
DT1= Zhiyao Duan, David Temperley
DT2= Zhiyao Duan, David Temperley
DT3= Zhiyao Duan, David Temperley
EF1= Anders Elowsson, Anders Friberg
KD2= Karin Dressler
RM1= Daniel Recoskie, Richard Mann
SB5= Sebastian Böck
SY4= Li Su, Yi-Hsuan Yang