2019:Multiple Fundamental Frequency Estimation & Tracking Results - MIREX Dataset

From MIREX Wiki

Introduction

These are the results for the 2018 running of the Multiple Fundamental Frequency Estimation and Tracking task on MIREX dataset. For background information about this task set please refer to the 2019:Multiple Fundamental Frequency Estimation & Tracking page.

General Legend

Sub code Submission name Abstract Contributors
AR2 qhear PDF Anton Runov
BK1 (piano subtask) PianoTranscriptor.2019 PDF Sebastian Böck, Rainer Kelz
CB1 Silvet PDF Chris Cannam, Emmanouil Benetos
CB2 Silvet Live PDF Chris Cannam, Emmanouil Benetos
HH2 (task1) mffet v.1 PDF Huang Hsiang-Yu
KN3 (piano subtask) AR_BEAM PDF Taegyun Kwon, Juhan Nam, Dasaem Jung
KNJ1 (piano subtask) AR_SIMPLE PDF Taegyun Kwon, Juhan Nam, Dasaem Jung
KY1 (task2) MetaAI_Ensemble_SingleDomain PDF Changhyun Kim, Sangeon Yong
KY2 (task2) MetaAI_SingleModel_MultiDomain PDF Changhyun Kim, Sangeon Yong
SBJ1-4 (task1) SBJ1-4 PDF Peter Steiner, Peter Birkholz, Azarakhsh Jalalvand
YK1 (task2) MetaAI_Ensemble_SingleDomain PDF Sangeon Yong, Changhyun Kim
YK2 (task2) MetaAI_SingleModel_SingleDomain PDF Sangeon Yong, Changhyun Kim

Task 1: Multiple Fundamental Frequency Estimation (MF0E)

MF0E Overall Summary Results

Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

Detailed Results

Precision Recall Accuracy Etot Esubs Emiss Efa
AR2 0.748 0.833 0.690 0.406 0.081 0.086 0.240
CB1 0.804 0.519 0.498 0.529 0.093 0.389 0.047
CB2 0.656 0.460 0.420 0.636 0.174 0.367 0.095
HH2 0.143 0.218 0.141 1.594 0.773 0.009 0.812
SBJ1 0.746 0.740 0.632 0.446 0.093 0.168 0.186
SBJ2 0.727 0.723 0.618 0.461 0.109 0.168 0.184
SBJ3 0.722 0.754 0.626 0.472 0.095 0.151 0.226
SBJ4 0.724 0.717 0.611 0.470 0.107 0.176 0.187

download these results as csv

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

Precision Recall Accuracy Etot Esubs Emiss Efa
AR2 0.762 0.852 0.703 0.388 0.062 0.086 0.240
CB1 0.851 0.551 0.527 0.497 0.062 0.389 0.047
CB2 0.747 0.527 0.479 0.568 0.106 0.367 0.095
HH2 0.243 0.380 0.240 1.432 0.611 0.009 0.812
SBJ1 0.763 0.758 0.647 0.428 0.074 0.168 0.186
SBJ2 0.747 0.744 0.635 0.440 0.087 0.168 0.184
SBJ3 0.739 0.773 0.640 0.454 0.077 0.151 0.226
SBJ4 0.743 0.737 0.628 0.450 0.087 0.176 0.187

download these results as csv

Individual Results Files for Task 1

AR2= Anton Runov
CB1= Chris Cannam, Emmanouil Benetos
CB2= Chris Cannam, Emmanouil Benetos
HH2= Huang Hsiang-Yu
SBJ1= Peter Steiner, Peter Birkholz, Azarakhsh Jalalvand
SBJ2= Peter Steiner, Peter Birkholz, Azarakhsh Jalalvand
SBJ3= Peter Steiner, Peter Birkholz, Azarakhsh Jalalvand
SBJ4= Peter Steiner, Peter Birkholz, Azarakhsh Jalalvand


Info about the filenames

The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

bs = bassoon, cl = clarinet, fl = flute, hn = horn, ob = oboe, vl = violin, cel = cello, gtr = guitar, sax = saxophone, bass = electric bass guitar

Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)

The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

Tukey-Kramer HSD Multi-Comparison

TeamID TeamID Lowerbound Mean Upperbound Significance
AR2 SBJ1 -0.6721 0.9875 2.6471 FALSE
AR2 SBJ3 0.0029 1.6625 3.3221 TRUE
AR2 SBJ2 0.6279 2.2875 3.9471 TRUE
AR2 SBJ4 1.1029 2.7625 4.4221 TRUE
AR2 CB1 2.0904 3.7500 5.4096 TRUE
AR2 CB2 3.5154 5.1750 6.8346 TRUE
AR2 HH2 4.7154 6.3750 8.0346 TRUE
SBJ1 SBJ3 -0.9846 0.6750 2.3346 FALSE
SBJ1 SBJ2 -0.3596 1.3000 2.9596 FALSE
SBJ1 SBJ4 0.1154 1.7750 3.4346 TRUE
SBJ1 CB1 1.1029 2.7625 4.4221 TRUE
SBJ1 CB2 2.5279 4.1875 5.8471 TRUE
SBJ1 HH2 3.7279 5.3875 7.0471 TRUE
SBJ3 SBJ2 -1.0346 0.6250 2.2846 FALSE
SBJ3 SBJ4 -0.5596 1.1000 2.7596 FALSE
SBJ3 CB1 0.4279 2.0875 3.7471 TRUE
SBJ3 CB2 1.8529 3.5125 5.1721 TRUE
SBJ3 HH2 3.0529 4.7125 6.3721 TRUE
SBJ2 SBJ4 -1.1846 0.4750 2.1346 FALSE
SBJ2 CB1 -0.1971 1.4625 3.1221 FALSE
SBJ2 CB2 1.2279 2.8875 4.5471 TRUE
SBJ2 HH2 2.4279 4.0875 5.7471 TRUE
SBJ4 CB1 -0.6721 0.9875 2.6471 FALSE
SBJ4 CB2 0.7529 2.4125 4.0721 TRUE
SBJ4 HH2 1.9529 3.6125 5.2721 TRUE
CB1 CB2 -0.2346 1.4250 3.0846 FALSE
CB1 HH2 0.9654 2.6250 4.2846 TRUE
CB2 HH2 -0.4596 1.2000 2.8596 FALSE

download these results as csv

2019 Accuracy Per Song Friedman Mean Rankstask1.friedman.Friedman Mean Ranks.png

Task 2:Note Tracking (NT)

NT Mixed Set Overall Summary Results

This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

AR2 CB1 CB2 KY1 KY2 YK1 YK2
Ave. F-Measure Onset-Offset 0.4055 0.3047 0.2064 0.4376 0.4150 0.3985 0.3915
Ave. F-Measure Onset Only 0.6244 0.5029 0.3742 0.6745 0.6635 0.6394 0.6422
Ave. F-Measure Chroma 0.4138 0.3207 0.2365 0.4482 0.4277 0.4116 0.4046
Ave. F-Measure Onset Only Chroma 0.6333 0.5343 0.4276 0.6894 0.6855 0.6570 0.6631

download these results as csv

Detailed Results

Precision Recall Ave. F-measure Ave. Overlap
AR2 0.391 0.442 0.406 0.881
CB1 0.312 0.304 0.305 0.865
CB2 0.201 0.230 0.206 0.862
KY1 0.457 0.432 0.438 0.845
KY2 0.420 0.424 0.415 0.848
YK1 0.346 0.494 0.399 0.844
YK2 0.343 0.479 0.392 0.846

download these results as csv

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

Precision Recall Ave. F-measure Ave. Overlap
AR2 0.399 0.451 0.414 0.877
CB1 0.329 0.320 0.321 0.860
CB2 0.229 0.265 0.237 0.858
KY1 0.468 0.443 0.448 0.846
KY2 0.432 0.437 0.428 0.848
YK1 0.357 0.510 0.412 0.845
YK2 0.354 0.496 0.405 0.846

download these results as csv


Results Based on Onset Only

Precision Recall Ave. F-measure Ave. Overlap
AR2 0.615 0.662 0.624 0.757
CB1 0.525 0.493 0.503 0.720
CB2 0.375 0.403 0.374 0.677
KY1 0.694 0.682 0.674 0.692
KY2 0.664 0.692 0.664 0.683
YK1 0.554 0.800 0.639 0.670
YK2 0.561 0.793 0.642 0.668

download these results as csv

Chroma Results Based on Onset Only

Precision Recall Ave. F-measure Ave. Overlap
AR2 0.623 0.673 0.633 0.731
CB1 0.558 0.524 0.534 0.700
CB2 0.426 0.465 0.428 0.652
KY1 0.708 0.697 0.689 0.686
KY2 0.684 0.716 0.686 0.671
YK1 0.569 0.823 0.657 0.659
YK2 0.578 0.820 0.663 0.654

download these results as csv


Friedman Tests for Note Tracking

The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

Tukey-Kramer HSD Multi-Comparison for Task2
TeamID TeamID Lowerbound Mean Upperbound Significance
KY1 KY2 -1.0880 0.4559 1.9998 FALSE
KY1 YK2 -0.2939 1.2500 2.7939 FALSE
KY1 YK1 -0.0880 1.4559 2.9998 FALSE
KY1 AR2 0.1325 1.6765 3.2204 TRUE
KY1 CB1 1.5002 3.0441 4.5880 TRUE
KY1 CB2 2.9267 4.4706 6.0145 TRUE
KY2 YK2 -0.7498 0.7941 2.3380 FALSE
KY2 YK1 -0.5439 1.0000 2.5439 FALSE
KY2 AR2 -0.3233 1.2206 2.7645 FALSE
KY2 CB1 1.0443 2.5882 4.1322 TRUE
KY2 CB2 2.4708 4.0147 5.5586 TRUE
YK2 YK1 -1.3380 0.2059 1.7498 FALSE
YK2 AR2 -1.1175 0.4265 1.9704 FALSE
YK2 CB1 0.2502 1.7941 3.3380 TRUE
YK2 CB2 1.6767 3.2206 4.7645 TRUE
YK1 AR2 -1.3233 0.2206 1.7645 FALSE
YK1 CB1 0.0443 1.5882 3.1322 TRUE
YK1 CB2 1.4708 3.0147 4.5586 TRUE
AR2 CB1 -0.1763 1.3676 2.9116 FALSE
AR2 CB2 1.2502 2.7941 4.3380 TRUE
CB1 CB2 -0.1175 1.4265 2.9704 FALSE

download these results as csv

2019 Accuracy Per Song Friedman Mean Rankstask2.onsetOnly.friedman.Friedman Mean Ranks.png

NT Piano-Only Overall Summary Results

This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger. 6 piano recordings are evaluated separately for this subtask.

AR2 BK1 CB1 CB2 KN3 KNJ1 KY1 KY2 YK1 YK2
Ave. F-Measure Onset-Offset 0.2339 0.4816 0.2380 0.1750 0.3431 0.3941 0.4476 0.4068 0.4031 0.3976
Ave. F-Measure Onset Only 0.6468 0.6464 0.6681 0.4970 0.6009 0.6653 0.6731 0.6713 0.7023 0.7032
Ave. F-Measure Chroma 0.2254 0.4868 0.2538 0.1863 0.3517 0.4000 0.4537 0.4129 0.4077 0.4041
Ave. F-Measure Onset Only Chroma 0.6118 0.6473 0.6787 0.5145 0.6110 0.6767 0.6787 0.6756 0.7140 0.7145

download these results as csv


Detailed Results

Precision Recall Ave. F-measure Ave. Overlap
AR2 0.264 0.212 0.234 0.806
BK1 0.527 0.447 0.482 0.824
CB1 0.275 0.211 0.238 0.813
CB2 0.209 0.153 0.175 0.797
KN3 0.451 0.299 0.343 0.769
KNJ1 0.475 0.349 0.394 0.809
KY1 0.511 0.407 0.448 0.803
KY2 0.460 0.374 0.407 0.819
YK1 0.414 0.398 0.403 0.804
YK2 0.406 0.395 0.398 0.819

download these results as csv

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

Precision Recall Ave. F-measure Ave. Overlap
AR2 0.254 0.204 0.225 0.801
BK1 0.532 0.453 0.487 0.824
CB1 0.292 0.226 0.254 0.801
CB2 0.221 0.164 0.186 0.796
KN3 0.465 0.306 0.352 0.770
KNJ1 0.483 0.354 0.400 0.809
KY1 0.519 0.413 0.454 0.809
KY2 0.468 0.379 0.413 0.821
YK1 0.419 0.402 0.408 0.809
YK2 0.413 0.402 0.404 0.818

download these results as csv

Results Based on Onset Only

Precision Recall Ave. F-measure Ave. Overlap
AR2 0.720 0.593 0.647 0.583
BK1 0.701 0.607 0.646 0.720
CB1 0.744 0.613 0.668 0.585
CB2 0.561 0.454 0.497 0.576
KN3 0.779 0.524 0.601 0.602
KNJ1 0.793 0.595 0.665 0.615
KY1 0.792 0.600 0.673 0.624
KY2 0.785 0.602 0.671 0.588
YK1 0.736 0.680 0.702 0.553
YK2 0.732 0.687 0.703 0.584

download these results as csv

Chroma Results Based on Onset Only

Precision Recall Ave. F-measure Ave. Overlap
AR2 0.681 0.561 0.612 0.582
BK1 0.702 0.608 0.647 0.713
CB1 0.755 0.623 0.679 0.585
CB2 0.580 0.470 0.514 0.577
KN3 0.799 0.531 0.611 0.598
KNJ1 0.811 0.603 0.677 0.606
KY1 0.799 0.605 0.679 0.617
KY2 0.791 0.606 0.676 0.582
YK1 0.749 0.691 0.714 0.543
YK2 0.745 0.697 0.715 0.571

download these results as csv

Individual Results Files for Task 2

AR2= Anton Runov
CB1= Chris Cannam, Emmanouil Benetos
CB2= Chris Cannam, Emmanouil Benetos
BK1= Sebastian Böck, Rainer Kelz
KN3= Taegyun Kwon, Juhan Nam, Dasaem Jung
KNJ1= Taegyun Kwon, Juhan Nam, Dasaem Jung
KY1= Changhyun Kim, Sangeon Yong
KY2= Changhyun Kim, Sangeon Yong
YK1= Sangeon Yong, Changhyun Kim
YK2= Sangeon Yong, Changhyun Kim

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2019:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_-_MIREX_Dataset&oldid=13268"