2010:Multiple Fundamental Frequency Estimation & Tracking Results

Introduction

These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the 2009:Multiple Fundamental Frequency Estimation & Tracking page.

General Legend

Sub code	Submission name	Abstract	Contributors
AR1	MultiF0_Yeh_IRCAM 1 Band MF0Est	PDF	Chunghsin Yeh, Axel Roebel
AR2	MultiF0_Yeh_IRCAM 20 Band MF0Est	PDF	Chunghsin Yeh, Axel Roebel
AR3	MultiF0_Yeh_IRCAM 1 Band Tracking MF0Est	PDF	Chunghsin Yeh, Axel Roebel
AR4	MultiF0_Yeh_IRCAM 20 Band Tracking MF0Est	PDF	Chunghsin Yeh, Axel Roebel
AR5	MultiF0_Yeh_IRCAM 1 Band Note Tracking	PDF	Chunghsin Yeh, Axel Roebel
AR6	MultiF0_Yeh_IRCAM 20 Band Note Tracking	PDF	Chunghsin Yeh, Axel Roebel
BD1	Frame-BasedMultiF0	PDF	Emmanouil Benetos, Simon Dixon
CRVRC1	MF0UJAV1T1 MultiF0 Est	PDF	Francisco Jesus Canadas Quesada, Francisco Jose Rodriguez Serrano, Pedro Vera-Candeas, Nicolas Ruiz Reyes, Julio José Carabias-Orti
CRVRC2	MF0UJAV1T2 Note Tracking	PDF	Francisco Jesus Canadas Quesada, Francisco Jose Rodriguez Serrano, Pedro Vera-Candeas, Nicolas Ruiz Reyes, Julio José Carabias-Orti
CRVRC3	MF0UJAV2T1 MultiF0 Est	PDF	Francisco Jesus Canadas Quesada, Francisco Jose Rodriguez Serrano, Pedro Vera-Candeas, Nicolas Ruiz Reyes, Julio José Carabias-Orti
CRVRC4	MF0UJAV2T2 Note Tracking	PDF	Francisco Jesus Canadas Quesada, Francisco Jose Rodriguez Serrano, Pedro Vera-Candeas, Nicolas Ruiz Reyes, Julio José Carabias-Orti
DCL1	Multi-Pitch ND Beta MultiF0 Est	PDF	Arnaud Dessein, Arshia Cont, Guillaume Lemaitre
DCL2	Multi-Pitch ND Beta Note Tracking	PDF	Arnaud Dessein, Arshia Cont, Guillaume Lemaitre
DHP1	DuanHanPardo_MultiF0EstTrack MultiF0 Est	PDF	Zhiyao Duan, Jinyu Han, Bryan Pardo
DHP2	DuanHanPardo_MultiF0EstTrack Note Tracking	PDF	Zhiyao Duan, Jinyu Han, Bryan Pardo
DHP3	DuanHanPardo_MultiF0EstTrack Instrument Tracking	PDF	Zhiyao Duan, Jinyu Han, Bryan Pardo
JW1	flexible harmonic model for multipitch estimation MultiF0 Est	PDF	Jun Wu
JW2	Harmonic Temporal Model for Multiple Pitch Estimation MultiF0 Est	PDF	Jun Wu
JW3	flexible harmonic model for multipitch estimation Note Tracking	PDF	Jun Wu
JW4	Harmonic Temporal Model for Multiple Pitch Estimation Note Tracking	PDF	Jun Wu
LYLC1	CTL_MF0E Multi F0 Est	PDF	Cheng-Te Lee, Yi-Hsuan Yang, Keng-Sheng Lin, Homer Chen
LYLC2	CTL_MF0E Note Tracking	PDF	Cheng-Te Lee, Yi-Hsuan Yang, Keng-Sheng Lin, Homer Chen
NNTOS1	MTINS	PDF	Masahiro Nakano, Tomohiko Nakamura, Iwao Tanuma, Nobutaka Ono, Shigeki Sagayama

Task 1: Multiple Fundamental Frequency Estimation (MF0E)

MF0E Overall Summary Results

Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

	AR1	AR2	AR3	AR4	BD1	CRVRC1	CRVRC3	DCL1	DHP1	JW1	JW2	LYLC1	NNTOS1
Accuracy	0.654	0.655	0.692	0.692	0.468	0.49	0.457	0.457	0.553	0.354	0.361	0.373	0.06
Accuracy Chroma	0.68	0.681	0.71	0.71	0.545	0.544	0.52	0.524	0.594	0.459	0.473	0.457	0.109

download these results as csv

Detailed Results

	Precision	Recall	Accuracy	Etot	Esubs	Emiss	Efa
AR1	0.721	0.799	0.654	0.448	0.096	0.105	0.247
AR2	0.722	0.799	0.655	0.446	0.095	0.106	0.245
AR3	0.752	0.829	0.692	0.391	0.084	0.088	0.219
AR4	0.753	0.828	0.692	0.390	0.084	0.088	0.218
BD1	0.716	0.485	0.468	0.558	0.167	0.349	0.043
CRVRC1	0.638	0.556	0.490	0.598	0.198	0.246	0.154
CRVRC3	0.515	0.637	0.457	0.837	0.227	0.136	0.474
DCL1	0.499	0.676	0.457	0.847	0.206	0.119	0.523
DHP1	0.712	0.632	0.553	0.518	0.129	0.239	0.150
JW1	0.369	0.579	0.354	1.146	0.361	0.060	0.725
JW2	0.384	0.541	0.361	1.033	0.382	0.077	0.575
LYLC1	0.584	0.399	0.373	0.681	0.228	0.373	0.080
NNTOS1	0.142	0.061	0.060	0.967	0.277	0.661	0.028

download these results as csv

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

	Precision	Recall	Accuracy	Etot	Esubs	Emiss	Efa
AR1	0.750	0.833	0.680	0.415	0.063	0.105	0.247
AR2	0.752	0.832	0.681	0.413	0.063	0.106	0.245
AR3	0.772	0.851	0.710	0.368	0.061	0.088	0.219
AR4	0.772	0.851	0.710	0.367	0.061	0.088	0.218
BD1	0.830	0.567	0.545	0.476	0.085	0.349	0.043
CRVRC1	0.707	0.621	0.544	0.533	0.134	0.246	0.154
CRVRC3	0.587	0.732	0.520	0.742	0.132	0.136	0.474
DCL1	0.574	0.775	0.524	0.748	0.107	0.119	0.523
DHP1	0.766	0.678	0.594	0.471	0.082	0.239	0.150
JW1	0.477	0.767	0.459	0.958	0.172	0.060	0.725
JW2	0.501	0.726	0.473	0.849	0.197	0.077	0.575
LYLC1	0.715	0.490	0.457	0.590	0.137	0.373	0.080
NNTOS1	0.303	0.112	0.109	0.916	0.227	0.661	0.028

download these results as csv

Individual Results Files for Task 1

Info about the filenames

The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

bs = bassoon, cl = clarinet, fl = flute, hn = horn, ob = oboe, vl = violin, cel = cello, gtr = guitar, sax = saxophone, bass = electric bass guitar

Run Times

file /nema-raid/www/mirex/results/2009/multif0/task1_runtimes.csv not found

TBA

Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)

The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

Tukey-Kramer HSD Multi-Comparison

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
AR4	AR3	-2.7597	0.1250	3.0097	FALSE
AR4	AR2	-0.7472	2.1375	5.0222	FALSE
AR4	AR1	-0.3222	2.5625	5.4472	FALSE
AR4	DHP1	1.1153	4.0000	6.8847	TRUE
AR4	CRVRC1	2.3653	5.2500	8.1347	TRUE
AR4	BD1	3.1653	6.0500	8.9347	TRUE
AR4	DCL1	3.4403	6.3250	9.2097	TRUE
AR4	CRVRC3	3.4403	6.3250	9.2097	TRUE
AR4	LYLC1	5.3653	8.2500	11.1347	TRUE
AR4	JW2	5.9153	8.8000	11.6847	TRUE
AR4	JW1	6.0903	8.9750	11.8597	TRUE
AR4	NNTOS1	8.5153	11.4000	14.2847	TRUE
AR3	AR2	-0.8722	2.0125	4.8972	FALSE
AR3	AR1	-0.4472	2.4375	5.3222	FALSE
AR3	DHP1	0.9903	3.8750	6.7597	TRUE
AR3	CRVRC1	2.2403	5.1250	8.0097	TRUE
AR3	BD1	3.0403	5.9250	8.8097	TRUE
AR3	DCL1	3.3153	6.2000	9.0847	TRUE
AR3	CRVRC3	3.3153	6.2000	9.0847	TRUE
AR3	LYLC1	5.2403	8.1250	11.0097	TRUE
AR3	JW2	5.7903	8.6750	11.5597	TRUE
AR3	JW1	5.9653	8.8500	11.7347	TRUE
AR3	NNTOS1	8.3903	11.2750	14.1597	TRUE
AR2	AR1	-2.4597	0.4250	3.3097	FALSE
AR2	DHP1	-1.0222	1.8625	4.7472	FALSE
AR2	CRVRC1	0.2278	3.1125	5.9972	TRUE
AR2	BD1	1.0278	3.9125	6.7972	TRUE
AR2	DCL1	1.3028	4.1875	7.0722	TRUE
AR2	CRVRC3	1.3028	4.1875	7.0722	TRUE
AR2	LYLC1	3.2278	6.1125	8.9972	TRUE
AR2	JW2	3.7778	6.6625	9.5472	TRUE
AR2	JW1	3.9528	6.8375	9.7222	TRUE
AR2	NNTOS1	6.3778	9.2625	12.1472	TRUE
AR1	DHP1	-1.4472	1.4375	4.3222	FALSE
AR1	CRVRC1	-0.1972	2.6875	5.5722	FALSE
AR1	BD1	0.6028	3.4875	6.3722	TRUE
AR1	DCL1	0.8778	3.7625	6.6472	TRUE
AR1	CRVRC3	0.8778	3.7625	6.6472	TRUE
AR1	LYLC1	2.8028	5.6875	8.5722	TRUE
AR1	JW2	3.3528	6.2375	9.1222	TRUE
AR1	JW1	3.5278	6.4125	9.2972	TRUE
AR1	NNTOS1	5.9528	8.8375	11.7222	TRUE
DHP1	CRVRC1	-1.6347	1.2500	4.1347	FALSE
DHP1	BD1	-0.8347	2.0500	4.9347	FALSE
DHP1	DCL1	-0.5597	2.3250	5.2097	FALSE
DHP1	CRVRC3	-0.5597	2.3250	5.2097	FALSE
DHP1	LYLC1	1.3653	4.2500	7.1347	TRUE
DHP1	JW2	1.9153	4.8000	7.6847	TRUE
DHP1	JW1	2.0903	4.9750	7.8597	TRUE
DHP1	NNTOS1	4.5153	7.4000	10.2847	TRUE
CRVRC1	BD1	-2.0847	0.8000	3.6847	FALSE
CRVRC1	DCL1	-1.8097	1.0750	3.9597	FALSE
CRVRC1	CRVRC3	-1.8097	1.0750	3.9597	FALSE
CRVRC1	LYLC1	0.1153	3.0000	5.8847	TRUE
CRVRC1	JW2	0.6653	3.5500	6.4347	TRUE
CRVRC1	JW1	0.8403	3.7250	6.6097	TRUE
CRVRC1	NNTOS1	3.2653	6.1500	9.0347	TRUE
BD1	DCL1	-2.6097	0.2750	3.1597	FALSE
BD1	CRVRC3	-2.6097	0.2750	3.1597	FALSE
BD1	LYLC1	-0.6847	2.2000	5.0847	FALSE
BD1	JW2	-0.1347	2.7500	5.6347	FALSE
BD1	JW1	0.0403	2.9250	5.8097	TRUE
BD1	NNTOS1	2.4653	5.3500	8.2347	TRUE
DCL1	CRVRC3	-2.8847	0.0000	2.8847	FALSE
DCL1	LYLC1	-0.9597	1.9250	4.8097	FALSE
DCL1	JW2	-0.4097	2.4750	5.3597	FALSE
DCL1	JW1	-0.2347	2.6500	5.5347	FALSE
DCL1	NNTOS1	2.1903	5.0750	7.9597	TRUE
CRVRC3	LYLC1	-0.9597	1.9250	4.8097	FALSE
CRVRC3	JW2	-0.4097	2.4750	5.3597	FALSE
CRVRC3	JW1	-0.2347	2.6500	5.5347	FALSE
CRVRC3	NNTOS1	2.1903	5.0750	7.9597	TRUE
LYLC1	JW2	-2.3347	0.5500	3.4347	FALSE
LYLC1	JW1	-2.1597	0.7250	3.6097	FALSE
LYLC1	NNTOS1	0.2653	3.1500	6.0347	TRUE
JW2	JW1	-2.7097	0.1750	3.0597	FALSE
JW2	NNTOS1	-0.2847	2.6000	5.4847	FALSE
JW1	NNTOS1	-0.4597	2.4250	5.3097	FALSE

download these results as csv

Task 2:Note Tracking (NT)

NT Mixed Set Overall Summary Results

This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

	AR5	AR6	CRVRC2	CRVRC4	DCL2	DHP2	JW3	JW4	LYLC2
Ave. F-Measure Onset-Offset	0.3239	0.3265	0.1577	0.0975	0.2404	0.1934	0.0366	0.0377	0.1179
Ave. F-Measure Onset Only	0.5297	0.5314	0.3299	0.2091	0.4031	0.4099	0.2765	0.2737	0.2889
Ave. F-Measure Chroma	0.3327	0.3357	0.1748	0.1231	0.2851	0.2120	0.0570	0.0575	0.1387
Ave. F-Measure Onset Only Chroma	0.5391	0.5413	0.3623	0.2616	0.4702	0.4626	0.3629	0.3469	0.3495

download these results as csv

Detailed Results

	Precision	Recall	Ave. F-measure	Ave. Overlap
AR5	0.259	0.443	0.324	0.885
AR6	0.261	0.447	0.327	0.884
CRVRC2	0.197	0.136	0.158	0.848
CRVRC4	0.111	0.089	0.097	0.809
DCL2	0.178	0.402	0.240	0.882
DHP2	0.154	0.270	0.193	0.842
JW3	0.037	0.038	0.037	0.855
JW4	0.040	0.037	0.038	0.849
LYLC2	0.122	0.119	0.118	0.788

download these results as csv

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

	Precision	Recall	Ave. F-measure	Ave. Overlap
AR5	0.266	0.456	0.333	0.881
AR6	0.268	0.460	0.336	0.879
CRVRC2	0.218	0.151	0.175	0.848
CRVRC4	0.140	0.112	0.123	0.804
DCL2	0.210	0.479	0.285	0.875
DHP2	0.170	0.295	0.212	0.837
JW3	0.056	0.060	0.057	0.854
JW4	0.059	0.058	0.058	0.846
LYLC2	0.142	0.140	0.139	0.788

download these results as csv

Results Based on Onset Only

	Precision	Recall	Ave. F-measure	Ave. Overlap
AR5	0.425	0.723	0.530	0.727
AR6	0.427	0.724	0.531	0.727
CRVRC2	0.409	0.285	0.330	0.663
CRVRC4	0.240	0.190	0.209	0.625
DCL2	0.305	0.643	0.403	0.728
DHP2	0.346	0.539	0.410	0.616
JW3	0.293	0.274	0.276	0.481
JW4	0.308	0.257	0.274	0.483
LYLC2	0.306	0.282	0.289	0.555

download these results as csv

Chroma Results Based on Onset Only

	Precision	Recall	Ave. F-measure	Ave. Overlap
AR5	0.432	0.738	0.539	0.693
AR6	0.435	0.739	0.541	0.692
CRVRC2	0.451	0.313	0.362	0.649
CRVRC4	0.299	0.238	0.262	0.607
DCL2	0.356	0.750	0.470	0.690
DHP2	0.389	0.612	0.463	0.582
JW3	0.379	0.364	0.363	0.473
JW4	0.386	0.329	0.347	0.478
LYLC2	0.368	0.344	0.350	0.528

download these results as csv