2013:Discovery of Repeated Themes & Sections Results

This task report is currently under construction. Plots (png files) are missing currently due to a technical problem. Please check back soon for these.

Introduction

The task: algorithms take a piece of music as input, and output a list of patterns repeated within that piece. A pattern is defined as a set of ontime-pitch pairs that occurs twice (i.e., is repeated at least once) in a piece of music. The second, third, etc. occurrences of the pattern will likely be shifted in time and/or transposed, relative to the first occurrence. Ideally an algorithm will be able to discover all exact and inexact occurrences of a pattern within a piece, so in evaluating this task we are interested in both

(1) whether an algorithm can discover one occurrence, up to time shift and transposition, and
(2) to what extent it can find all occurrences.

The metrics establishment recall, establishment precision and establishment F1 address (1), and the metrics occurrence recall, occurrence precision, and occurrence F1 address (2).

Existing approaches to music structure analysis in MIR tend to focus on segmentation. The contribution of this task is to afford access to the note content itself; not just requiring algorithms to label time windows. The ground truth also contains nested patterns, reflecting the often hierarchical nature of musical repetition.

Ground Truth and Algorithms

The ground truth, called the Johannes Kepler University Patterns Test Database (JKUPTD-Aug2013), is based on motifs and themes in Barlow and Morgenstern (1953), Schoenberg (1967), and Bruhn (1993). Repeated sections are based on those marked by the composer. These annotations are supplemented with some of our own where necessary. A Development Database (JKUPDD-Aug2013) released in March enabled participants to try out their algorithms. For each piece in the Development and Test Databases, symbolic and synthesised audio versions are crossed with monophonic and polyphonic versions, giving up to four versions of the task in total: symPoly, symMono, audPoly, and audMono. Algorithms submitted to the task are are shown in Table 1.

Results

In Brief

To avoid a bias toward the more numerous submissions of Meredith (2013), DM10 was preselected for comparison with Nieto and Farbood's (2013) submissions, based on reported performance for the Development Database. Figure 2 shows establishment recall results on a per-pattern basis for the symbolic-polyphonic version of the task. DM10 outperforms NF2 according to Friedman's test ( $\chi ^{2}(1)=12.7368,\ p<.001$ ), suggesting the former is preferable for discovering at least one occurrence of each ground truth pattern. This result addresses point (1) from the introduction.

Figure 3 shows occurrence recall results on a per-pattern basis for the symbolic-polyphonic version of the task. Again, DM10 outperforms NF2 according to Friedman's test ( $\chi ^{2}(1)=11.2667,\ p<.001$ ), suggesting the former is preferable for retrieving all occurrences of a discovered ground truth pattern. This result addresses point (2) from the introduction.

It should be noted, however, that according to Friedman's test, algorithm NF2 is significantly faster than DM10 ( $\chi ^{2}(1)=4.000,\ p<.05$ ), suggesting that the former is preferable for rapid summarisation.

The results are closer for the symbolic-monophonic version of the task. According to Friedman's test ( $\chi ^{2}(1)=2.9512,\ p=.0858$ , ns), NF1 and DM10 show no significant difference for establishment recall on a per-pattern basis (please see Figure 24). This result relates to point (1) from the introduction. Figure 25 shows occurrence recall results on a per-pattern basis for the symbolic-monophonic version of the task. DM10 outperforms NF1 according to Friedman's test ( $\chi ^{2}(1)=11.6364,\ p<.001$ ), suggesting the former is preferable for retrieving all occurrences of a discovered ground truth pattern. This result relates to point (2) from the introduction.

For the audio versions of the task, algorithms NF3 (audMono) and NF4 (audPoly) were the only submissions. Nieto and Farbood (2013) are credited with being the first to define an algorithm for discovering repeated note content in a synthesised audio file that is time-locked to a symbolic representation. We include the results for this version of the task for the sake of comparison in future years.

symPoly

Figure 2. Test figure. Proper figures to follow.

AlgId	TaskVersion	Piece	n_P	n_Q	P_est	R_est	F1_est	P_occ(c=.75)	R_occ(c=.75)	F_1occ(c=.75)	P_3	R_3	TLF_1	runtime	FFTP_est	FFP	P_occ(c=.5)	R_occ(c=.5)	F_1occ(c=.5)
NF2	symPoly	piece1	5	5	0.240	0.222	0.231	0.000	0.000	0.000	0.143	0.142	0.142	15.000	0.222	0.143	0.000	0.000	0.000
NF2	symPoly	piece2	5	27.000	0.277	0.446	0.342	0.793	0.793	0.793	0.078	0.253	0.120	1221.000	0.399	0.235	0.409	0.411	0.410
NF2	symPoly	piece3	10.000	20.000	0.695	0.584	0.635	0.703	0.320	0.440	0.473	0.439	0.455	34.000	0.355	0.473	0.603	0.373	0.461
NF2	symPoly	piece4	5	2	0.667	0.272	0.386	0.885	0.885	0.885	0.609	0.240	0.344	3.000	0.272	0.609	0.885	0.885	0.885
NF2	symPoly	piece5	13.000	18.000	0.564	0.334	0.419	0.690	0.393	0.501	0.417	0.345	0.377	153.000	0.245	0.549	0.601	0.488	0.539
DM5	symPoly	piece1	5	23.000	0.332	0.545	0.412	0.743	0.804	0.772	0.228	0.441	0.301	474.000	0.463	0.531	0.460	0.593	0.518
DM5	symPoly	piece2	5	38.000	0.287	0.447	0.349	0.385	0.770	0.513	0.235	0.339	0.278	19896.000	0.274	0.238	0.328	0.770	0.460
DM5	symPoly	piece3	10.000	10.000	0.330	0.423	0.371	0.000	0.000	0.000	0.285	0.365	0.320	762.000	0.288	0.370	0.318	0.471	0.379
DM5	symPoly	piece4	5	4	0.400	0.349	0.372	0.000	0.000	0.000	0.243	0.187	0.211	14.000	0.349	0.243	0.238	0.195	0.214
DM5	symPoly	piece5	13.000	33.000	0.305	0.300	0.303	0.693	0.805	0.745	0.301	0.328	0.314	36299.000	0.160	0.328	0.620	0.765	0.685
DM6	symPoly	piece1	5	23.000	0.215	0.381	0.275	0.000	0.000	0.000	0.163	0.346	0.221	500.000	0.315	0.365	0.454	0.207	0.284
DM6	symPoly	piece2	5	38.000	0.090	0.283	0.136	0.000	0.000	0.000	0.046	0.175	0.073	23294.000	0.207	0.102	0.100	0.033	0.050
DM6	symPoly	piece3	10.000	10.000	0.145	0.164	0.154	0.000	0.000	0.000	0.152	0.204	0.174	771.000	0.143	0.209	0.000	0.000	0.000
DM6	symPoly	piece4	5	4	0.321	0.234	0.271	0.000	0.000	0.000	0.165	0.127	0.143	13.000	0.234	0.165	0.250	0.125	0.167
DM6	symPoly	piece5	13.000	33.000	0.119	0.189	0.146	0.000	0.000	0.000	0.084	0.193	0.117	37646.000	0.135	0.175	0.000	0.000	0.000
DM7	symPoly	piece1	5	23.000	0.316	0.522	0.393	0.406	0.439	0.422	0.203	0.377	0.264	532.000	0.467	0.409	0.405	0.447	0.425
DM7	symPoly	piece2	5	38.000	0.673	0.614	0.642	0.686	0.965	0.802	0.598	0.465	0.523	19926.000	0.368	0.562	0.633	0.935	0.755
DM7	symPoly	piece3	10.000	10.000	0.742	0.612	0.671	0.429	0.632	0.511	0.608	0.497	0.547	783.000	0.400	0.715	0.458	0.627	0.529
DM7	symPoly	piece4	5	4	0.555	0.311	0.398	0.000	0.000	0.000	0.457	0.247	0.321	14.000	0.311	0.457	0.370	0.486	0.420
DM7	symPoly	piece5	13.000	33.000	0.656	0.401	0.498	0.794	0.934	0.859	0.656	0.405	0.501	35325.000	0.154	0.533	0.757	0.894	0.820
DM8	symPoly	piece1	5	37.000	0.457	0.733	0.563	0.479	0.454	0.466	0.299	0.514	0.378	55.000	0.535	0.648	0.348	0.639	0.451
DM8	symPoly	piece2	5	67.000	0.379	0.749	0.503	0.512	0.851	0.640	0.326	0.591	0.420	1319.000	0.223	0.211	0.401	0.826	0.540
DM8	symPoly	piece3	10.000	20.000	0.425	0.488	0.454	0.547	0.834	0.661	0.385	0.417	0.401	77.000	0.324	0.337	0.475	0.693	0.563
DM8	symPoly	piece4	5	21.000	0.399	0.636	0.491	0.358	0.632	0.457	0.276	0.370	0.316	3.000	0.426	0.307	0.265	0.431	0.328
DM8	symPoly	piece5	13.000	69.000	0.463	0.461	0.462	0.648	0.838	0.731	0.417	0.416	0.417	3002.000	0.208	0.525	0.567	0.819	0.670
DM9	symPoly	piece1	5	37.000	0.240	0.375	0.293	0.000	0.000	0.000	0.197	0.385	0.261	49.000	0.316	0.436	0.500	0.307	0.381
DM9	symPoly	piece2	5	67.000	0.149	0.408	0.218	0.656	0.492	0.562	0.078	0.314	0.124	1335.000	0.199	0.129	0.656	0.329	0.438
DM9	symPoly	piece3	10.000	20.000	0.177	0.225	0.198	0.000	0.000	0.000	0.125	0.201	0.154	91.000	0.152	0.146	0.406	0.317	0.356
DM9	symPoly	piece4	5	21.000	0.321	0.446	0.373	0.444	0.375	0.407	0.180	0.256	0.212	2.000	0.293	0.284	0.243	0.356	0.289
DM9	symPoly	piece5	13.000	69.000	0.138	0.257	0.179	0.000	0.000	0.000	0.095	0.293	0.143	2961.000	0.162	0.234	0.000	0.000	0.000
DM10	symPoly	piece1	5	37.000	0.395	0.535	0.454	0.406	0.439	0.422	0.281	0.422	0.337	53.000	0.498	0.485	0.384	0.515	0.440
DM10	symPoly	piece2	5	67.000	0.621	0.785	0.693	0.556	0.948	0.701	0.531	0.609	0.568	1287.000	0.313	0.313	0.512	0.917	0.657
DM10	symPoly	piece3	10.000	20.000	0.670	0.557	0.608	0.592	0.831	0.691	0.641	0.474	0.545	89.000	0.330	0.751	0.509	0.782	0.617
DM10	symPoly	piece4	5	21.000	0.503	0.508	0.505	0.472	0.941	0.628	0.368	0.326	0.346	3.000	0.415	0.556	0.306	0.726	0.430
DM10	symPoly	piece5	13.000	69.000	0.678	0.530	0.595	0.643	0.897	0.749	0.631	0.448	0.524	3108.000	0.214	0.652	0.565	0.887	0.690

download these results as csv Table 2. Caption to go here.

2013:Discovery of Repeated Themes & Sections Results

Contents

Introduction

Ground Truth and Algorithms

Results

In Brief

symPoly

symMono

audPoly

audMono

Navigation menu

Views

Personal tools

MIREX by Year

Results by Year

Account Request

Search

Navigation

Tools