2013:Discovery of Repeated Themes & Sections Results
This task report is currently under construction. Plots (png files) are missing currently due to a technical problem. Please check back soon for these.
Contents
Introduction
The task: algorithms take a piece of music as input, and output a list of patterns repeated within that piece. A pattern is defined as a set of ontime-pitch pairs that occurs twice (i.e., is repeated at least once) in a piece of music. The second, third, etc. occurrences of the pattern will likely be shifted in time and/or transposed, relative to the first occurrence. Ideally an algorithm will be able to discover all exact and inexact occurrences of a pattern within a piece, so in evaluating this task we are interested in both
- (1) whether an algorithm can discover one occurrence, up to time shift and transposition, and
- (2) to what extent it can find all occurrences.
The metrics establishment recall, establishment precision and establishment F1 address (1), and the metrics occurrence recall, occurrence precision, and occurrence F1 address (2).
Existing approaches to music structure analysis in MIR tend to focus on segmentation. The contribution of this task is to afford access to the note content itself; not just requiring algorithms to label time windows. The ground truth also contains nested patterns, reflecting the often hierarchical nature of musical repetition.
Ground Truth and Algorithms
The ground truth, called the Johannes Kepler University Patterns Test Database (JKUPTD-Aug2013), is based on motifs and themes in Barlow and Morgenstern (1953), Schoenberg (1967), and Bruhn (1993). Repeated sections are based on those marked by the composer. These annotations are supplemented with some of our own where necessary. A Development Database (JKUPDD-Aug2013) released in March enabled participants to try out their algorithms. For each piece in the Development and Test Databases, symbolic and synthesised audio versions are crossed with monophonic and polyphonic versions, giving up to four versions of the task in total: symPoly, symMono, audPoly, and audMono. Algorithms submitted to the task are are shown in Table 1.
Results
In Brief
To avoid a bias toward the more numerous submissions of Meredith (2013), DM10 was preselected for comparison with Nieto and Farbood (2013), based on reported performance for the Development Database. Figure 2 shows establishment recall results for the symbolic-polyphonic version of the task, on a per-pattern basis. DM10 outperforms NF2 according to Friedman's test (chi2(1) = 12.7368, p < .001), suggesting the former is superior at discovering at least one occurrence of each ground truth pattern.
symPoly
Figure 2. Test figure. Proper figures to follow.
| AlgId | TaskVersion | Piece | n_P | n_Q | P_est | R_est | F1_est | P_occ(c=.75) | R_occ(c=.75) | F_1occ(c=.75) | P_3 | R_3 | TLF_1 | runtime | FRT | FFTP_est | FFP | P_occ(c=.5) | R_occ(c=.5) | F_1occ(c=.5) | P | R | F_1 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NF2 | symPoly | piece1 | 5 | 5 | 0.240 | 0.222 | 0.231 | 0.000 | 0.000 | 0.000 | 0.143 | 0.142 | 0.142 | 15.000 | 0.000 | 0.222 | 0.143 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| NF2 | symPoly | piece2 | 5 | 27.000 | 0.277 | 0.446 | 0.342 | 0.793 | 0.793 | 0.793 | 0.078 | 0.253 | 0.120 | 1221.000 | 0.000 | 0.399 | 0.235 | 0.409 | 0.411 | 0.410 | 0.000 | 0.000 | 0.000 |
| NF2 | symPoly | piece3 | 10.000 | 20.000 | 0.695 | 0.584 | 0.635 | 0.703 | 0.320 | 0.440 | 0.473 | 0.439 | 0.455 | 34.000 | 0.000 | 0.355 | 0.473 | 0.603 | 0.373 | 0.461 | 0.000 | 0.000 | 0.000 |
| NF2 | symPoly | piece4 | 5 | 2 | 0.667 | 0.272 | 0.386 | 0.885 | 0.885 | 0.885 | 0.609 | 0.240 | 0.344 | 3.000 | 0.000 | 0.272 | 0.609 | 0.885 | 0.885 | 0.885 | 0.000 | 0.000 | 0.000 |
| NF2 | symPoly | piece5 | 13.000 | 18.000 | 0.564 | 0.334 | 0.419 | 0.690 | 0.393 | 0.501 | 0.417 | 0.345 | 0.377 | 153.000 | 0.000 | 0.245 | 0.549 | 0.601 | 0.488 | 0.539 | 0.000 | 0.000 | 0.000 |
| DM5 | symPoly | piece1 | 5 | 23.000 | 0.332 | 0.545 | 0.412 | 0.743 | 0.804 | 0.772 | 0.228 | 0.441 | 0.301 | 474.000 | 0.000 | 0.463 | 0.531 | 0.460 | 0.593 | 0.518 | 0.000 | 0.000 | 0.000 |
| DM5 | symPoly | piece2 | 5 | 38.000 | 0.287 | 0.447 | 0.349 | 0.385 | 0.770 | 0.513 | 0.235 | 0.339 | 0.278 | 19896.000 | 0.000 | 0.274 | 0.238 | 0.328 | 0.770 | 0.460 | 0.000 | 0.000 | 0.000 |
| DM5 | symPoly | piece3 | 10.000 | 10.000 | 0.330 | 0.423 | 0.371 | 0.000 | 0.000 | 0.000 | 0.285 | 0.365 | 0.320 | 762.000 | 0.000 | 0.288 | 0.370 | 0.318 | 0.471 | 0.379 | 0.000 | 0.000 | 0.000 |
| DM5 | symPoly | piece4 | 5 | 4 | 0.400 | 0.349 | 0.372 | 0.000 | 0.000 | 0.000 | 0.243 | 0.187 | 0.211 | 14.000 | 0.000 | 0.349 | 0.243 | 0.238 | 0.195 | 0.214 | 0.000 | 0.000 | 0.000 |
| DM5 | symPoly | piece5 | 13.000 | 33.000 | 0.305 | 0.300 | 0.303 | 0.693 | 0.805 | 0.745 | 0.301 | 0.328 | 0.314 | 36299.000 | 0.000 | 0.160 | 0.328 | 0.620 | 0.765 | 0.685 | 0.000 | 0.000 | 0.000 |
| DM6 | symPoly | piece1 | 5 | 23.000 | 0.215 | 0.381 | 0.275 | 0.000 | 0.000 | 0.000 | 0.163 | 0.346 | 0.221 | 500.000 | 0.000 | 0.315 | 0.365 | 0.454 | 0.207 | 0.284 | 0.000 | 0.000 | 0.000 |
| DM6 | symPoly | piece2 | 5 | 38.000 | 0.090 | 0.283 | 0.136 | 0.000 | 0.000 | 0.000 | 0.046 | 0.175 | 0.073 | 23294.000 | 0.000 | 0.207 | 0.102 | 0.100 | 0.033 | 0.050 | 0.000 | 0.000 | 0.000 |
| DM6 | symPoly | piece3 | 10.000 | 10.000 | 0.145 | 0.164 | 0.154 | 0.000 | 0.000 | 0.000 | 0.152 | 0.204 | 0.174 | 771.000 | 0.000 | 0.143 | 0.209 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| DM6 | symPoly | piece4 | 5 | 4 | 0.321 | 0.234 | 0.271 | 0.000 | 0.000 | 0.000 | 0.165 | 0.127 | 0.143 | 13.000 | 0.000 | 0.234 | 0.165 | 0.250 | 0.125 | 0.167 | 0.000 | 0.000 | 0.000 |
| DM6 | symPoly | piece5 | 13.000 | 33.000 | 0.119 | 0.189 | 0.146 | 0.000 | 0.000 | 0.000 | 0.084 | 0.193 | 0.117 | 37646.000 | 0.000 | 0.135 | 0.175 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| DM7 | symPoly | piece1 | 5 | 23.000 | 0.316 | 0.522 | 0.393 | 0.406 | 0.439 | 0.422 | 0.203 | 0.377 | 0.264 | 532.000 | 0.000 | 0.467 | 0.409 | 0.405 | 0.447 | 0.425 | 0.000 | 0.000 | 0.000 |
| DM7 | symPoly | piece2 | 5 | 38.000 | 0.673 | 0.614 | 0.642 | 0.686 | 0.965 | 0.802 | 0.598 | 0.465 | 0.523 | 19926.000 | 0.000 | 0.368 | 0.562 | 0.633 | 0.935 | 0.755 | 0.000 | 0.000 | 0.000 |
| DM7 | symPoly | piece3 | 10.000 | 10.000 | 0.742 | 0.612 | 0.671 | 0.429 | 0.632 | 0.511 | 0.608 | 0.497 | 0.547 | 783.000 | 0.000 | 0.400 | 0.715 | 0.458 | 0.627 | 0.529 | 0.000 | 0.000 | 0.000 |
| DM7 | symPoly | piece4 | 5 | 4 | 0.555 | 0.311 | 0.398 | 0.000 | 0.000 | 0.000 | 0.457 | 0.247 | 0.321 | 14.000 | 0.000 | 0.311 | 0.457 | 0.370 | 0.486 | 0.420 | 0.000 | 0.000 | 0.000 |
| DM7 | symPoly | piece5 | 13.000 | 33.000 | 0.656 | 0.401 | 0.498 | 0.794 | 0.934 | 0.859 | 0.656 | 0.405 | 0.501 | 35325.000 | 0.000 | 0.154 | 0.533 | 0.757 | 0.894 | 0.820 | 0.000 | 0.000 | 0.000 |
| DM8 | symPoly | piece1 | 5 | 37.000 | 0.457 | 0.733 | 0.563 | 0.479 | 0.454 | 0.466 | 0.299 | 0.514 | 0.378 | 55.000 | 0.000 | 0.535 | 0.648 | 0.348 | 0.639 | 0.451 | 0.000 | 0.000 | 0.000 |
| DM8 | symPoly | piece2 | 5 | 67.000 | 0.379 | 0.749 | 0.503 | 0.512 | 0.851 | 0.640 | 0.326 | 0.591 | 0.420 | 1319.000 | 0.000 | 0.223 | 0.211 | 0.401 | 0.826 | 0.540 | 0.000 | 0.000 | 0.000 |
| DM8 | symPoly | piece3 | 10.000 | 20.000 | 0.425 | 0.488 | 0.454 | 0.547 | 0.834 | 0.661 | 0.385 | 0.417 | 0.401 | 77.000 | 0.000 | 0.324 | 0.337 | 0.475 | 0.693 | 0.563 | 0.000 | 0.000 | 0.000 |
| DM8 | symPoly | piece4 | 5 | 21.000 | 0.399 | 0.636 | 0.491 | 0.358 | 0.632 | 0.457 | 0.276 | 0.370 | 0.316 | 3.000 | 0.000 | 0.426 | 0.307 | 0.265 | 0.431 | 0.328 | 0.000 | 0.000 | 0.000 |
| DM8 | symPoly | piece5 | 13.000 | 69.000 | 0.463 | 0.461 | 0.462 | 0.648 | 0.838 | 0.731 | 0.417 | 0.416 | 0.417 | 3002.000 | 0.000 | 0.208 | 0.525 | 0.567 | 0.819 | 0.670 | 0.000 | 0.000 | 0.000 |
| DM9 | symPoly | piece1 | 5 | 37.000 | 0.240 | 0.375 | 0.293 | 0.000 | 0.000 | 0.000 | 0.197 | 0.385 | 0.261 | 49.000 | 0.000 | 0.316 | 0.436 | 0.500 | 0.307 | 0.381 | 0.000 | 0.000 | 0.000 |
| DM9 | symPoly | piece2 | 5 | 67.000 | 0.149 | 0.408 | 0.218 | 0.656 | 0.492 | 0.562 | 0.078 | 0.314 | 0.124 | 1335.000 | 0.000 | 0.199 | 0.129 | 0.656 | 0.329 | 0.438 | 0.000 | 0.000 | 0.000 |
| DM9 | symPoly | piece3 | 10.000 | 20.000 | 0.177 | 0.225 | 0.198 | 0.000 | 0.000 | 0.000 | 0.125 | 0.201 | 0.154 | 91.000 | 0.000 | 0.152 | 0.146 | 0.406 | 0.317 | 0.356 | 0.000 | 0.000 | 0.000 |
| DM9 | symPoly | piece4 | 5 | 21.000 | 0.321 | 0.446 | 0.373 | 0.444 | 0.375 | 0.407 | 0.180 | 0.256 | 0.212 | 2.000 | 0.000 | 0.293 | 0.284 | 0.243 | 0.356 | 0.289 | 0.000 | 0.000 | 0.000 |
| DM9 | symPoly | piece5 | 13.000 | 69.000 | 0.138 | 0.257 | 0.179 | 0.000 | 0.000 | 0.000 | 0.095 | 0.293 | 0.143 | 2961.000 | 0.000 | 0.162 | 0.234 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| DM10 | symPoly | piece1 | 5 | 37.000 | 0.395 | 0.535 | 0.454 | 0.406 | 0.439 | 0.422 | 0.281 | 0.422 | 0.337 | 53.000 | 0.000 | 0.498 | 0.485 | 0.384 | 0.515 | 0.440 | 0.000 | 0.000 | 0.000 |
| DM10 | symPoly | piece2 | 5 | 67.000 | 0.621 | 0.785 | 0.693 | 0.556 | 0.948 | 0.701 | 0.531 | 0.609 | 0.568 | 1287.000 | 0.000 | 0.313 | 0.313 | 0.512 | 0.917 | 0.657 | 0.000 | 0.000 | 0.000 |
| DM10 | symPoly | piece3 | 10.000 | 20.000 | 0.670 | 0.557 | 0.608 | 0.592 | 0.831 | 0.691 | 0.641 | 0.474 | 0.545 | 89.000 | 0.000 | 0.330 | 0.751 | 0.509 | 0.782 | 0.617 | 0.000 | 0.000 | 0.000 |
| DM10 | symPoly | piece4 | 5 | 21.000 | 0.503 | 0.508 | 0.505 | 0.472 | 0.941 | 0.628 | 0.368 | 0.326 | 0.346 | 3.000 | 0.000 | 0.415 | 0.556 | 0.306 | 0.726 | 0.430 | 0.000 | 0.000 | 0.000 |
| DM10 | symPoly | piece5 | 13.000 | 69.000 | 0.678 | 0.530 | 0.595 | 0.643 | 0.897 | 0.749 | 0.631 | 0.448 | 0.524 | 3108.000 | 0.000 | 0.214 | 0.652 | 0.565 | 0.887 | 0.690 | 0.000 | 0.000 | 0.000 |
download these results as csv Table 2. Caption to go here.
