MIREX Wiki - User contributions [en]

2020:Patterns for Prediction Results

2020-10-20T14:37:50Z

Berit Janssen: /* Discussion */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono implicit task are listed in Table 1. There were no submissions to the symPoly, audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. As in previous years, this served as a foil for the implicit subtask. As one of last year's submissions (GenDetect (EP1)) already reached almost-perfect accuracy on these foils, we also evaluate how well the submitted models can discriminate the true continuation from a foil generated by last year's best-performing music prediction model, CopyForward (see [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf abstract]).

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH16
| PDMTransformer || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || Vane Wu, YuanLiang Dong, Yu Hong (Tencent Music)

|}

'''Table 1. Algorithms submitted to the symMono task of Patterns for Prediction 2019 / 2020.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2020:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>Observations</th>
<th>Accuracy</th>
<th>Mean Probability</th>
<th>Variance of Probability</th>
</tr>
<tr>
<th>Model</th>
<th>Foil</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>Markov</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH16</th>
<th>Markov</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH16</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==
This year's submission, PDMTransformer, is based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features. It has higher accuracy for the distinction of true and foil continuations than GenDetect, last year's highest accuracy submission. It outputs similar probabilities for the CopyForward foils and true continuations (scoring, e.g., 0.96 probability for the foils and 0.97 to the true continuation), which explains the lower mean probability (resulting from normalization of the two probability scores) of PDMTransformer as compared to GenDetect.

2020:Patterns for Prediction Results

2020-10-20T14:33:05Z

Berit Janssen: /* Discussion */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono implicit task are listed in Table 1. There were no submissions to the symPoly, audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. As in previous years, this served as a foil for the implicit subtask. As one of last year's submissions (GenDetect (EP1)) already reached almost-perfect accuracy on these foils, we also evaluate how well the submitted models can discriminate the true continuation from a foil generated by last year's best-performing music prediction model, CopyForward (see [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf abstract]).

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH16
| PDMTransformer || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || Vane Wu, YuanLiang Dong, Yu Hong (Tencent Music)

|}

'''Table 1. Algorithms submitted to the symMono task of Patterns for Prediction 2019 / 2020.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2020:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>Observations</th>
<th>Accuracy</th>
<th>Mean Probability</th>
<th>Variance of Probability</th>
</tr>
<tr>
<th>Model</th>
<th>Foil</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>Markov</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH16</th>
<th>Markov</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH16</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==
This year's submission, PDMTransform, is based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features. It has higher accuracy for the distinction of true and foil continuations than GenDetect, last year's highest accuracy submission. It outputs similar probabilities for the CopyForward foils and true continuations (scoring, e.g., 0.96 probability for the foils and 0.97 to the true continuation), which explains the lower mean probability (resulting from normalization of the two probability scores) of PDMTransform as compared to GenDetect.

2020:Patterns for Prediction Results

2020-10-13T08:09:56Z

Berit Janssen: /* Implicit task: discriminate true and foil continuation */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono implicit task are listed in Table 1. There were no submissions to the symPoly, audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. As in previous years, this served as a foil for the implicit subtask. As one of last year's submissions (GenDetect (EP1)) already reached almost-perfect accuracy on these foils, we also evaluate how well the submitted models can discriminate the true continuation from a foil generated by last year's best-performing music prediction model, CopyForward (see [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf abstract]).

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH16
| PDMTransformer || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || Vane Wu, YuanLiang Dong, Yu Hong (Tencent Music)

|}

'''Table 1. Algorithms submitted to the symMono task of Patterns for Prediction 2019 / 2020.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2020:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>Observations</th>
<th>Accuracy</th>
<th>Mean Probability</th>
<th>Variance of Probability</th>
</tr>
<tr>
<th>Model</th>
<th>Foil</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>Markov</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH16</th>
<th>Markov</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH16</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==
This year's submission, based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features, has higher accuracy for the distinction of true and foil continuations. However, the confidence of the model to prefer one continuation over another (mean probability) is rather low.

2020:Patterns for Prediction Results

2020-10-13T08:09:37Z

Berit Janssen: /* Datasets and Algorithms */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono implicit task are listed in Table 1. There were no submissions to the symPoly, audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. As in previous years, this served as a foil for the implicit subtask. As one of last year's submissions (GenDetect (EP1)) already reached almost-perfect accuracy on these foils, we also evaluate how well the submitted models can discriminate the true continuation from a foil generated by last year's best-performing music prediction model, CopyForward (see [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf abstract]).

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH16
| PDMTransformer || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || Vane Wu, YuanLiang Dong, Yu Hong (Tencent Music)

|}

'''Table 1. Algorithms submitted to the symMono task of Patterns for Prediction 2019 / 2020.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2020:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>Observations</th>
<th>Accuracy</th>
<th>Mean Probability</th>
<th>Variance of Probability</th>
</tr>
<tr>
<th>Model</th>
<th>Foil</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>Markov</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH1</th>
<th>Markov</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==
This year's submission, based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features, has higher accuracy for the distinction of true and foil continuations. However, the confidence of the model to prefer one continuation over another (mean probability) is rather low.

2020:MIREX2020 Results

2020-10-09T12:03:03Z

Berit Janssen: /* Results by Task (More results are coming) */

==Results by Task (More results are coming) ==
* [[2020:Audio Fingerprinting Results|Audio Fingerprinting Results]]
* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2020/results/ame/adc04/ ADC04 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2020/results/ame/mrx05/ MIREX05 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2020/results/ame/ind08/ INDIAN08 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2020/results/ame/mrx09_0db/ MIREX09 0dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2020/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2020/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2020/results/ame/orchset/ ORCHSET15 Dataset]  
* [https://www.music-ir.org/mirex/wiki/2019:Patterns_for_Prediction_Results Patterns for Prediction Results]  

2020:Patterns for Prediction Results

2020-10-09T12:02:20Z

Berit Janssen: /* Contribution */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono implicit task are listed in Table 1. There were no submissions to the symPoly, audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. As in previous years, this served as a foil for the implicit subtask. As one of last year's submissions (GenDetect (EP1)) already reached almost-perfect accuracy on these foils, we also evaluate how well the submitted models can discriminate the true continuation from a foil generated by last year's best-performing music prediction model, CopyForward (see [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf abstract]).

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH16
| XGBoost || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || Vane Wu, YuanLiang Dong, Yu Hong (Tencent Music)

|}

'''Table 1. Algorithms submitted to the symMono task of Patterns for Prediction 2019 / 2020.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2020:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>Observations</th>
<th>Accuracy</th>
<th>Mean Probability</th>
<th>Variance of Probability</th>
</tr>
<tr>
<th>Model</th>
<th>Foil</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>Markov</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH1</th>
<th>Markov</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==
This year's submission, based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features, has higher accuracy for the distinction of true and foil continuations. However, the confidence of the model to prefer one continuation over another (mean probability) is rather low.

2020:Patterns for Prediction Results

2020-10-09T12:00:26Z

Berit Janssen: /* Discussion */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono implicit task are listed in Table 1. There were no submissions to the symPoly, audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. As in previous years, this served as a foil for the implicit subtask. As one of last year's submissions (GenDetect (EP1)) already reached almost-perfect accuracy on these foils, we also evaluate how well the submitted models can discriminate the true continuation from a foil generated by last year's best-performing music prediction model, CopyForward (see [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf abstract]).

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH16
| XGBoost || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || Vane Wu, YuanLiang Dong, Yu Hong (Tencent Music)

|}

'''Table 1. Algorithms submitted to the symMono task of Patterns for Prediction 2019 / 2020.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2020:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>Observations</th>
<th>Accuracy</th>
<th>Mean Probability</th>
<th>Variance of Probability</th>
</tr>
<tr>
<th>Model</th>
<th>Foil</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>Markov</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH1</th>
<th>Markov</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==
This year's submission, based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features, has higher accuracy for the distinction of true and foil continuations. However, the confidence of the model to prefer one continuation over another (mean probability) is rather low.

2020:Patterns for Prediction Results

2020-10-09T11:59:56Z

Berit Janssen: /* Implicit task: discriminate true and foil continuation */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono implicit task are listed in Table 1. There were no submissions to the symPoly, audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. As in previous years, this served as a foil for the implicit subtask. As one of last year's submissions (GenDetect (EP1)) already reached almost-perfect accuracy on these foils, we also evaluate how well the submitted models can discriminate the true continuation from a foil generated by last year's best-performing music prediction model, CopyForward (see [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf abstract]).

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH16
| XGBoost || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || Vane Wu, YuanLiang Dong, Yu Hong (Tencent Music)

|}

'''Table 1. Algorithms submitted to the symMono task of Patterns for Prediction 2019 / 2020.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2020:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>Observations</th>
<th>Accuracy</th>
<th>Mean Probability</th>
<th>Variance of Probability</th>
</tr>
<tr>
<th>Model</th>
<th>Foil</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>Markov</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH1</th>
<th>Markov</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==
This year's submission, based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features, has higher accuracy for the distinction of true and foil continuations. However, the confidence of the model to give one choice over another (mean probability) is rather low.

2020:Patterns for Prediction Results

2020-10-09T11:59:04Z

Berit Janssen: /* Discussion */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono implicit task are listed in Table 1. There were no submissions to the symPoly, audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. As in previous years, this served as a foil for the implicit subtask. As one of last year's submissions (GenDetect (EP1)) already reached almost-perfect accuracy on these foils, we also evaluate how well the submitted models can discriminate the true continuation from a foil generated by last year's best-performing music prediction model, CopyForward (see [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf abstract]).

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH16
| XGBoost || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || Vane Wu, YuanLiang Dong, Yu Hong (Tencent Music)

|}

'''Table 1. Algorithms submitted to the symMono task of Patterns for Prediction 2019 / 2020.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2020:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>Observations</th>
<th>Accuracy</th>
<th>Mean Probability</th>
<th>Variance</th>
</tr>
<tr>
<th>Model</th>
<th>Foil</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>Markov</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH1</th>
<th>Markov</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==
This year's submission, based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features, has higher accuracy for the distinction of true and foil continuations. However, the confidence of the model to give one choice over another (mean probability) is rather low.

2020:Patterns for Prediction Results

2020-10-09T11:53:03Z

Berit Janssen: /* Implicit task: discriminate true and foil continuation */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono implicit task are listed in Table 1. There were no submissions to the symPoly, audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. As in previous years, this served as a foil for the implicit subtask. As one of last year's submissions (GenDetect (EP1)) already reached almost-perfect accuracy on these foils, we also evaluate how well the submitted models can discriminate the true continuation from a foil generated by last year's best-performing music prediction model, CopyForward (see [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf abstract]).

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH16
| XGBoost || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || Vane Wu, YuanLiang Dong, Yu Hong (Tencent Music)

|}

'''Table 1. Algorithms submitted to the symMono task of Patterns for Prediction 2019 / 2020.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2020:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>Observations</th>
<th>Accuracy</th>
<th>Mean Probability</th>
<th>Variance</th>
</tr>
<tr>
<th>Model</th>
<th>Foil</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>Markov</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH1</th>
<th>Markov</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH1</th>
<th>CopyForward</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==

Our discussion

2020:Patterns for Prediction Results

2020-10-09T11:48:33Z

Berit Janssen: /* Datasets and Algorithms */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono implicit task are listed in Table 1. There were no submissions to the symPoly, audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. As in previous years, this served as a foil for the implicit subtask. As one of last year's submissions (GenDetect (EP1)) already reached almost-perfect accuracy on these foils, we also evaluate how well the submitted models can discriminate the true continuation from a foil generated by last year's best-performing music prediction model, CopyForward (see [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf abstract]).

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH16
| XGBoost || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || Vane Wu, YuanLiang Dong, Yu Hong (Tencent Music)

|}

'''Table 1. Algorithms submitted to the symMono task of Patterns for Prediction 2019 / 2020.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2020:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>mean_probability</th>
<th>var_prob</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<th>EP1_Markov</th>
<th>mono</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1_CopyForward</th>
<th>mono</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH1_Markov</th>
<th>mono</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH1_CopyForward</th>
<th>mono</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</tbody>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==

Our discussion

2020:Patterns for Prediction Results

2020-10-09T11:43:15Z

Berit Janssen: /* Datasets and Algorithms */

2020:Patterns for Prediction Results

2020-10-09T11:42:15Z

Berit Janssen: /* Datasets and Algorithms */

2020:Patterns for Prediction Results

2020-10-09T11:40:49Z

Berit Janssen: /* Datasets and Algorithms */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH16
| XGBoost || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || Vane Wu, YuanLiang Dong, Yu Hong (Tencent Music)

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2020:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>mean_probability</th>
<th>var_prob</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<th>EP1_Markov</th>
<th>mono</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1_CopyForward</th>
<th>mono</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH1_Markov</th>
<th>mono</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH1_CopyForward</th>
<th>mono</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</tbody>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==

Our discussion

2020:Patterns for Prediction Results

2020-10-07T19:55:31Z

Berit Janssen: /* Datasets and Algorithms */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH16
| XGBoost || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || [Vane Wu]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2020:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>mean_probability</th>
<th>var_prob</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<th>EP1_Markov</th>
<th>mono</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1_CopyForward</th>
<th>mono</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH1_Markov</th>
<th>mono</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH1_CopyForward</th>
<th>mono</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</tbody>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==

Our discussion

2020:Patterns for Prediction Results

2020-10-07T15:37:29Z

Berit Janssen:

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH1
| XGBoost || style="text-align: center;" | || [Vane Wu]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2020:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>mean_probability</th>
<th>var_prob</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<th>EP1_Markov</th>
<th>mono</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1_CopyForward</th>
<th>mono</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH1_Markov</th>
<th>mono</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH1_CopyForward</th>
<th>mono</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</tbody>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==

Our discussion

2020:Patterns for Prediction Results

2020-10-07T15:36:45Z

Berit Janssen:

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH1
| XGBoost || style="text-align: center;" | || [Vane Wu]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2012:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2020:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>mean_probability</th>
<th>var_prob</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<th>EP1_Markov</th>
<th>mono</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1_CopyForward</th>
<th>mono</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH1_Markov</th>
<th>mono</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH1_CopyForward</th>
<th>mono</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</tbody>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==

Our discussion

2020:Patterns for Prediction Results

2020-10-07T15:36:13Z

Berit Janssen: Created page with "== Introduction == '''In brief''': We look for (1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt. (2)..."

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YH1
| XGBoost || style="text-align: center;" | || [Vane Wu]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2012:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2019:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2019] (GenDetect).

===symMono===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>mean_probability</th>
<th>var_prob</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<th>EP1_Markov</th>
<th>mono</th>
<td>500.0</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1_CopyForward</th>
<th>mono</th>
<td>500.0</td>
<td>0.774</td>
<td>0.738</td>
<td>0.132</td>
</tr>
<tr>
<th>YH1_Markov</th>
<th>mono</th>
<td>500.0</td>
<td>0.996</td>
<td>0.961</td>
<td>0.006</td>
</tr>
<tr>
<th>YH1_CopyForward</th>
<th>mono</th>
<td>500.0</td>
<td>0.916</td>
<td>0.507</td>
<td>0.005</td>
</tr>
</tbody>
</table>

'''Table 1.''' Discrimination scores of the submitted algorithms.

== Discussion ==

Our discussion

2019:Patterns for Prediction Results

2019-11-02T16:50:47Z

Berit Janssen: /* Implicit task: discriminate true and foil continuation */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2019:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf PDF] || [https://mcgill.ca/music/timothy-de-reuse Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf PDF] || [https://mcgill.ca/music/timothy-de-reuse Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YB2
| MLM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart], [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]
|-
! YB5
| MLM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart], [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2018:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2018] (BachProp and Seq2SeqP4P).

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProp</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProp</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 3.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>Observations</th>
<th>Accuracy</th>
<th>Mean Probability</th>
<th>Variance</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>GenDetect</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>1.000</td>
<td>0.000</td>
</tr>
<tr>
<th>BachProp</th>
<th>mono</th>
<td>499</td>
<td>0.844</td>
<td>0.498</td>
<td>0.006</td>
</tr>
<tr>
<th>GenDetect</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.992</td>
<td>0.002</td>
</tr>
<tr>
<th>BachProp</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.519</td>
<td>0.006</td>
</tr>
<tr>
<th>MLM(2)</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.527</td>
<td>0.004</td>
</tr>
<tr>
<th>MLM(5)</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.554</td>
<td>0.012</td>
</tr>
</table>

'''Table 4.''' Discrimination scores of the submitted algorithms.

2019:MIREX2019 Results

2019-10-31T20:22:17Z

Berit Janssen: /* Results by Task (More results are coming) */

==Overall Results Poster==
Coming soon

==Results by Task (More results are coming) ==
* Multiple Fundamental Frequency Estimation & Tracking Results
** [[2019:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_%2D_MIREX_Dataset | MIREX Dataset]]  
** [[2019:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_%2D_Su_Dataset | Su Dataset]]  
* [https://www.music-ir.org/mirex/wiki/2019:Music_Detection_Results Music Detection Results]  
* [https://www.music-ir.org/mirex/wiki/2019:Patterns_for_Prediction_Results Patterns for Prediction Results]  

2019:Patterns for Prediction Results

2019-10-31T20:21:08Z

Berit Janssen: /* Datasets and Algorithms */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2019:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf PDF] || [https://mcgill.ca/music/timothy-de-reuse Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf PDF] || [https://mcgill.ca/music/timothy-de-reuse Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YB2
| MLM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart], [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]
|-
! YB5
| MLM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart], [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2018:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2018] (BachProp and Seq2SeqP4P).

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProp</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProp</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 3.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>crossentropy</th>
<th>nll_var</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>GenDetect</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>0.313</td>
<td>0.000</td>
</tr>
<tr>
<th>BachProp</th>
<th>mono</th>
<td>499</td>
<td>0.864</td>
<td>0.693</td>
<td>0.000</td>
</tr>
<tr>
<th>GenDetect</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.318</td>
<td>0.001</td>
</tr>
<tr>
<th>BachProp</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.682</td>
<td>0.001</td>
</tr>
<tr>
<th>MLM</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.668</td>
<td>0.003</td>
</tr>
<tr>
<th>MLM</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.646</td>
<td>0.010</td>
</tr>
</table>

'''Table 4.''' Discrimination scores of the submitted algorithms.

2019:Patterns for Prediction Results

2019-10-31T20:20:38Z

Berit Janssen: /* Datasets and Algorithms */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2019:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf PDF] || [https://mcgill.ca/music/timothy-de-reuse Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf PDF] || [https://mcgill.ca/music/timothy-de-reuse Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
|-
! YB2
| MLM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart] [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]
|-
! YB5
| MLM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart] [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2018:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2018] (BachProp and Seq2SeqP4P).

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProp</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProp</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 3.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>crossentropy</th>
<th>nll_var</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>GenDetect</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>0.313</td>
<td>0.000</td>
</tr>
<tr>
<th>BachProp</th>
<th>mono</th>
<td>499</td>
<td>0.864</td>
<td>0.693</td>
<td>0.000</td>
</tr>
<tr>
<th>GenDetect</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.318</td>
<td>0.001</td>
</tr>
<tr>
<th>BachProp</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.682</td>
<td>0.001</td>
</tr>
<tr>
<th>MLM</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.668</td>
<td>0.003</td>
</tr>
<tr>
<th>MLM</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.646</td>
<td>0.010</td>
</tr>
</table>

'''Table 4.''' Discrimination scores of the submitted algorithms.

2019:Patterns for Prediction Results

2019-10-31T20:16:27Z

Berit Janssen: /* Datasets and Algorithms */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2019:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf PDF] || [https://mcgill.ca/music/timothy-de-reuse Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf PDF] || [https://mcgill.ca/music/timothy-de-reuse Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| MLM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart] [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]
|-
! YB5
| MLM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart] [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2018:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2018] (BachProp and Seq2SeqP4P).

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProp</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProp</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 3.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>crossentropy</th>
<th>nll_var</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>GenDetect</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>0.313</td>
<td>0.000</td>
</tr>
<tr>
<th>BachProp</th>
<th>mono</th>
<td>499</td>
<td>0.864</td>
<td>0.693</td>
<td>0.000</td>
</tr>
<tr>
<th>GenDetect</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.318</td>
<td>0.001</td>
</tr>
<tr>
<th>BachProp</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.682</td>
<td>0.001</td>
</tr>
<tr>
<th>MLM</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.668</td>
<td>0.003</td>
</tr>
<tr>
<th>MLM</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.646</td>
<td>0.010</td>
</tr>
</table>

'''Table 4.''' Discrimination scores of the submitted algorithms.

2019:Patterns for Prediction Results

2019-10-31T20:09:11Z

Berit Janssen: /* Tables */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2019:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| MLM || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| MLM || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2018:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2018] (BachProp and Seq2SeqP4P).

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProp</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProp</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 3.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>crossentropy</th>
<th>nll_var</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>GenDetect</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>0.313</td>
<td>0.000</td>
</tr>
<tr>
<th>BachProp</th>
<th>mono</th>
<td>499</td>
<td>0.864</td>
<td>0.693</td>
<td>0.000</td>
</tr>
<tr>
<th>GenDetect</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.318</td>
<td>0.001</td>
</tr>
<tr>
<th>BachProp</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.682</td>
<td>0.001</td>
</tr>
<tr>
<th>MLM</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.668</td>
<td>0.003</td>
</tr>
<tr>
<th>MLM</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.646</td>
<td>0.010</td>
</tr>
</table>

'''Table 4.''' Discrimination scores of the submitted algorithms.

2019:Patterns for Prediction Results

2019-10-31T20:08:08Z

Berit Janssen: /* Explicit task: generate music given a prime */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2019:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| MLM || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| MLM || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2018:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2018] (BachProp and Seq2SeqP4P).

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProp</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 3.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>crossentropy</th>
<th>nll_var</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>0.313</td>
<td>0.000</td>
</tr>
<tr>
<th>FC1</th>
<th>mono</th>
<td>499</td>
<td>0.864</td>
<td>0.693</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.318</td>
<td>0.001</td>
</tr>
<tr>
<th>FC1</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.682</td>
<td>0.001</td>
</tr>
<tr>
<th>YB2</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.668</td>
<td>0.003</td>
</tr>
<tr>
<th>YB5</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.646</td>
<td>0.010</td>
</tr>
</table>

'''Table 4.''' Discrimination scores of the submitted algorithms.

2019:Patterns for Prediction Results

2019-10-29T18:28:58Z

Berit Janssen: /* Results */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2019:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| MLM || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| MLM || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the [https://www.music-ir.org/mirex/wiki/2018:Patterns_for_Prediction_Results#Datasets_and_Algorithms submissions from 2018] (BachProp and Seq2SeqP4P).

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.
===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 3.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>crossentropy</th>
<th>nll_var</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>0.313</td>
<td>0.000</td>
</tr>
<tr>
<th>FC1</th>
<th>mono</th>
<td>499</td>
<td>0.864</td>
<td>0.693</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.318</td>
<td>0.001</td>
</tr>
<tr>
<th>FC1</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.682</td>
<td>0.001</td>
</tr>
<tr>
<th>YB2</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.668</td>
<td>0.003</td>
</tr>
<tr>
<th>YB5</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.646</td>
<td>0.010</td>
</tr>
</table>

'''Table 4.''' Discrimination scores of the submitted algorithms.

2019:Patterns for Prediction Results

2019-10-29T18:27:08Z

Berit Janssen: /* Results */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2019:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| MLM || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| MLM || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the submissions from 2018 (BachProp and Seq2SeqP4P, see [https://www.music-ir.org/mirex/wiki/2018:Patterns_for_Prediction_Results]).

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.
===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 3.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>crossentropy</th>
<th>nll_var</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>0.313</td>
<td>0.000</td>
</tr>
<tr>
<th>FC1</th>
<th>mono</th>
<td>499</td>
<td>0.864</td>
<td>0.693</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.318</td>
<td>0.001</td>
</tr>
<tr>
<th>FC1</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.682</td>
<td>0.001</td>
</tr>
<tr>
<th>YB2</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.668</td>
<td>0.003</td>
</tr>
<tr>
<th>YB5</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.646</td>
<td>0.010</td>
</tr>
</table>

'''Table 4.''' Discrimination scores of the submitted algorithms.

2019:Patterns for Prediction Results

2019-10-29T18:26:16Z

Berit Janssen: /* Results */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2019:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| MLM || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| MLM || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

For reference purposes, we also include results of the submissions from 2018 (BachProp and Seq2SeqP4P).

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.
===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 3.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>crossentropy</th>
<th>nll_var</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>0.313</td>
<td>0.000</td>
</tr>
<tr>
<th>FC1</th>
<th>mono</th>
<td>499</td>
<td>0.864</td>
<td>0.693</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.318</td>
<td>0.001</td>
</tr>
<tr>
<th>FC1</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.682</td>
<td>0.001</td>
</tr>
<tr>
<th>YB2</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.668</td>
<td>0.003</td>
</tr>
<tr>
<th>YB5</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.646</td>
<td>0.010</td>
</tr>
</table>

'''Table 4.''' Discrimination scores of the submitted algorithms.

2019:Patterns for Prediction Results

2019-10-29T18:25:32Z

Berit Janssen: /* Datasets and Algorithms */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2019:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| GenDetect || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| MLM || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| MLM || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.
===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 3.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>crossentropy</th>
<th>nll_var</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>0.313</td>
<td>0.000</td>
</tr>
<tr>
<th>FC1</th>
<th>mono</th>
<td>499</td>
<td>0.864</td>
<td>0.693</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.318</td>
<td>0.001</td>
</tr>
<tr>
<th>FC1</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.682</td>
<td>0.001</td>
</tr>
<tr>
<th>YB2</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.668</td>
<td>0.003</td>
</tr>
<tr>
<th>YB5</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.646</td>
<td>0.010</td>
</tr>
</table>

'''Table 4.''' Discrimination scores of the submitted algorithms.

File:2019 poly cs.png

2019-10-29T13:54:59Z

Berit Janssen: P4P 2019 poly cs

== Summary ==
P4P 2019 poly cs

2019:Patterns for Prediction Results

2019-10-29T13:41:46Z

Berit Janssen: /* Contribution */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2019:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.
===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 3.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>crossentropy</th>
<th>nll_var</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>0.313</td>
<td>0.000</td>
</tr>
<tr>
<th>FC1</th>
<th>mono</th>
<td>499</td>
<td>0.864</td>
<td>0.693</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.318</td>
<td>0.001</td>
</tr>
<tr>
<th>FC1</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.682</td>
<td>0.001</td>
</tr>
<tr>
<th>YB2</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.668</td>
<td>0.003</td>
</tr>
<tr>
<th>YB5</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.646</td>
<td>0.010</td>
</tr>
</table>

'''Table 4.''' Discrimination scores of the submitted algorithms.

2019:Patterns for Prediction Results

2019-10-24T19:13:41Z

Berit Janssen: /* Tables */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2018:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.
===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 3.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>crossentropy</th>
<th>nll_var</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>0.313</td>
<td>0.000</td>
</tr>
<tr>
<th>FC1</th>
<th>mono</th>
<td>499</td>
<td>0.864</td>
<td>0.693</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.318</td>
<td>0.001</td>
</tr>
<tr>
<th>FC1</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.682</td>
<td>0.001</td>
</tr>
<tr>
<th>YB2</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.668</td>
<td>0.003</td>
</tr>
<tr>
<th>YB5</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.646</td>
<td>0.010</td>
</tr>
</table>

'''Table 4.''' Discrimination scores of the submitted algorithms.

2019:Patterns for Prediction Results

2019-10-24T19:13:19Z

Berit Janssen: /* Datasets and Algorithms */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2018:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2019.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 1.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.
===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>crossentropy</th>
<th>nll_var</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>0.313</td>
<td>0.000</td>
</tr>
<tr>
<th>FC1</th>
<th>mono</th>
<td>499</td>
<td>0.864</td>
<td>0.693</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.318</td>
<td>0.001</td>
</tr>
<tr>
<th>FC1</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.682</td>
<td>0.001</td>
</tr>
<tr>
<th>YB2</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.668</td>
<td>0.003</td>
</tr>
<tr>
<th>YB5</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.646</td>
<td>0.010</td>
</tr>
</table>

'''Table 3.''' Discrimination scores of the submitted algorithms.

2019:Patterns for Prediction Results

2019-10-23T19:39:45Z

Berit Janssen: /* Tables */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2018:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2018. Seg2SegPvP and BachProp are models based on LSTM networks.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 1.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.
===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

===symMono/symPoly===
====Implicit task: discriminate true and foil continuation====
<table border="1" class="dataframe">
<tr style="text-align: right;">
<th></th>
<th></th>
<th>nr_obs</th>
<th>accuracy</th>
<th>crossentropy</th>
<th>nll_var</th>
</tr>
<tr>
<th>model</th>
<th>data</th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>EP1</th>
<th>mono</th>
<td>500</td>
<td>1.000</td>
<td>0.313</td>
<td>0.000</td>
</tr>
<tr>
<th>FC1</th>
<th>mono</th>
<td>499</td>
<td>0.864</td>
<td>0.693</td>
<td>0.000</td>
</tr>
<tr>
<th>EP1</th>
<th>poly</th>
<td>500</td>
<td>0.998</td>
<td>0.318</td>
<td>0.001</td>
</tr>
<tr>
<th>FC1</th>
<th>poly</th>
<td>499</td>
<td>0.916</td>
<td>0.682</td>
<td>0.001</td>
</tr>
<tr>
<th>YB2</th>
<th>poly</th>
<td>499</td>
<td>0.703</td>
<td>0.668</td>
<td>0.003</td>
</tr>
<tr>
<th>YB5</th>
<th>poly</th>
<td>499</td>
<td>0.731</td>
<td>0.646</td>
<td>0.010</td>
</tr>
</table>

'''Table 3.''' Discrimination scores of the submitted algorithms.

2019:Patterns for Prediction Results

2019-10-23T19:23:25Z

Berit Janssen: /* Tables */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2018:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2018. Seg2SegPvP and BachProp are models based on LSTM networks.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>

'''Table 1.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.
===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

'''Table 2.''' Pitch overlap of the algorithmic continuations with the true continuation - mean, median and standard deviation.

2019:Patterns for Prediction Results

2019-10-23T19:22:03Z

Berit Janssen:

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2018:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2018. Seg2SegPvP and BachProp are models based on LSTM networks.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

==Tables==
===symMono===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Modulo12Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.502</td>
<td>0.516</td>
<td>0.219</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.596</td>
<td>0.612</td>
<td>0.292</td>
</tr>
<tr>
<th>Markov</th>
<td>0.583</td>
<td>0.608</td>
<td>0.195</td>
</tr>
<tr>
<th>Seq2SeqP4P</th>
<td>0.087</td>
<td>0.000</td>
<td>0.121</td>
</tr>
</table>
===symPoly===
====Explicit task: generate music given a prime====
<table border="1" class="dataframe">
<tr>
<th></th>
<th colspan="3" halign="left">Pitch</th>
</tr>
<tr>
<th></th>
<th>mean</th>
<th>median</th>
<th>std</th>
</tr>
<tr>
<th>Model</th>
<th></th>
<th></th>
<th></th>
</tr>
<tr>
<th>BachProb</th>
<td>0.455</td>
<td>0.466</td>
<td>0.139</td>
</tr>
<tr>
<th>CopyForward</th>
<td>0.594</td>
<td>0.598</td>
<td>0.238</td>
</tr>
<tr>
<th>Markov</th>
<td>0.506</td>
<td>0.508</td>
<td>0.176</td>
</tr>
</table>

2019:Patterns for Prediction Results

2019-10-23T19:15:34Z

Berit Janssen: /* Explicit task: generate music given a prime */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2018:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2018. Seg2SegPvP and BachProp are models based on LSTM networks.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|800px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

2019:Patterns for Prediction Results

2019-10-23T19:15:07Z

Berit Janssen: /* Explicit task: generate music given a prime */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2018:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2018. Seg2SegPvP and BachProp are models based on LSTM networks.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|1000px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====
[[File:2019_poly_cs.png|1000px]]

'''Figure 3.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_poly_pitch.png|800px]]

'''Figure 4.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

File:2019 poly pitch.png

2019-10-23T19:14:43Z

Berit Janssen: 2019 P4P poly pitch score

== Summary ==
2019 P4P poly pitch score

2019:Patterns for Prediction Results

2019-10-23T19:10:04Z

Berit Janssen: /* Explicit task: generate music given a prime */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2018:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2018. Seg2SegPvP and BachProp are models based on LSTM networks.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

[[File:2019_mono_pitch.png|1000px]]

'''Figure 2.''' Pitch overlap of the algorithmically generated continuations with the true continuation.

===symPoly===
====Explicit task: generate music given a prime====

File:2019 mono pitch.png

2019-10-23T19:05:54Z

Berit Janssen: 2019 P4P SymMono pitch score

== Summary ==
2019 P4P SymMono pitch score

2019:Patterns for Prediction Results

2019-10-23T19:04:07Z

Berit Janssen: /* Explicit task: generate music given a prime */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2018:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2018. Seg2SegPvP and BachProp are models based on LSTM networks.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|1000px]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

===symPoly===
====Explicit task: generate music given a prime====

2019:Patterns for Prediction Results

2019-10-23T19:03:09Z

Berit Janssen: /* Explicit task: generate music given a prime */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2018:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2018. Seg2SegPvP and BachProp are models based on LSTM networks.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png]]

'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

===symPoly===
====Explicit task: generate music given a prime====

2019:Patterns for Prediction Results

2019-10-23T19:02:49Z

Berit Janssen:

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2018:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symPoly - Task 1
!
!
|-
! TD1
| CopyForward || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
|-
|- style="background: green;"
! Task Version
! symMono - Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
|- style="background: green;"
! Task Version
! symPoly Task 2
!
!
|-
! EP1
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
|-
! YB2
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
|-
! YB5
| ModelName? || style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]

|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2018. Seg2SegPvP and BachProp are models based on LSTM networks.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitch-onset pairs are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2019:Patterns_for_Prediction#Evaluation_Procedure]].)

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2019_mono_cs.png|800px]]
'''Figure 1.''' Precision, recall and F1 (cardinality score) in quarter note onsets from prediction start.

===symPoly===
====Explicit task: generate music given a prime====

File:2019 mono cs.png

2019-10-23T18:59:40Z

Berit Janssen: Cardinality Score SymMono

== Summary ==
Cardinality Score SymMono

2019:Patterns for Prediction Results

2019-10-23T18:51:07Z

Berit Janssen: /* Datasets and Algorithms */

2019:Patterns for Prediction Results

2019-10-23T18:30:10Z

2019:Patterns for Prediction

2019-07-02T13:20:10Z

Berit Janssen: /* Evaluation Procedure */

== Description ==
'''In brief''': (1) Algorithms that take an excerpt of music as input (the ''prime''), and output a predicted ''continuation'' of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

Your task captains are [http://beritjanssen.com/ Berit Janssen] (berit.janssen), [http://tomcollinsresearch.net/ Tom Collins] (tomthecollins), and Iris Yuping Ren (yuping.ren.iris all at gmail.com). Please copy in all three of us if you have questions/comments.

The '''submission deadline''' is '''TO BE DETERMINED'''.

'''Relation to the pattern discovery task''': The Patterns for Prediction task is an offshoot of the [https://www.music-ir.org/mirex/wiki/2013:Discovery_of_Repeated_Themes_%26_Sections Discovery of Repeated Themes & Sections task] (2013-2017). We hope to run the former (Patterns for Prediction) task and pause the latter (Discovery of Repeated Themes & Sections). In future years we may run both.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next ''N'' musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; [https://www.music-ir.org/mirex/wiki/2013:Discovery_of_Repeated_Themes_%26_Sections MIREX Discovery of Repeated Themes & Sections task]; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt ([https://www.music-ir.org/mirex/abstracts/2013/DM10.pdf Meredith, 2013]). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

==Data==
The Patterns for Prediction Development Dataset (PPDD-Sep2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it is not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Here are the PPDD-Sep2018 variants for download:
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_aud_mono_small.zip audio, monophonic, small] (92 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_aud_mono_medium.zip audio, monophonic, medium] (850 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_aud_mono_large.zip audio, monophonic, large] (8.46 GB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_aud_poly_small.zip audio, polyphonic, small] (137 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_aud_poly_medium.zip audio, polyphonic, medium] (1.35 GB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_aud_poly_large.zip audio, polyphonic, large] (13.44 GB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_sym_mono_small.zip symbolic, monophonic, small] (< 1 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_sym_mono_medium.zip symbolic, monophonic, medium] (3 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_sym_mono_large.zip symbolic, monophonic, large] (32 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_sym_poly_small.zip symbolic, polyphonic, small] (< 1 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_sym_poly_medium.zip symbolic, polyphonic, medium] (9 MB)
*[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-sep2018/PPDD-Sep2018_sym_poly_large.zip symbolic, polyphonic, large] (64 MB)
("Large" datasets were compressed using the [https://www.mankier.com/1/7za p7zip] package, installed via "brew install p7zip on Mac".)

===Some examples===
[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-jul2018/examples/0a983538-61b5-4b9d-9ad9-23e05f548e5c.wav This prime] finishes with two G’s followed by a D above. Looking at the [http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-jul2018/examples/0a983538-61b5-4b9d-9ad9-23e05f548e5c.png piano roll] or listening to the linked file, we can see/hear that this pitch pattern, in the exact same rhythm, has happened before (see bars 17-18 transition in the piano roll). Therefore, we and/or an algorithm, might predict that the first note of the continuation will follow the pattern established in the previous occurrence, returning to G 1.5 beats later.

[http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-jul2018/examples/001f5992-527d-4e04-8869-afa7cbb74cd0.wav This] is another example where a previous occurrence of a pattern might help predict the contents of the continuation. Not all excerpts contain patterns (in fact, one of the motivations for running the task is to interrogate the idea that patterns are abundant in music and always informative in terms of predicting what comes next). [http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-jul2018/examples/fc2fda7c-9f55-4bf3-8fa8-f337e35aa20f.wav This one], for instance, does not seem to contain many clues for what will come next. And finally, [http://tomcollinsresearch.net/research/data/mirex/ppdd/ppdd-jul2018/examples/b9261e74-125a-429e-ae27-5b51abdc7d81.wav this one] might not contain any obvious patterns, but other strategies (such as schematic or tonal expectations) might be recruited in order to predict the contents of the continuation.

(These examples are from an earlier version of the dataset, PPDD-Jul2018, but the above observations apply also to the current version of the dataset.)

===Preparation of the data===
Preparation of the monophonic datasets was more involved than the polyphonic datasets: for both, we imported each MIDI file, quantised it using a subset of the Farey sequence of order 6 (Collins, Krebs, et al., 2014), and then excerpted a prime and continuation at a randomly selected time. For the monophonic datasets, we filtered for:
*channels that contained at least 20 events in the prime;
*channels that were at least 80% monophonic at the outset, meaning that at least 80% of their segments (Pardo & Birmingham, 2002) contained no more than one event;
*channels where the maximum inter-ontime interval in the prime was no more than 8 quarter-note beats.
*we then "skylined" these channels (independently) so that no two events had the same start time (maximum MNN chosen in event of a clash), and double-checked that they still contained at least 20 events;
*one suitable channel was then selected at random, and the prime appears in the dataset if the continuation contained at least 10 events.
If any of the above could not be satisfied for the given input, we skipped this MIDI file.

For the polyphonic data, we applied the minimum note criteria of 20 in the prime and 10 in the continuation, as well as the prime maximum inter-ontime interval of 8, but it was not necessary to measure monophony or perform skylining.

Audio files were generated by importing the corresponding CSV and descriptor files and using a sample bank of piano notes from the [https://magenta.tensorflow.org/datasets/nsynth Google Magenta NSynth dataset] (Engel et al., 2017) to construct and export the waveform.

The foil continuations were generated using a Markov model of order 1 over the whole texture (polyphonic) or channel (monophonic) in question, and there was '''no''' attempt to nest this generation process in any other process cognisant of repetitive or phrasal structure. See Collins and Laney (2017) for details of the state space and transition matrix.

==Submission Format==
In terms of input representations, we will evaluate 4 largely independent versions of the task: audio, monophonic; audio, polyphonic; symbolic, monophonic; symbolic, polyphonic. Participants may submit algorithms to 1 or more of these versions, and should list these versions clearly in their readme. '''Irrespective of input representation''', all output for subtask (1) should be in "ontime", "MNN" CSV files. The CSV may contain other information, but "ontime" and "MNN" should be in the first two columns, respectively. All output for subtask (2) should be an indication whether of the two presented continuations, "A" or "B" is judged by the algorithm to be genuine. This should be one CSV file for an entire dataset, with first column "id" referring to the file name of a prime-continuation pair, second column "A" containing a likelihood value in [0, 1] for the genuineness of the continuation in folder A, and column "B" similarly for the continuation in folder B.

All submissions should be statically linked to all dependencies and include a README file including the following information:

*input representation(s), should be 1 or more of "audio, monophonic"; "audio, polyphonic"; "symbolic, monophonic"; "symbolic, polyphonic";
*subtasks you would like your algorithm to be evaluated on, should be "1", "2", or "1 and 2" (see first sentences of [[2018:Patterns_for_Prediction#Description]] for a reminder);
*command line calling format for all executables and an example formatted set of commands;
*number of threads/cores used or whether this should be specified on the command line;
*expected memory footprint;
*expected runtime;
*any required environments and versions, e.g. Python, Java, Bash, MATLAB.

===Example Command Line Calling Format===

Python:

python <your_script_name.py> -i <input_folder> -o <output_folder>

==Evaluation Procedure==
'''In brief''': For subtask (1), we match the algorithmic output with the original continuation and compute a match score (see implementation at [https://github.com/BeritJanssen/PatternsForPrediction/tree/mirex2019 GitHub]). For subtask (2), we count up how many times an algorithm judged the genuine continuation as most likely.

The input excerpt ends with a final note event: <math>(x_0, y_0)</math>, where <math>x_0</math> is ontime (start time measured in quarter-note beats starting with 0 for bar 1 beat 1), <math>y_0</math> is pitch, represented by MNN.

The algorithm predicts the continuations: <math>(\hat{x}_1, \hat{y}_1)</math>, <math>(\hat{x}_2, \hat{y}_2)</math>, ..., <math>(\hat{x}_{n^\prime}, \hat{y}_{n^\prime})</math>, where <math>\hat{x}_i</math> are predicted ontimes, and <math>\hat{y}_i</math> are predicted MNNs. The true continuations are notated <math>(x_1, y_1), (x_2, y_2),..., (x_n, y_n)</math>. The predicted continuation ontimes are strictly increasing, that is <math>x_0 < \hat{x}_1 < \cdots < \hat{x}_{n^\prime}</math>, and so are the true continuation ontimes, that is <math>x_0 < x_1 < \cdots < x_n</math>.

===Subtask 1===
We represent each note in the true and algorithmic continuation as a point in a two-dimensional space of onset and pitch, giving the point-set <math>\mathbf{P}</math> for the true continuation, and <math>\mathbf{Q}</math> for the algorithmic continuation. We calculate differences between all points <math>p_i</math> in <math>\mathbf{P}</math> and <math>q_j</math> in <math>\mathbf{Q}</math>, which represent the translation vectors <math>\mathbf{T}</math> to transform a given algorithmically generated note into a note from the true continuation:

<math>
\text{cp}(\mathbf{P},\mathbf{Q}) = \max_\mathbf{T} |\{q_j | q_j \in \mathbf{Q} \wedge q_j + \mathbf{T} \in \mathbf{P}\}|
</math>

We define recall as the number of correctly predicted notes, divided by the cardinality of the true continuation point set <math>\mathbf{P}</math>. Since there exists at least one point in <math>\mathbf{Q}</math> which can be translated by any vector to a point in <math>\mathbf{P}</math>, we subtract <math>1</math> from numerator and denominator to scale to <math>[0,1]</math>.

<math>
\text{Rec} = (\text{cp}(\mathbf{P},\mathbf{Q}) - 1) / (|\mathbf{P}| - 1)
</math>

Precision is the number of correctly predicted notes, divided by the cardinality of the point set of the algorithmic continuation <math>\mathbf{Q}</math>, scaled in the same way:

<math>
\text{Prec} = (\text{cp}(\mathbf{P},\mathbf{Q}) - 1) / (|\mathbf{Q}| - 1)
</math>

===Entropy===
Some existing work in this area (e.g., Conklin & Witten, 1995; Pearce & Wiggins, 2006; Temperley, 2007) evaluates algorithm performance in terms of entropy. If we have time to collect human listeners' judgments of likely (or not) continuations for given excerpts, then we will be in a position to compare the entropy of listener-generated distributions with the corresponding algorithm distributions. This would open up the possibility of entropy-based metrics, but we consider this of secondary importance to the metrics outlined above.

==Questions (Q), Answers (A), and Comments (C)==

Q. Instead of evaluating continuations, have you considered evaluating an algorithm's ability to predict content between two timepoints, or before a timepoint?

A. Yes we considered including this also, but opted not to for sake of simplicity. Furthermore, these alternatives do not have the same intuitive appeal as predicting future events.

Q. Why do some files sound like they contain a drum track rendered on piano?

A. Some of the MIDI files import as a single channel, but upon listening to them it is evident that they contain multiple instruments. For the sake of simplicity, we removed percussion channels where possible, but if everything was squashed down into a single channel, there was not much we could do.

C. to_the_sun--at--gmx.com writes: "This is exactly what I'm interested in! I have an open-source project called The Amanuensis (https://github.com/to-the-sun/amanuensis) that uses an algorithm to predict where in the future beats are likely to fall.

"Amanuensis constructs a cohesive song structure, using the best of what you give it, looping around you and growing in real-time as you play. All you have to do is jam and fully written songs will flow out behind you wherever you go.

"My algorithm right now is only rhythm-based and I'm sure it's not sophisticated enough to be entered into your contest, but I would be very interested in the possibility of using any of the algorithms that are, in place of mine in The Amanuensis. Would any of your participants be interested in some collaboration? What I can bring to the table would be a real-world application for these algorithms, already set for implementation."

Q. I'm interested in performing this task on the symbolic dataset, but I don't have an audio-based algorithm. It was unclear to me if the inputs are audio, symbolic, both, or either.

A. We have clarified, at the top of [[2018:Patterns_for_Prediction#Submission_Format]], that submissions in 1-4 representational categories are acceptable. It's also OK, say, for an audio-based algorithm to make use of the descriptor file in order to determine beat locations. (You could do this by looking at the <math>u = \mathrm{bpm}</math> value, and then you would know that the main beats in the WAV file are at <math>0, 60/u, 2 \cdot 60/u,\ldots</math> sec.)

==Time and Hardware Limits==

A total runtime limit of 72 hours will be imposed on each submission.

==Seeking Contributions==

*We would like to evaluate against real (not just synthesized-from-MIDI) audio versions. If you have a good idea of how we might make this available to participants, let us know. We would be happy to acknowledge individuals and/or companies for helping out in this regard.

*More suggestions/comments/ideas on the task is always welcome!

==Acknowledgments==

Thank you to Anja Volk, Darrell Conklin, Srikanth Cherla, David Meredith, Matevz Pesek, and Gissel Velarde for discussions!

==References==
*Cherla, S., Weyde, T., Garcez, A., and Pearce, M. (2013). A distributed model for multiple-viewpoint melodic prediction. In In ''Proceedings of the International Society for Music Information Retrieval Conference'' (pp. 15-20). Curitiba, Brazil.

*Collins, T. (2011). "[http://oro.open.ac.uk/30103/ Improved methods for pattern discovery in music, with applications in automated stylistic composition]". PhD Thesis.

*Collins, T., Böck, S., Krebs, F., & Widmer, G. (2014). [http://tomcollinsresearch.net/pdf/collinsEtAlAES2014.pdf Bridging the audio-symbolic gap: The discovery of repeated note content directly from polyphonic music audio]. In ''Proceedings of the Audio Engineering Society's 53rd Conference on Semantic Audio''. London, UK.

*Collins, T., Tillmann, B., Barrett, F. S., Delbé, C., & Janata, P. (2014). [http://psycnet.apa.org/journals/rev/121/1/33/ A combined model of sensory and cognitive representations underlying tonal expectations in music: From audio signals to behavior]. ''Psychological Review, 121''(1), 33-65.

*Collins T., & Laney, R. (2017). [http://jcms.org.uk/issues/Vol1Issue2/computer-generated-stylistic-compositions/computer-generated-stylistic-compositions.html Computer-generated stylistic compositions with long-term repetitive and phrasal structure]. ''Journal of Creative Music Systems, 1''(2).

*Conklin, D., and Witten, I. H. (1995). Multiple viewpoint systems for music prediction. ''Journal of New Music Research, 24''(1), 51-73.

*Elmsley, A., Weyde, T., & Armstrong, N. (2017). Generating time: Rhythmic perception, prediction and production with recurrent neural networks. ''Journal of Creative Music Systems, 1''(2).

*Engel, J., Resnick, C., Roberts, A., Dieleman, S., Eck, D., Simonyan, K., & Norouzi, M. (2017). Neural audio synthesis of musical notes with WaveNet autoencoders. https://arxiv.org/abs/1704.01279

*Gjerdingen, R. O. (1989). Using connectionist models to explore complex musical patterns. ''Computer Music Journal, 13''(3), 67-75.

*Gjerdingen, R. (2007). Music in the galant style. New York, NY: Oxford University Press.

*Hadjeres, G., Pachet, F., & Nielsen, F. (2016). Deepbach: A steerable model for Bach chorales generation. arXiv preprint arXiv:1612.01010.

*Huron, D. (2006). Sweet Anticipation. Music and the Psychology of Expectation. Cambridge, MA: MIT Press.

*Janssen, B., Burgoyne, J. A., & Honing, H. (2017). Predicting variation of folk songs: A corpus analysis study on the memorability of melodies. ''Frontiers in Psychology, 8'', 621.

*Janssen, B., van Kranenburg, P., & Volk, A. (2017). Finding occurrences of melodic segments in folk songs employing symbolic similarity measures. ''Journal of New Music Research, 46''(2), 118-134.

*Koelsch, S., Gunter, T. C., Wittfoth, M., & Sammler, D. (2005). Interaction between syntax processing in language and in music: an ERP study. ''Journal of Cognitive Neuroscience, 17''(10), 1565-1577.

*Lerdahl, F., and Jackendoff, R. (1983). "A generative theory of tonal music. Cambridge, MA: MIT Press.

*Margulis, E. H. (2014). ''On repeat: How music plays the mind''. New York, NY: Oxford University Press.

*Meredith, D. (1999). The computational representation of octave equivalence in the Western staff notation system. In ''Proceedings of the Cambridge Music Processing Colloquium''. Cambridge, UK.

*Meredith, D. (2013). COSIATEC and SIATECCompress: Pattern discovery by geometric compression. In ''Proceedings of the 10th Annual Music Information Retrieval Evaluation eXchange (MIREX'13)''. Curitiba, Brazil.

*Pardo, B., & Birmingham, W. P. (2002). Algorithms for chordal analysis. ''Computer Music Journal, 26''(2), 27-49.

*Pearce, M. T., & Wiggins, G. A. (2006). Melody: The influence of context and learning. ''Music Perception, 23''(5), 377–405.

*Raffel, C. (2016). "Learning-based methods for comparing sequences, with applications to audio-to-MIDI alignment and matching". PhD Thesis.

*Ren, I.Y., Koops, H.V, Volk, A., Swierstra, W. (2017). In search of the consensus among musical pattern discovery algorithms. In ''Proceedings of the International Society for Music Information Retrieval Conference'' (pp. 671-678). Suzhou, China.

*Roberts, A., Engel, J., Raffel, C., Hawthorne, C., & Eck, D. (2018). A hierarchical latent vector model for learning long-term structure in music. In ''Proceedings of the International Conference on Machine Learning'' (pp. 4361-4370). Stockholm, Sweden.

*Rohrmeier, M., & Pearce, M. (2018). Musical syntax I: theoretical perspectives. In ''Springer Handbook of Systematic Musicology'' (pp. 473-486). Berlin, Germany: Springer.

*Schellenberg, E. G. (1997). Simplifying the implication-realization model of melodic expectancy. ''Music Perception, 14''(3), 295-318.

*Schmuckler, M. A. (1989). Expectation in music: Investigation of melodic and harmonic processes. ''Music Perception, 7''(2), 109-149.

*Sturm, B.L., Santos, J.F., Ben-Tal., O., & Korshunova, I. (2016), Music transcription modelling and composition using deep learning. In ''Proceedings of the International Conference on Computer Simulation of Musical Creativity''. Huddersfield, UK.

*Temperley, D. (2007). ''Music and probability''. Cambridge, MA: MIT Press.

*Widmer, G. (2017). Getting closer to the essence of music: The con espressione manifesto. ''ACM Transactions on Intelligent Systems and Technology (TIST), 8''(2), 19.

2018:Patterns for Prediction Results

2018-09-23T18:39:41Z

Berit Janssen: /* Results */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2018:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono
!
!
|-
! EN1
| Seq2SeqP4P || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/EN1.pdf PDF] || [http://ericpnichols.com/ Eric Nichols]
|-
! FC1
| BachProp || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/FC1.pdf PDF] || [https://scholar.google.com/citations?user=rpZVNKYAAAAJ&hl=en Florian Colombo]
|-
! MM1
| First-order Markov model || style="text-align: center;" | Task captains || For purposes of comparison
|-
|- style="background: green;"
! Task Version
! symPoly
!
!
|-
! FC1
| BachProp || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/FC1.pdf PDF] || [https://scholar.google.com/citations?user=rpZVNKYAAAAJ&hl=en Florian Colombo]
|-
! MM1
| First-order Markov model || style="text-align: center;" | Task captains || For purposes of comparison
|-
|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2018. Seg2SegPvP and BachProp are models based on LSTM networks.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitches and inter-onset intervals (with relation to the last onset of the prime) are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2018:Patterns_for_Prediction#Evaluation_Procedure]].)

===SymMono===
For the implicit task, the two LSTM based models make their best predictions close to the cut-off point (i.e., last event of the prime). As the onset time after the cut-off point increases, the Markov Model outperforms the LSTM based models, with the exception of recall of inter-onset interval (cf. Figure 4), where FC1 consistently performs better than the Markov Model. Possibly in consequence of the poorer pitch performance, also fewer relevant pitch-ioi pairs were detected by the LSTM models as onset time after cutoff point increases.

For the explicit task, only FC1 was submitted. It outperforms chance level significantly, at 0.87, i.e., picking the correct continuation in almost 90% of the cases (See Table 1).

===SymPoly===
Only one LSTM model was submitted to SymPoly (FC1), with results comparable to SymMono (see Figures 10-18, Table 1).

==Discussion==

...

Berit Janssen, Iris Ren, Tom Collins.

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2018_mono_R_pitch.png|600px]]

'''Figure 1.''' Recall of generated pitches after cutoff point.

[[File:2018_mono_P_pitch.png|600px]]

'''Figure 2.''' Precision of generated pitches after cutoff point.

[[File:2018_mono_F1_pitch.png|600px]]

'''Figure 3.''' F1 measure of generated pitches after cutoff point.

[[File:2018_mono_R_ioi.png|600px]]

'''Figure 4.''' Recall of generated inter-onset intervals after cutoff point.

[[File:2018_mono_P_ioi.png|600px]]

'''Figure 5.''' Precision of generated inter-onset intervals after cutoff point.

[[File:2018_mono_F1_ioi.png|600px]]

'''Figure 6.''' F1 measure of generated inter-onset intervals after cutoff point.

[[File:2018_mono_R_pairs.png|600px]]

'''Figure 7.''' Recall of generated pitch-ioi pairs after cutoff point.

[[File:2018_mono_P_pairs.png|600px]]

'''Figure 8.''' Precision of generated pitch-ioi pairs after cutoff point.

[[File:2018_mono_F1_pitch.png|600px]]

'''Figure 9.''' F1 measure of generated pitch-ioi pairs after cutoff point.

===Implicit Task: Polyphonic===

[[File:2018_poly_R_pitch.png|600px]]

'''Figure 10.''' Recall of generated pitches after cutoff point.

[[File:2018_poly_P_pitch.png|600px]]

'''Figure 11.''' Precision of generated pitches after cutoff point.

[[File:2018_poly_F1_pitch.png|600px]]

'''Figure 12.''' F1 measure of generated pitches after cutoff point.

[[File:2018_poly_R_ioi.png|600px]]

'''Figure 13.''' Recall of generated inter-onset intervals after cutoff point.

[[File:2018_poly_P_ioi.png|600px]]

'''Figure 14.''' Precision of generated inter-onset intervals after cutoff point.

[[File:2018_poly_F1_ioi.png|600px]]

'''Figure 15.''' F1 measure of generated inter-onset intervals after cutoff point.

[[File:2018_poly_R_pairs.png|600px]]

'''Figure 16.''' Recall of generated pitch-ioi pairs after cutoff point.

[[File:2018_poly_P_pairs.png|600px]]

'''Figure 17.''' Precision of generated pitch-ioi pairs after cutoff point.

[[File:2018_poly_F1_pairs.png|600px]]

'''Figure 18.''' F1 measure of generated pitch-ioi pairs after cutoff point.

==Tables==
====Implicit task: decide which of two continuations is the true one, given the prime====
{| border="1" cellspacing="0" style="text-align: left; width: 280px;"
|-
! width="120" | Algorithm
! width="80" | Monophonic
! width="80" | Polyphonic
|-
|-
! FC1
| 0.87 || 0.92
|-
|-
! Sig. > chance
| 0.54 || 0.54
|-
|}

2018:Patterns for Prediction Results

2018-09-23T18:35:22Z

Berit Janssen: /* SymPoly */

== Introduction ==

'''In brief''':

We look for

(1) Algorithms that take an excerpt of music as input (the prime), and output a predicted continuation of the excerpt.

(2) Additionally or alternatively, algorithms that take a prime and one or more continuations as input, and output the likelihood that each continuation is the genuine extension of the prime.

'''In more detail''': One facet of human nature comprises the tendency to form predictions about what will happen in the future (Huron, 2006). Music, consisting of complex temporally extended sequences, provides an excellent setting for the study of prediction, and this topic has received attention from fields including but not limited to psychology (Collins, Tillmann, et al., 2014; Janssen, Burgoyne and Honing, 2017; Schellenberg, 1997; Schmukler, 1989), neuroscience (Koelsch et al., 2005), music theory (Gjerdingen, 2007; Lerdahl & Jackendoff, 1983; Rohrmeier & Pearce, 2018), music informatics (Conklin & Witten, 1995; Cherla et al., 2013), and machine learning (Elmsley, Weyde, & Armstrong, 2017; Hadjeres, Pachet, & Nielsen, 2016; Gjerdingen, 1989; Roberts et al., 2018; Sturm et al., 2016). In particular, we are interested in the way exact and inexact repetition occurs over the short, medium, and long term in pieces of music (Margulis, 2014; Widmer, 2016), and how these repetitions may interact with "schematic, veridical, dynamic, and conscious" expectations (Huron, 2006) in order to form a basis for successful prediction.

We call for algorithms that may model such expectations so as to predict the next musical events based on given, foregoing events (the prime). We invite contributions from all fields mentioned above (not just pattern discovery researchers), as different approaches may be complementary in terms of predicting correct continuations of a musical excerpt. We would like to explore these various approaches to music prediction in a MIREX task. For subtask (1) above (see "In brief"), the development and test datasets will contain an excerpt of a piece up until a cut-off point, after which the algorithm is supposed to generate the next N musical events up until 10 quarter-note beats, and we will quantitatively evaluate the extent to which an algorithm's continuation corresponds to the genuine continuation of the piece. For subtask (2), in addition to containing a prime, the development and test datasets will also contain continuations of the prime, one of which will be genuine, and the algorithm should rate the likelihood that each continuation is the genuine extension of the prime, which again will be evaluated quantitatively.

What is the relationship between pattern discovery and prediction? The last five years have seen an increasing interest in algorithms that discover or generate patterned data, leveraging methods beyond typical (e.g., Markovian) limits (Collins & Laney, 2017; MIREX Discovery of Repeated Themes & Sections task; Janssen, van Kranenburg and Volk, 2017; Ren et al., 2017; Widmer, 2016). One of the observations to emerge from the above-mentioned MIREX pattern discovery task is that an algorithm that is "good" at discovering patterns ought to be extendable to make "good" predictions for what will happen next in a given music excerpt (Meredith, 2013). Furthermore, evaluating the ability to predict may provide a stronger (or at least complementary) evaluation of an algorithm's pattern discovery capabilities, compared to evaluating its output against expert-annotated patterns, where the notion of "ground truth" has been debated (Meredith, 2013).

== Contribution ==

...

For a more detailed introduction to the task, please see [[2018:Patterns for Prediction]].

== Datasets and Algorithms ==

The Patterns for Prediction Development Dataset (PPDD-Jul2018) has been prepared by processing a randomly selected subset of the [http://colinraffel.com/projects/lmd/ Lakh MIDI Dataset] (LMD, Raffel, 2016). It has audio and symbolic versions crossed with monophonic and polyphonic versions. The audio is generated from the symbolic representation, so it is not "expressive". The symbolic data is presented in CSV format. For example,

20,64,62,0.5,0
20.66667,65,63,0.25,0
21,67,64,0.5,0
...

would be the start of a prime where the first event had ontime 20 (measured in quarter-note beats -- equivalent to bar 6 beat 1 if the time signature were 4-4), MIDI note number (MNN) 64, estimated morphetic pitch number 62 (see [http://tomcollinsresearch.net/research/data/mirex/ppdd/mnn_mpn.pdf p. 352] from Collins, 2011 for a diagrammatic explanation; for more details, see Meredith, 1999), duration 0.5 in quarter-note beats, and channel 0. Re-exports to MIDI are also provided, mainly for listening purposes. We also provide a descriptor file containing the original Lakh MIDI Dataset id, the BPM, time signature, and a key estimate. The audio dataset contains all these files, plus WAV files. Therefore, the audio and symbolic variants are identical to one another, apart from the presence of WAV files. All other variants are non-identical, although there may be some overlap, as they were all chosen from LMD originally.

The provenance of the Patterns for Prediction Test Dataset (PPTD) will '''not''' be disclosed, but it shares simiarlity with LMD and not from LMD, if you are concerned about overfitting.

There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.

Submissions to the symMono and symPoly variants of the tasks are listed in Table 1. There were no submissions to the audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask, and can also serve as a point of comparison for the explicit task. It should be noted, however, that this model had access to the full song/piece – '''not just the prime''' – so it is at an advantage compared to EN1 and FC1 in the explicit task.

{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
|- style="background: green;"
! Task Version
! symMono
!
!
|-
! EN1
| Seq2SeqP4P || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/EN1.pdf PDF] || [http://ericpnichols.com/ Eric Nichols]
|-
! FC1
| BachProp || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/FC1.pdf PDF] || [https://scholar.google.com/citations?user=rpZVNKYAAAAJ&hl=en Florian Colombo]
|-
! MM1
| First-order Markov model || style="text-align: center;" | Task captains || For purposes of comparison
|-
|- style="background: green;"
! Task Version
! symPoly
!
!
|-
! FC1
| BachProp || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/FC1.pdf PDF] || [https://scholar.google.com/citations?user=rpZVNKYAAAAJ&hl=en Florian Colombo]
|-
! MM1
| First-order Markov model || style="text-align: center;" | Task captains || For purposes of comparison
|-
|}

'''Table 1. Algorithms submitted to Patterns for Prediction 2018. Seg2SegPvP and BachProp are models based on LSTM networks.'''

== Results ==

We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitches and inter-onset intervals (with relation to the last onset of the prime) are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2018:Patterns_for_Prediction#Evaluation_Procedure]].)

For the explicit tasks of SymMono and SymPoly, the two submitted algorithms overall perform worse than the first-order Markov Model introduced by the task captains for comparison purposes. However, there are some cases where the LSTMs outperform the Markov Model (see below). For the implicit task of SymMono and SymPoly, FC1 outperforms chance level significantly, and by a wide margin.

===SymMono===
For the implicit task, the two LSTM based models miss more relevant pitches than the Markov Model, especially in recall of inter-onset interval (cf. Figure 4), FC1 performs better than the Markov Model. Possibly in consequence of the poorer pitch performance, few relevant pitch-ioi pairs were detected by the LSTM models.

For the explicit task, only FC1 was submitted. It outperforms chance level significantly, at 0.88, i.e., picking the correct continuation at almost 90% of the cases (See Table 1).

===SymPoly===
Only one LSTM model was submitted to SymPoly (FC1), with results comparable to SymMono (see Figures 10-18, Table 1).

==Discussion==

...

Berit Janssen, Iris Ren, Tom Collins.

==Figures==
===symMono===
====Explicit task: generate music given a prime====

[[File:2018_mono_R_pitch.png|600px]]

'''Figure 1.''' Recall of generated pitches after cutoff point.

[[File:2018_mono_P_pitch.png|600px]]

'''Figure 2.''' Precision of generated pitches after cutoff point.

[[File:2018_mono_F1_pitch.png|600px]]

'''Figure 3.''' F1 measure of generated pitches after cutoff point.

[[File:2018_mono_R_ioi.png|600px]]

'''Figure 4.''' Recall of generated inter-onset intervals after cutoff point.

[[File:2018_mono_P_ioi.png|600px]]

'''Figure 5.''' Precision of generated inter-onset intervals after cutoff point.

[[File:2018_mono_F1_ioi.png|600px]]

'''Figure 6.''' F1 measure of generated inter-onset intervals after cutoff point.

[[File:2018_mono_R_pairs.png|600px]]

'''Figure 7.''' Recall of generated pitch-ioi pairs after cutoff point.

[[File:2018_mono_P_pairs.png|600px]]

'''Figure 8.''' Precision of generated pitch-ioi pairs after cutoff point.

[[File:2018_mono_F1_pitch.png|600px]]

'''Figure 9.''' F1 measure of generated pitch-ioi pairs after cutoff point.

===Implicit Task: Polyphonic===

[[File:2018_poly_R_pitch.png|600px]]

'''Figure 10.''' Recall of generated pitches after cutoff point.

[[File:2018_poly_P_pitch.png|600px]]

'''Figure 11.''' Precision of generated pitches after cutoff point.

[[File:2018_poly_F1_pitch.png|600px]]

'''Figure 12.''' F1 measure of generated pitches after cutoff point.

[[File:2018_poly_R_ioi.png|600px]]

'''Figure 13.''' Recall of generated inter-onset intervals after cutoff point.

[[File:2018_poly_P_ioi.png|600px]]

'''Figure 14.''' Precision of generated inter-onset intervals after cutoff point.

[[File:2018_poly_F1_ioi.png|600px]]

'''Figure 15.''' F1 measure of generated inter-onset intervals after cutoff point.

[[File:2018_poly_R_pairs.png|600px]]

'''Figure 16.''' Recall of generated pitch-ioi pairs after cutoff point.

[[File:2018_poly_P_pairs.png|600px]]

'''Figure 17.''' Precision of generated pitch-ioi pairs after cutoff point.

[[File:2018_poly_F1_pairs.png|600px]]

'''Figure 18.''' F1 measure of generated pitch-ioi pairs after cutoff point.

==Tables==
====Implicit task: decide which of two continuations is the true one, given the prime====
{| border="1" cellspacing="0" style="text-align: left; width: 280px;"
|-
! width="120" | Algorithm
! width="80" | Monophonic
! width="80" | Polyphonic
|-
|-
! FC1
| 0.87 || 0.92
|-
|-
! Sig. > chance
| 0.54 || 0.54
|-
|}