2019:Patterns for Prediction - Revision history

James.owers: /* Subtask 2 */

2019-11-03T19:32:39Z

‎Subtask 2

James.owers: /* Subtask 1 */

2019-11-03T19:32:20Z

‎Subtask 1

James.owers: /* The Pitch Score */

2019-11-03T19:31:45Z

‎The Pitch Score

James.owers: /* Description */

2019-11-03T19:30:51Z

‎Description

James.owers: /* Evaluation Procedure */

2019-11-03T19:26:15Z

‎Evaluation Procedure

James.owers: /* Description */ Clarify two subtasks

2019-11-03T18:53:30Z

‎Description: Clarify two subtasks

James.owers: /* Subtask 1 */ Added the gif

2019-08-27T10:07:30Z

‎Subtask 1: Added the gif

James.owers: /* Evaluation Procedure */ missed hat

2019-08-05T09:17:51Z

‎Evaluation Procedure: missed hat

Tom Collins: /* Evaluation Procedure */

2019-08-01T05:30:40Z

‎Evaluation Procedure

James.owers: /* Subtask 1 */ Another update to equations. Defined T explicitly. Removed unnecessary subscripts.

2019-07-31T14:00:30Z

‎Subtask 1: Another update to equations. Defined T explicitly. Removed unnecessary subscripts.

@@ Line 146: / Line 146: @@
 Some existing work in this area (e.g., Conklin & Witten, 1995; Pearce & Wiggins, 2006; Temperley, 2007) evaluates algorithm performance in terms of entropy. If we have time to collect human listeners' judgments of likely (or not) continuations for given excerpts, then we will be in a position to compare the entropy of listener-generated distributions with the corresponding algorithm distributions. This would open up the possibility of entropy-based metrics, but we consider this of secondary importance to the metrics outlined above.
-===Subtask 2===
+===Subtask 2 - Implicit Task===
 For subtask two, we provide each model with a prime and continuation and it returns the probability that this is the true continuation. We only provide two continuations - the true continuation, and a foil continuation generated by our baseline markov model. We then softmax these values such that the sum of probabilities for the two continuations is 1. The best model should assign a high probability to the true continuation (1) and a low continuation to the foil continuation (0).

@@ Line 101: / Line 101: @@
-===Subtask 1===
+===Subtask 1 - Explicit Task===
 For subtask 1, the algorithm is provided with a prime, and produces a continuation. We clip this continuation at 10 beats long (plus the time for the last note to finish). We then compare the generated continuation with the true continuation using two different scores: the cardinality and pitch score. In short, the cardinality score attempts to find the best overlap be between the continuations, and the pitch score makes a histogram of pitches for both continuations and checks the overlap between them.
@@ Line 145: / Line 145: @@
 ====A note on Entropy====
 Some existing work in this area (e.g., Conklin & Witten, 1995; Pearce & Wiggins, 2006; Temperley, 2007) evaluates algorithm performance in terms of entropy. If we have time to collect human listeners' judgments of likely (or not) continuations for given excerpts, then we will be in a position to compare the entropy of listener-generated distributions with the corresponding algorithm distributions. This would open up the possibility of entropy-based metrics, but we consider this of secondary importance to the metrics outlined above.
 ===Subtask 2===

@@ Line 139: / Line 139: @@
 The cardinality rewards continuations that are the correct 'shape' - it will give the maximum score to a continuation which is a copy of the true continuation but transposed one semitone down. This is not a very musically correct continuation! To counter this in some way, we also asses the pitches of the continuations more directly.
-To do this we create two *normalised* histograms of the pitches in both continuations. The score is simply the overlap between the overlaid histograms (maximum score is 1, minimum is 0).
+To do this we create two '''normalised''' histograms of the pitches in both continuations. The score is simply the overlap between the overlaid histograms (maximum score is 1, minimum is 0).
 We then repeat the process but disregard octave (i.e. take the MNN pitch modulo 12).
 ====A note on Entropy====

@@ Line 1: / Line 1: @@
 == Description ==
 '''In brief''', there are two subtasks:
-* Subtask 1) Algorithms that take an excerpt of music as input (the ''prime''), and output a predicted ''continuation'' of the excerpt
+* Subtask 1) '''The Explicit Task''' - Algorithms that take an excerpt of music as input (the ''prime''), and output a predicted ''continuation'' of the excerpt
-* Subtask 2) Algorithms that are given a prime and a candidate continuation and return the probability that this is the true continuation of the provided prime (i.e. the notes which occur immediately after)
+* Subtask 2) '''The Implicit Task''' - Algorithms that are given a prime and a candidate continuation and return the probability that this is the true continuation of the provided prime (i.e. the notes which occur immediately after)
 Your task captains are: [http://beritjanssen.com/ Berit Janssen] (berit.janssen), [https://sites.google.com/view/iyr/home Iris YuPing Ren] (yuping.ren.iris), [https://jamesowers.github.io/ James Owers] (james.f.owers), and [http://tomcollinsresearch.net/ Tom Collins] (tomthecollins, all at gmail.com). Please copy in all four of us if you have questions/comments.

@@ Line 87: / Line 87: @@
 ==Evaluation Procedure==
-'''In brief''': For subtask (1), we match the algorithmic output with the original continuation and compute a match score (see implementation at [https://github.com/BeritJanssen/PatternsForPrediction/tree/mirex2019 GitHub]). For subtask (2), we count up how many times an algorithm judged the genuine continuation as most likely.
+'''In brief''':
 The input excerpt ends with a final note event: <math>(x_0, y_0)</math>, where <math>x_0</math> is ontime (start time measured in quarter-note beats starting with 0 for bar 1 beat 1), <math>y_0</math> is pitch, represented by MNN.
-The algorithm predicts the continuations: <math>(\hat{x}_1, \hat{y}_1)</math>, <math>(\hat{x}_2, \hat{y}_2)</math>, ..., <math>(\hat{x}_{n^\prime}, \hat{y}_{n^\prime})</math>, where <math>\hat{x}_i</math> are predicted ontimes, and <math>\hat{y}_i</math> are predicted MNNs. The true continuations are notated <math>(x_1, y_1), (x_2, y_2),..., (x_n, y_n)</math>. The predicted continuation ontimes are strictly non-decreasing, that is <math>\hat{x}_0 \leq \hat{x}_1 \leq \cdots \leq \hat{x}_{n^\prime}</math>, and so are the true continuation ontimes, that is <math>x_0 \leq x_1 \leq \cdots \leq x_n</math>.
+For subtask 1, the algorithm predicts the continuations: <math>(\hat{x}_1, \hat{y}_1)</math>, <math>(\hat{x}_2, \hat{y}_2)</math>, ..., <math>(\hat{x}_{n^\prime}, \hat{y}_{n^\prime})</math>, where <math>\hat{x}_i</math> are predicted ontimes, and <math>\hat{y}_i</math> are predicted MNNs. The true continuations are notated <math>(x_1, y_1), (x_2, y_2),..., (x_n, y_n)</math>. The predicted continuation ontimes are strictly non-decreasing, that is <math>\hat{x}_0 \leq \hat{x}_1 \leq \cdots \leq \hat{x}_{n^\prime}</math>, and so are the true continuation ontimes, that is <math>x_0 \leq x_1 \leq \cdots \leq x_n</math>.
 ===Subtask 1===
 We represent each note in the true and algorithmic continuation as a point in a two-dimensional space of onset and pitch, giving the point-set <math>\mathbf{P} = \{ (x_1, y_1), (x_2, y_2),..., (x_n, y_n) \}</math> for the true continuation, and <math>\mathbf{Q} = \{ (\hat{x}_1, \hat{y}_1), (\hat{x}_2, \hat{y}_2),..., (\hat{x}_{n^\prime}, \hat{y}_{n^\prime}) \}</math> for the algorithmic continuation. We calculate differences between all points <math>p = (x_i, y_i)</math> in <math>\mathbf{P}</math> and <math>q = (\hat{x}_j, \hat{y}_j)</math> in <math>\mathbf{Q}</math>, which represent the translation vectors <math>\mathbf{T}</math> that transform a given algorithmically generated note into a note from the true continuation:
@@ Line 122: / Line 135: @@
 </math>
 ===Entropy===
 Some existing work in this area (e.g., Conklin & Witten, 1995; Pearce & Wiggins, 2006; Temperley, 2007) evaluates algorithm performance in terms of entropy. If we have time to collect human listeners' judgments of likely (or not) continuations for given excerpts, then we will be in a position to compare the entropy of listener-generated distributions with the corresponding algorithm distributions. This would open up the possibility of entropy-based metrics, but we consider this of secondary importance to the metrics outlined above.
 ==Questions (Q), Answers (A), and Comments (C)==

@@ Line 101: / Line 101: @@
 <math>
-\text{cs}(\mathbf{P},\mathbf{Q}) =  \max_{t \in \mathbf{T}} \left|\left\{q \; \forall \; q \in \mathbf{Q} \;\, | \;\, (q + t) \in \mathbf{P}\right\}\right|
+\text{CS}(\mathbf{P},\mathbf{Q}) =  \max_{t \in \mathbf{T}} \left|\left\{q \; \forall \; q \in \mathbf{Q} \;\, | \;\, (q + t) \in \mathbf{P}\right\}\right|
 </math>
 We define recall as the number of correctly predicted notes, divided by the cardinality of the true continuation point set <math>\mathbf{P}</math>. Since there exists at least one point in <math>\mathbf{Q}</math> which can be translated by any vector to a point in <math>\mathbf{P}</math>, we subtract <math>1</math> from numerator and denominator to scale to <math>[0,1]</math>.

@@ Line 91: / Line 91: @@
 The input excerpt ends with a final note event: <math>(x_0, y_0)</math>, where <math>x_0</math> is ontime (start time measured in quarter-note beats starting with 0 for bar 1 beat 1), <math>y_0</math> is pitch, represented by MNN.
-The algorithm predicts the continuations: <math>(\hat{x}_1, \hat{y}_1)</math>, <math>(\hat{x}_2, \hat{y}_2)</math>, ..., <math>(\hat{x}_{n^\prime}, \hat{y}_{n^\prime})</math>, where <math>\hat{x}_i</math> are predicted ontimes, and <math>\hat{y}_i</math> are predicted MNNs. The true continuations are notated <math>(x_1, y_1), (x_2, y_2),..., (x_n, y_n)</math>. The predicted continuation ontimes are strictly increasing, that is <math>x_0 < \hat{x}_1 < \cdots < \hat{x}_{n^\prime}</math>, and so are the true continuation ontimes, that is <math>x_0 < x_1 < \cdots < x_n</math>.
+The algorithm predicts the continuations: <math>(\hat{x}_1, \hat{y}_1)</math>, <math>(\hat{x}_2, \hat{y}_2)</math>, ..., <math>(\hat{x}_{n^\prime}, \hat{y}_{n^\prime})</math>, where <math>\hat{x}_i</math> are predicted ontimes, and <math>\hat{y}_i</math> are predicted MNNs. The true continuations are notated <math>(x_1, y_1), (x_2, y_2),..., (x_n, y_n)</math>. The predicted continuation ontimes are strictly non-decreasing, that is <math>x_0 \leq \hat{x}_1 \leq \cdots \leq \hat{x}_{n^\prime}</math>, and so are the true continuation ontimes, that is <math>x_0 \leq x_1 \leq \cdots \leq x_n</math>.
 ===Subtask 1===
-We represent each note in the true and algorithmic continuation as a point in a two-dimensional space of onset and pitch, giving the point-set <math>\mathbf{P}</math> for the true continuation, and <math>\mathbf{Q}</math> for the algorithmic continuation. We calculate differences between all points <math>p</math> in <math>\mathbf{P}</math> and <math>q</math> in <math>\mathbf{Q}</math>, which represent the translation vectors <math>\mathbf{T}</math> to transform a given algorithmically generated note into a note from the true continuation:
+We represent each note in the true and algorithmic continuation as a point in a two-dimensional space of onset and pitch, giving the point-set <math>\mathbf{P} = \{ (x_1, y_1), (x_2, y_2),..., (x_n, y_n) \}</math> for the true continuation, and <math>\mathbf{Q} = \{ (\hat{x}_1, \hat{y}_1), (\hat{x}_2, \hat{y}_2),..., (\hat{x}_{n^\prime}, \hat{y}_{n^\prime}) \}</math> for the algorithmic continuation. We calculate differences between all points <math>p = (x_i, y_i)</math> in <math>\mathbf{P}</math> and <math>q = (\hat{x}_j, \hat{y}_j)</math> in <math>\mathbf{Q}</math>, which represent the translation vectors <math>\mathbf{T}</math> that transform a given algorithmically generated note into a note from the true continuation:
 <math>\mathbf{T} = \left\{p - q \; \forall \; p \in \mathbf{P},\, q \in \mathbf{Q}\right\}</math>