2020:Patterns for Prediction Results - Revision history

Berit Janssen: /* Discussion */

2020-10-20T14:37:50Z

‎Discussion

Berit Janssen: /* Discussion */

2020-10-20T14:33:05Z

‎Discussion

Berit Janssen: /* Implicit task: discriminate true and foil continuation */

2020-10-13T08:09:56Z

‎Implicit task: discriminate true and foil continuation

Berit Janssen: /* Datasets and Algorithms */

2020-10-13T08:09:37Z

‎Datasets and Algorithms

Berit Janssen: /* Contribution */

2020-10-09T12:02:20Z

‎Contribution

Berit Janssen: /* Discussion */

2020-10-09T12:00:26Z

‎Discussion

Berit Janssen: /* Implicit task: discriminate true and foil continuation */

2020-10-09T11:59:56Z

‎Implicit task: discriminate true and foil continuation

Berit Janssen: /* Discussion */

2020-10-09T11:59:04Z

‎Discussion

Berit Janssen: /* Implicit task: discriminate true and foil continuation */

2020-10-09T11:53:03Z

‎Implicit task: discriminate true and foil continuation

Berit Janssen: /* Datasets and Algorithms */

2020-10-09T11:48:33Z

‎Datasets and Algorithms

← Older revision		Revision as of 14:37, 20 October 2020
Line 122:		Line 122:

	== Discussion ==		== Discussion ==
−	This year's submission, ~~PDMTransform~~, is based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features. It has higher accuracy for the distinction of true and foil continuations than GenDetect, last year's highest accuracy submission. It outputs similar probabilities for the CopyForward foils and true continuations (scoring, e.g., 0.96 probability for the foils and 0.97 to the true continuation), which explains the lower mean probability (resulting from normalization of the two probability scores) of ~~PDMTransform~~ as compared to GenDetect.	+	This year's submission, PDMTransformer, is based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features. It has higher accuracy for the distinction of true and foil continuations than GenDetect, last year's highest accuracy submission. It outputs similar probabilities for the CopyForward foils and true continuations (scoring, e.g., 0.96 probability for the foils and 0.97 to the true continuation), which explains the lower mean probability (resulting from normalization of the two probability scores) of PDMTransformer as compared to GenDetect.

← Older revision		Revision as of 14:33, 20 October 2020
Line 122:		Line 122:

	== Discussion ==		== Discussion ==
−	This year's submission, based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features, has higher accuracy for the distinction of true and foil continuations. ~~However~~, the ~~confidence of~~ the ~~model to prefer one~~ continuation ~~over another~~ (~~mean~~ probability) ~~is rather low~~.	+	This year's submission, PDMTransform, is based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features. It has higher accuracy for the distinction of true and foil continuations than GenDetect, last year's highest accuracy submission. It outputs similar probabilities for the CopyForward foils and true continuations (scoring, e.g., 0.96 probability for the foils and 0.97 to the true continuation), which explains the lower mean probability (resulting from normalization of the two probability scores) of PDMTransform as compared to GenDetect.

@@ Line 102: / Line 102: @@
      </tr>
      <tr>
-       <th>YH1</th>
+       <th>YH16</th>
        <th>Markov</th>
        <td>500.0</td>
@@ Line 110: / Line 110: @@
      </tr>
      <tr>
-       <th>YH1</th>
+       <th>YH16</th>
        <th>CopyForward</th>
        <td>500.0</td>

@@ Line 53: / Line 53: @@
 	|-
          ! YH16
-         | XGBoost || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || Vane Wu, YuanLiang Dong, Yu Hong (Tencent Music)
+         | PDMTransformer || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2020/YH16.pdf PDF] || Vane Wu, YuanLiang Dong, Yu Hong (Tencent Music)
 |}

← Older revision		Revision as of 12:02, 9 October 2020
Line 16:		Line 16:

	== Contribution ==		== Contribution ==
−
−	~~...~~

	For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].		For a more detailed introduction to the task, please see [[2020:Patterns for Prediction]].

← Older revision		Revision as of 12:00, 9 October 2020
Line 124:		Line 124:

	== Discussion ==		== Discussion ==
−	This year's submission, based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features, has higher accuracy for the distinction of true and foil continuations. However, the confidence of the model to ~~give~~ one ~~choice~~ over another (mean probability) is rather low.	+	This year's submission, based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features, has higher accuracy for the distinction of true and foil continuations. However, the confidence of the model to prefer one continuation over another (mean probability) is rather low.

@@ Line 77: / Line 77: @@
        <th>Accuracy</th>
        <th>Mean Probability</th>
-       <th>Variance</th>
+       <th>Variance of Probability</th>
      </tr>
      <tr>

← Older revision		Revision as of 11:59, 9 October 2020
Line 124:		Line 124:

	== Discussion ==		== Discussion ==
−		+	This year's submission, based on a Music Transformer encoding of musical events, and Xtreme Gradient Boost of various classifiers trained on the resulting features, has higher accuracy for the distinction of true and foil continuations. However, the confidence of the model to give one choice over another (mean probability) is rather low.
−	~~Our discussion~~

@@ Line 71: / Line 71: @@
 ====Implicit task: discriminate true and foil continuation====
 <table border="1" class="dataframe">
      <tr style="text-align: right;">
        <th></th>
        <th></th>
-       <th>nr_obs</th>
+       <th>Observations</th>
-       <th>accuracy</th>
+       <th>Accuracy</th>
-       <th>mean_probability</th>
+       <th>Mean Probability</th>
-       <th>var_prob</th>
+       <th>Variance</th>
      </tr>
      <tr>
-       <th>model</th>
+       <th>Model</th>
-       <th>data</th>
+       <th>Foil</th>
        <th></th>
        <th></th>
@@ Line 88: / Line 87: @@
        <th></th>
      </tr>
      <tr>
-       <th>EP1_Markov</th>
+       <th>EP1</th>
-       <th>mono</th>
+       <th>Markov</th>
        <td>500.0</td>
        <td>1.000</td>
@@ Line 99: / Line 96: @@
      </tr>
      <tr>
-       <th>EP1_CopyForward</th>
+       <th>EP1</th>
-       <th>mono</th>
+       <th>CopyForward</th>
        <td>500.0</td>
        <td>0.774</td>
@@ Line 107: / Line 104: @@
      </tr>
      <tr>
-       <th>YH1_Markov</th>
+       <th>YH1</th>
-       <th>mono</th>
+       <th>Markov</th>
        <td>500.0</td>
        <td>0.996</td>
@@ Line 115: / Line 112: @@
      </tr>
      <tr>
-       <th>YH1_CopyForward</th>
+       <th>YH1</th>
-       <th>mono</th>
+       <th>CopyForward</th>
        <td>500.0</td>
        <td>0.916</td>
@@ Line 122: / Line 119: @@
        <td>0.005</td>
      </tr>
 </table>

@@ Line 36: / Line 36: @@
 There are small (100 pieces), medium (1,000 pieces), and large (10,000 pieces) variants of each dataset, to cater to different approaches to the task (e.g., a point-set pattern discovery algorithm developer may not want/need as many training examples as a neural network researcher). Each prime lasts approximately 35 sec (according to the BPM value in the original MIDI file) and each continuation covers the subsequent 10 quarter-note beats. We would have liked to provide longer primes (as 35 sec affords investigation of medium- but not really long-term structure), but we have to strike a compromise between ideal and tractable scenarios.
-Submissions to the symMono implicit task are listed in Table 1. There were no submissions to the symPoly, audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. This enabled evaluation of the implicit subtask.
+Submissions to the symMono implicit task are listed in Table 1. There were no submissions to the symPoly, audMono or audPoly variants of the tasks this year. The task captains prepared a first-order Markov model (MM) over a state space of measure beat and key-centralized MIDI note number. As in previous years, this served as a foil for the implicit subtask. As one of last year's submissions (GenDetect (EP1)) already reached almost-perfect accuracy on these foils, we also evaluate how well the submitted models can discriminate the true continuation from a foil generated by last year's best-performing music prediction model, CopyForward (see [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf abstract]).
 {| border="1" cellspacing="0" style="text-align: left; width: 800px;"