2019:Patterns for Prediction Results - Revision history

Tom Collins: /* Discussion */

2019-11-05T13:40:16Z

‎Discussion

Tom Collins: /* Discussion */

2019-11-05T13:39:57Z

‎Discussion

Tom Collins: /* Discussion */

2019-11-05T13:38:27Z

‎Discussion

Tom Collins: /* Discussion */

2019-11-05T13:37:12Z

‎Discussion

Tom Collins at 13:13, 5 November 2019

2019-11-05T13:13:36Z

Tom Collins at 09:14, 5 November 2019

2019-11-05T09:14:43Z

Berit Janssen: /* Implicit task: discriminate true and foil continuation */

2019-11-02T16:50:47Z

‎Implicit task: discriminate true and foil continuation

Berit Janssen: /* Datasets and Algorithms */

2019-10-31T20:21:08Z

‎Datasets and Algorithms

Berit Janssen: /* Datasets and Algorithms */

2019-10-31T20:20:38Z

‎Datasets and Algorithms

Berit Janssen: /* Datasets and Algorithms */

2019-10-31T20:16:27Z

‎Datasets and Algorithms

@@ Line 281: / Line 281: @@
 == Discussion ==
-CopyForward (a method based on geometric pattern discovery) significantly outperforms the baseline Markov model and BachProp (a method based on recurrent neural networks) on the explicit subtask. The abstract for CopyForward suggests this relatively high level of performance is due to successful modeling of dynamic expectancies – that is, it uses information from the prime exclusively (no training data) to predict the content of the continuation. BachProp, on the other hand, attempts to model schematic (or corpus-based) expectancies too, combining information from the prime with information from a training process. It remains to be seen whether BachProp (or deep-learning approaches in general) could improve upon CopyForward's continuations when the latter's dependence on modeling of dynamic expectancies leads unsuccessful predictions.
+CopyForward (a method based on geometric pattern discovery) significantly outperforms the baseline Markov model and BachProp (a method based on recurrent neural networks) on the explicit subtask. The abstract for CopyForward suggests this relatively high level of performance is due to successful modeling of dynamic expectancies – that is, it uses information from the prime exclusively (no training data) to predict the content of the continuation. BachProp, on the other hand, attempts to model schematic (or corpus-based) expectancies too, combining information from the prime with information from a training process. It remains to be seen whether BachProp (or deep-learning approaches in general) could improve upon CopyForward's continuations when the latter's dependence on modeling of dynamic expectancies leads to unsuccessful predictions.
 All algorithms submitted this year to the implicit subtask achieved significantly-above-chance performance. MLM employed an LSTM and GenDetect employed a gradient boosting classifier. GenDetect performed significantly better than previous and current submissions on this task, discriminating between all monophonic true and foil continuations correctly, and all but one polyphonic true and foil continuations. If we run this subtask again next year, we probably ought to identify a different, more sophisticated algorithm for generating the foils.

@@ Line 281: / Line 281: @@
 == Discussion ==
-CopyForward (a method based on geometric pattern discovery) significantly outperforms the baseline Markov model and BachProp (a method based on recurrent neural networks) on the explicit subtask. The abstract for CopyForward suggests this relatively high level of performance is due to successful modeling of dynamic expectancies – that is, it uses information from the prime exclusively (no training data) to predict the content of the continuation. BachProp, on the other hand, attempts to model schematic (or corpus-based) expectancies too, combining information from the prime with information from a training process. It remains to be seen whether BachProp (or deep-learning approaches in general) could improve upon CopyForward's continuations when its dependence on modeling of dynamic expectancies leads unsuccessful predictions.
+CopyForward (a method based on geometric pattern discovery) significantly outperforms the baseline Markov model and BachProp (a method based on recurrent neural networks) on the explicit subtask. The abstract for CopyForward suggests this relatively high level of performance is due to successful modeling of dynamic expectancies – that is, it uses information from the prime exclusively (no training data) to predict the content of the continuation. BachProp, on the other hand, attempts to model schematic (or corpus-based) expectancies too, combining information from the prime with information from a training process. It remains to be seen whether BachProp (or deep-learning approaches in general) could improve upon CopyForward's continuations when the latter's dependence on modeling of dynamic expectancies leads unsuccessful predictions.
 All algorithms submitted this year to the implicit subtask achieved significantly-above-chance performance. MLM employed an LSTM and GenDetect employed a gradient boosting classifier. GenDetect performed significantly better than previous and current submissions on this task, discriminating between all monophonic true and foil continuations correctly, and all but one polyphonic true and foil continuations. If we run this subtask again next year, we probably ought to identify a different, more sophisticated algorithm for generating the foils.

@@ Line 281: / Line 281: @@
 == Discussion ==
-CopyForward (a method based on geometric pattern discovery) significantly outperforms the baseline Markov model and BachProp (a method based on recurrent neural networks) on the explicit subtask. Reading the abstract for CopyForward, it seems this relatively high level of performance is due to successful modeling of dynamic expectancies – that is, it uses information from the prime exclusively (not any training data) to predict the content of the continuation. BachProp, on the other hand, attempts to model schematic (or corpus-based) expectancies too, combining information from the prime with information from a training process. It remains to be seen whether BachProp (or deep-learning approaches in general) could improve upon CopyForward's continuations when its dependence on modeling of dynamic expectancies leads unsuccessful predictions.
+CopyForward (a method based on geometric pattern discovery) significantly outperforms the baseline Markov model and BachProp (a method based on recurrent neural networks) on the explicit subtask. The abstract for CopyForward suggests this relatively high level of performance is due to successful modeling of dynamic expectancies – that is, it uses information from the prime exclusively (no training data) to predict the content of the continuation. BachProp, on the other hand, attempts to model schematic (or corpus-based) expectancies too, combining information from the prime with information from a training process. It remains to be seen whether BachProp (or deep-learning approaches in general) could improve upon CopyForward's continuations when its dependence on modeling of dynamic expectancies leads unsuccessful predictions.
 All algorithms submitted this year to the implicit subtask achieved significantly-above-chance performance. MLM employed an LSTM and GenDetect employed a gradient boosting classifier. GenDetect performed significantly better than previous and current submissions on this task, discriminating between all monophonic true and foil continuations correctly, and all but one polyphonic true and foil continuations. If we run this subtask again next year, we probably ought to identify a different, more sophisticated algorithm for generating the foils.

← Older revision		Revision as of 13:37, 5 November 2019
Line 282:		Line 282:

	CopyForward (a method based on geometric pattern discovery) significantly outperforms the baseline Markov model and BachProp (a method based on recurrent neural networks) on the explicit subtask. Reading the abstract for CopyForward, it seems this relatively high level of performance is due to successful modeling of dynamic expectancies – that is, it uses information from the prime exclusively (not any training data) to predict the content of the continuation. BachProp, on the other hand, attempts to model schematic (or corpus-based) expectancies too, combining information from the prime with information from a training process. It remains to be seen whether BachProp (or deep-learning approaches in general) could improve upon CopyForward's continuations when its dependence on modeling of dynamic expectancies leads unsuccessful predictions.		CopyForward (a method based on geometric pattern discovery) significantly outperforms the baseline Markov model and BachProp (a method based on recurrent neural networks) on the explicit subtask. Reading the abstract for CopyForward, it seems this relatively high level of performance is due to successful modeling of dynamic expectancies – that is, it uses information from the prime exclusively (not any training data) to predict the content of the continuation. BachProp, on the other hand, attempts to model schematic (or corpus-based) expectancies too, combining information from the prime with information from a training process. It remains to be seen whether BachProp (or deep-learning approaches in general) could improve upon CopyForward's continuations when its dependence on modeling of dynamic expectancies leads unsuccessful predictions.
		+
		+	All algorithms submitted this year to the implicit subtask achieved significantly-above-chance performance. MLM employed an LSTM and GenDetect employed a gradient boosting classifier. GenDetect performed significantly better than previous and current submissions on this task, discriminating between all monophonic true and foil continuations correctly, and all but one polyphonic true and foil continuations. If we run this subtask again next year, we probably ought to identify a different, more sophisticated algorithm for generating the foils.

← Older revision		Revision as of 13:13, 5 November 2019
Line 280:		Line 280:

	== Discussion ==		== Discussion ==
		+
		+	CopyForward (a method based on geometric pattern discovery) significantly outperforms the baseline Markov model and BachProp (a method based on recurrent neural networks) on the explicit subtask. Reading the abstract for CopyForward, it seems this relatively high level of performance is due to successful modeling of dynamic expectancies – that is, it uses information from the prime exclusively (not any training data) to predict the content of the continuation. BachProp, on the other hand, attempts to model schematic (or corpus-based) expectancies too, combining information from the prime with information from a training process. It remains to be seen whether BachProp (or deep-learning approaches in general) could improve upon CopyForward's continuations when its dependence on modeling of dynamic expectancies leads unsuccessful predictions.

← Older revision		Revision as of 09:14, 5 November 2019
Line 278:		Line 278:

	'''Table 4.''' Discrimination scores of the submitted algorithms.		'''Table 4.''' Discrimination scores of the submitted algorithms.
		+
		+	== Discussion ==

@@ Line 214: / Line 214: @@
        <th></th>
        <th></th>
-       <th>nr_obs</th>
+       <th>Observations</th>
-       <th>accuracy</th>
+       <th>Accuracy</th>
-       <th>crossentropy</th>
+       <th>Mean Probability</th>
-       <th>nll_var</th>
+       <th>Variance</th>
      </tr>
      <tr>
@@ Line 232: / Line 232: @@
        <td>500</td>
        <td>1.000</td>
-       <td>0.313</td>
+       <td>1.000</td>
        <td>0.000</td>
      </tr>
@@ Line 239: / Line 239: @@
        <th>mono</th>
        <td>499</td>
-       <td>0.864</td>
+       <td>0.844</td>
-       <td>0.693</td>
+       <td>0.498</td>
-       <td>0.000</td>
+       <td>0.006</td>
      </tr>
      <tr>
@@ Line 248: / Line 248: @@
        <td>500</td>
        <td>0.998</td>
-       <td>0.318</td>
+       <td>0.992</td>
-       <td>0.001</td>
+       <td>0.002</td>
      </tr>
      <tr>
@@ Line 256: / Line 256: @@
        <td>499</td>
        <td>0.916</td>
-       <td>0.682</td>
+       <td>0.519</td>
-       <td>0.001</td>
+       <td>0.006</td>
      </tr>
      <tr>
-       <th>MLM</th>
+       <th>MLM(2)</th>
        <th>poly</th>
        <td>499</td>
        <td>0.703</td>
-       <td>0.668</td>
+       <td>0.527</td>
-       <td>0.003</td>
+       <td>0.004</td>
      </tr>
      <tr>
-       <th>MLM</th>
+       <th>MLM(5)</th>
        <th>poly</th>
        <td>499</td>
        <td>0.731</td>
-       <td>0.646</td>
+       <td>0.554</td>
-       <td>0.010</td>
+       <td>0.012</td>
      </tr>
 </table>
 '''Table 4.''' Discrimination scores of the submitted algorithms.

@@ Line 82: / Line 82: @@
 	|-
          ! YB2
-	| MLM  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart] [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]
+	| MLM  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart], [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]
 	|-
          ! YB5
-         | MLM  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart] [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]
+         | MLM  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart], [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]
 |}

@@ Line 70: / Line 70: @@
 	|-
          ! EP1
-	| GenDetect  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
+	| GenDetect  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
 	|-
          |- style="background: green;"
@@ Line 79: / Line 79: @@
 	|-
          ! EP1
-	| GenDetect  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
+	| GenDetect  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [http://metacreation.net/members/jeff-ens/ Jeff Ens], [http://philippepasquier.com/publications Philippe Pasquier]
 	|-
          ! YB2

@@ Line 52: / Line 52: @@
 	|-
          ! TD1
-	| CopyForward  ||  style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
+	| CopyForward  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf PDF] || [https://mcgill.ca/music/timothy-de-reuse Timothy de Reuse]
 	|-
          |- style="background: green;"
@@ Line 61: / Line 61: @@
 	|-
          ! TD1
-	| CopyForward  ||  style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Timothy de Reuse]
+	| CopyForward  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/TD1.pdf PDF] || [https://mcgill.ca/music/timothy-de-reuse Timothy de Reuse]
 	|-
          |- style="background: green;"
@@ Line 70: / Line 70: @@
 	|-
          ! EP1
-	| GenDetect  ||  style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
+	| GenDetect  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
 	|-
          |- style="background: green;"
@@ Line 79: / Line 79: @@
 	|-
          ! EP1
-	| GenDetect  ||  style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
+	| GenDetect  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/EP1.pdf PDF] || [author.link.missing Jeff Ens, Philippe Pasquier]
 	|-
          ! YB2
-	| MLM  ||  style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
+	| MLM  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart] [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]
 	|-
          ! YB5
-         | MLM  ||  style="text-align: center;" | [no.link.yet PDF] || [author.link.missing Adrien Ycart, Emmanouil Benetos]
+         | MLM  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2019/YB2.pdf PDF] || [http://www.eecs.qmul.ac.uk/~ay304/ Adrien Ycart] [http://www.eecs.qmul.ac.uk/profiles/benetosemmanouil.html Emmanouil Benetos]
 |}