2018:Patterns for Prediction Results - Revision history

Tom Collins at 09:13, 5 November 2019

2019-11-05T09:13:02Z

Tom Collins: /* Implicit Task: Polyphonic */

2019-11-05T09:11:20Z

‎Implicit Task: Polyphonic

Berit Janssen: /* Results */

2018-09-23T18:39:41Z

‎Results

Berit Janssen: /* SymPoly */

2018-09-23T18:35:22Z

‎SymPoly

Berit Janssen: /* SymPoly */

2018-09-23T18:34:59Z

‎SymPoly

Berit Janssen: /* Implicit task: decide which of two continuations is the true one, given the prime */

2018-09-23T18:34:35Z

‎Implicit task: decide which of two continuations is the true one, given the prime

Berit Janssen: /* Implicit Task: Monophonic */

2018-09-23T18:29:32Z

‎Implicit Task: Monophonic

Berit Janssen: /* Datasets and Algorithms */

2018-09-23T18:14:19Z

‎Datasets and Algorithms

Berit Janssen: /* SymMono */

2018-09-23T18:13:27Z

‎SymMono

Berit Janssen: /* Results */

2018-09-23T18:12:05Z

‎Results

@@ Line 87: / Line 87: @@
 ===SymPoly===
 Only one LSTM model was submitted to SymPoly (FC1), with results comparable to SymMono (see Figures 10-18, Table 1).
 ==Figures==

@@ Line 134: / Line 134: @@
 '''Figure 9.''' F1 measure of generated pitch-ioi pairs after cutoff point.
-===Implicit Task: Polyphonic===
+===Explicit Task: Polyphonic===
 [[File:2018_poly_R_pitch.png|600px]]

@@ Line 79: / Line 79: @@
 We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitches and inter-onset intervals (with relation to the last onset of the prime) are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2018:Patterns_for_Prediction#Evaluation_Procedure]].)
 ===SymMono===
-For the implicit task, the two LSTM based models miss more relevant pitches than the Markov Model, especially in recall of inter-onset interval (cf. Figure 4), FC1 performs better than the Markov Model. Possibly in consequence of the poorer pitch performance, few relevant pitch-ioi pairs were detected by the LSTM models.
+For the implicit task, the two LSTM based models make their best predictions close to the cut-off point (i.e., last event of the prime). As the onset time after the cut-off point increases, the Markov Model outperforms the LSTM based models, with the exception of recall of inter-onset interval (cf. Figure 4), where FC1 consistently performs better than the Markov Model. Possibly in consequence of the poorer pitch performance, also fewer relevant pitch-ioi pairs were detected by the LSTM models as onset time after cutoff point increases.
-For the explicit task, only FC1 was submitted. It outperforms chance level significantly, at 0.88, i.e., picking the correct continuation at almost 90% of the cases (See Table 1).
+For the explicit task, only FC1 was submitted. It outperforms chance level significantly, at 0.87, i.e., picking the correct continuation in almost 90% of the cases (See Table 1).
 ===SymPoly===

@@ Line 88: / Line 88: @@
 ===SymPoly===
-Only one LSTM model was submitted to SymPoly (FC1), with results comparable to SymMono (see Figures 10-18).
+Only one LSTM model was submitted to SymPoly (FC1), with results comparable to SymMono (see Figures 10-18, Table 1).
 ==Discussion==

← Older revision		Revision as of 18:34, 23 September 2018
Line 191:		Line 191:
	\|-		\|-
	\|}		\|}
−
−	~~===SymPoly===~~
−	~~[https://www.dropbox.com/s/h8b731nlu8v1re0/2017_drts_poly_compression.csv?dl=0 Click to download SymPoly pattern retrieval results table]~~
−
−	~~[https://www.dropbox.com/s/2q0rj40szi2ybjn/2017_drts_poly_patterns.csv?dl=0 Click to download SymPoly compression results table]~~

@@ Line 182: / Line 182: @@
 	! width="80" | Polyphonic
         |-
-        ! FC1 | 0.87 | 0.92
+       |-
-        ! Sig. > chance | 0.54 | 0.54
+        ! FC1
         |-
 |}

@@ Line 175: / Line 175: @@
 ==Tables==
-===Implicit Task: Monophonic===
+====Implicit task: decide which of two continuations is the true one, given the prime====
 ===SymPoly===

@@ Line 67: / Line 67: @@
 	|-
 	! FC1
-	| Algo name here  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/FC1.pdf PDF] || [https://scholar.google.com/citations?user=rpZVNKYAAAAJ&hl=en Florian Colombo]
+	| BachProp  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/FC1.pdf PDF] || [https://scholar.google.com/citations?user=rpZVNKYAAAAJ&hl=en Florian Colombo]
 	|-
          ! MM1

@@ Line 78: / Line 78: @@
 == Results ==
-An intro spiel here...
+We measure the performance of an algorithm to 1) predict a continuation, given a prime (explicit task), and 2) decide which of two versions is the true or foil continuation, given a prime (implicit task). To evaluate performance at the explicit task, we compare the true continuation to the generated continuation, and measure how many pitches and inter-onset intervals (with relation to the last onset of the prime) are correctly predicted at various time intervals after the last note of the prime. To evaluate performance at the implicit task, we measure accuracy as the number of correct decisions, divided by the total amount of decisions. (For mathematical definitions of the various metrics, please see [[2018:Patterns_for_Prediction#Evaluation_Procedure]].)
-(For mathematical definitions of the various metrics, please see [[2018:Patterns_for_Prediction#Evaluation_Procedure]].)
+For the explicit tasks of SymMono and SymPoly, the two submitted algorithms overall perform worse than the first-order Markov Model introduced by the task captains for comparison purposes. However, there are some cases where the LSTMs outperform the Markov Model (see below). For the implicit task of SymMono and SymPoly, FC1 outperforms chance level significantly, and by a wide margin.
 ===SymMono===
-Here are some results (cf. Figures 1-3), and some interpretation. Don't forget these as well (Figures 4-6), showing something.
+For the implicit task, the two LSTM based models miss more relevant pitches than the Markov Model, especially in recall of inter-onset interval (cf. Figure 2), they perform close to or better than the Markov Model. Possibly in consequence of the poorer pitch performance, few relevant pitch-ioi pairs were detected by the LSTM models.
-Remarks on runtime appropriate here too.
+For the explicit task, only FC1 was submitted. It outperforms chance level significantly, at 0.88, i.e., picking the correct continuation at almost 90% of the cases.
 ===SymPoly===
-And so on.
+Only one LSTM model was submitted to SymPoly (FC1), with results comparable to SymMono (see Figures 10-18).
 ==Discussion==