Difference between revisions of "2018:Drum Transcription Results"

From MIREX Wiki
m
m
Line 33: Line 33:
 
| RV1-3
 
| RV1-3
 
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/RV1.pdf PDF]
 
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/RV1.pdf PDF]
−
| [http://ifs.tuwien.ac.at/user/426 Richard Vogl]
+
| [http://ifs.tuwien.ac.at/~vogl/ Richard Vogl]
 
|}
 
|}
  

Revision as of 09:32, 20 September 2018

Introduction

The drum transcription task was reintroduced last year after it's first edition in 2005. Two out of the three original datasets used in 2005 were available and have been used for evaluation also this year. For those datasets the results from 2005 may be compared to this years results. Additionally to the two datasets from 2005, three new datasets were used in the evaluation. For training the algorithms, the public training set from 2005 plus additional training data taken from the new datasets was provided to the participants. For these datasets only the three main drum instruments (kick drum, snare drum, and hi-hat) are considered.

This year additionally to the 3 instrument class task, a second sub task, an 8 instrument class transcription task, was introduced. To this new evaluation and training datasets were used this year.


Submissions

Abstract Contributors
AR5, JAR1-3 PDF Celine Jacques, Achille Aknin, Axel Roebel
CS2,4 PDF Carl Southall
JS1 PDF Julien Schroeter
RV1-3 PDF Richard Vogl


3 Class Results

The overall results represent the mean values over all datasets.

Overall

Algorithm mean fm sum fm KD mean fm KD sum fm SD mean fm SD sum fm HH mean fm HH sum fm
JS2 0.62 0.64 0.78 0.78 0.63 0.63 0.40 0.40
JAR2 0.68 0.73 0.78 0.78 0.65 0.65 0.53 0.53
JS3 0.61 0.62 0.76 0.76 0.63 0.63 0.36 0.36
JS4 0.57 0.58 0.73 0.73 0.61 0.61 0.32 0.32
RV2 0.65 0.72 0.75 0.75 0.65 0.65 0.50 0.50
RV1 0.69 0.74 0.78 0.78 0.69 0.69 0.53 0.53
JAR1 0.69 0.72 0.78 0.78 0.65 0.65 0.53 0.53
JAR3 0.69 0.73 0.79 0.79 0.65 0.65 0.54 0.54
JS1 0.61 0.62 0.75 0.75 0.59 0.59 0.43 0.43
JAR5 0.67 0.71 0.77 0.77 0.63 0.63 0.53 0.53
CS2 0.63 0.70 0.78 0.78 0.59 0.59 0.50 0.50
CS4 0.62 0.68 0.77 0.77 0.58 0.58 0.47 0.47

download these results as csv

2005 baseline: 0.670 (YGO)

The best overall result from 2005 is only provided to put the current results into perspective. Since the overall result form 2005 was calculated on different datasets it is problematic to compare them directly.

IDMT subset

Algorithm mean fm sum fm KD mean fm KD sum fm SD mean fm SD sum fm HH mean fm HH sum fm
JS2 0.59 0.64 0.72 0.72 0.64 0.64 0.44 0.44
JAR2 0.63 0.68 0.73 0.73 0.59 0.59 0.53 0.53
JS3 0.57 0.63 0.71 0.71 0.67 0.67 0.38 0.38
JS4 0.55 0.61 0.69 0.69 0.69 0.69 0.32 0.32
RV2 0.66 0.73 0.74 0.74 0.71 0.71 0.54 0.54
RV1 0.66 0.72 0.75 0.75 0.72 0.72 0.53 0.53
JAR1 0.63 0.69 0.73 0.73 0.68 0.68 0.49 0.49
JAR3 0.64 0.69 0.74 0.74 0.60 0.60 0.54 0.54
JS1 0.58 0.62 0.69 0.69 0.56 0.56 0.48 0.48
JAR5 0.62 0.67 0.72 0.72 0.56 0.56 0.54 0.54
CS2 0.58 0.63 0.67 0.67 0.58 0.58 0.49 0.49
CS4 0.53 0.57 0.63 0.63 0.50 0.50 0.44 0.44

download these results as csv

2005 baseline: 0.753 (CD)

KT subset

Algorithm mean fm sum fm KD mean fm KD sum fm SD mean fm SD sum fm HH mean fm HH sum fm
JS2 0.63 0.65 0.77 0.77 0.68 0.68 0.40 0.40
JAR2 0.64 0.65 0.77 0.77 0.67 0.67 0.46 0.46
JS3 0.62 0.64 0.76 0.76 0.69 0.69 0.36 0.36
JS4 0.59 0.60 0.72 0.72 0.68 0.68 0.33 0.33
RV2 0.63 0.67 0.76 0.76 0.68 0.68 0.45 0.45
RV1 0.65 0.68 0.80 0.80 0.68 0.68 0.47 0.47
JAR1 0.64 0.65 0.75 0.75 0.67 0.67 0.45 0.45
JAR3 0.65 0.66 0.78 0.78 0.68 0.68 0.47 0.47
JS1 0.61 0.62 0.73 0.73 0.59 0.59 0.43 0.43
JAR5 0.62 0.63 0.74 0.74 0.64 0.64 0.46 0.46
CS2 0.60 0.64 0.76 0.76 0.58 0.58 0.44 0.44
CS4 0.56 0.60 0.72 0.72 0.57 0.57 0.39 0.39

download these results as csv

2005 baseline: 0.617 (YGO)

RBMA subset

Algorithm mean fm sum fm KD mean fm KD sum fm SD mean fm SD sum fm HH mean fm HH sum fm
JS2 0.57 0.60 0.80 0.80 0.47 0.47 0.39 0.39
JAR2 0.70 0.73 0.87 0.87 0.55 0.55 0.57 0.57
JS3 0.55 0.58 0.79 0.79 0.48 0.48 0.33 0.33
JS4 0.50 0.53 0.76 0.76 0.46 0.46 0.27 0.27
RV2 0.70 0.74 0.91 0.91 0.60 0.60 0.55 0.55
RV1 0.72 0.74 0.91 0.91 0.62 0.62 0.56 0.56
JAR1 0.68 0.72 0.87 0.87 0.45 0.45 0.56 0.56
JAR3 0.69 0.73 0.87 0.87 0.56 0.56 0.57 0.57
JS1 0.56 0.60 0.76 0.76 0.43 0.43 0.43 0.43
JAR5 0.67 0.71 0.83 0.83 0.53 0.53 0.56 0.56
CS2 0.67 0.71 0.87 0.87 0.55 0.55 0.56 0.56
CS4 0.66 0.70 0.85 0.85 0.50 0.50 0.56 0.56

download these results as csv

MDB subset

Algorithm mean fm sum fm KD mean fm KD sum fm SD mean fm SD sum fm HH mean fm HH sum fm
JS2 0.59 0.58 0.73 0.73 0.58 0.58 0.43 0.43
JAR2 0.65 0.65 0.64 0.64 0.62 0.62 0.59 0.59
JS3 0.54 0.54 0.68 0.68 0.51 0.51 0.39 0.39
JS4 0.48 0.48 0.62 0.62 0.44 0.44 0.33 0.33
RV2 0.55 0.55 0.44 0.44 0.54 0.54 0.58 0.58
RV1 0.64 0.65 0.55 0.55 0.60 0.60 0.57 0.57
JAR1 0.68 0.66 0.66 0.66 0.61 0.61 0.60 0.60
JAR3 0.67 0.67 0.69 0.69 0.64 0.64 0.59 0.59
JS1 0.59 0.57 0.72 0.72 0.60 0.60 0.46 0.46
JAR5 0.66 0.66 0.68 0.68 0.63 0.63 0.58 0.58
CS2 0.63 0.61 0.69 0.69 0.48 0.48 0.63 0.63
CS4 0.65 0.63 0.75 0.75 0.56 0.56 0.59 0.59

download these results as csv

MDB-Drums [1]

GEN subset

Algorithm mean fm sum fm KD mean fm KD sum fm SD mean fm SD sum fm HH mean fm HH sum fm
JS2-3 0.73 0.75 0.87 0.87 0.78 0.78 0.33 0.33
JAR2 0.79 0.81 0.87 0.87 0.80 0.80 0.52 0.52
JS3-3 0.75 0.77 0.88 0.88 0.79 0.79 0.36 0.36
JS4-3 0.75 0.77 0.87 0.87 0.79 0.79 0.36 0.36
RV2 0.70 0.75 0.89 0.89 0.75 0.75 0.38 0.38
RV1 0.78 0.81 0.91 0.91 0.80 0.80 0.50 0.50
JAR1 0.81 0.83 0.87 0.87 0.83 0.83 0.55 0.55
JAR3 0.78 0.81 0.88 0.88 0.79 0.79 0.51 0.51
JS1-3 0.70 0.72 0.84 0.84 0.76 0.76 0.34 0.34
JAR5 0.76 0.78 0.86 0.86 0.77 0.77 0.51 0.51
CS2-3 0.68 0.75 0.92 0.92 0.75 0.75 0.38 0.38
CS4-3 0.68 0.75 0.92 0.92 0.76 0.76 0.38 0.38

download these results as csv

8 Class Results

The overall results represent the mean values over all datasets.

Overall

Algorithm mean fm sum fm BD mean fm BD sum fm SD mean fm SD sum fm TT mean fm TT sum fm HH mean fm HH sum fm CY mean fm CY sum fm RD mean fm RD sum fm CB mean fm CB sum fm CL mean fm CL sum fm
JS4 0.54 0.59 0.73 0.73 0.48 0.48 0.29 0.29 0.46 0.46 0.39 0.39 0.50 0.50 0.82 0.82 0.90 0.90
JS2 0.58 0.61 0.79 0.79 0.52 0.52 0.19 0.19 0.52 0.52 0.32 0.32 0.41 0.41 0.80 0.80 0.90 0.90
JS3 0.57 0.62 0.77 0.77 0.51 0.51 0.23 0.23 0.50 0.50 0.41 0.41 0.47 0.47 0.82 0.82 0.90 0.90
JS1 0.55 0.57 0.75 0.75 0.49 0.49 0.15 0.15 0.52 0.52 0.20 0.20 0.20 0.20 0.68 0.68 0.90 0.90
CS4 0.50 0.56 0.79 0.79 0.51 0.51 0.14 0.14 0.62 0.62 0.20 0.20 0.13 0.13 0.02 0.02 0.02 0.02
CS2 0.47 0.54 0.78 0.78 0.50 0.50 0.15 0.15 0.60 0.60 0.19 0.19 0.14 0.14 0.02 0.02 0.02 0.02
RV3 0.62 0.68 0.78 0.78 0.52 0.52 0.37 0.37 0.58 0.58 0.38 0.38 0.65 0.65 0.82 0.82 0.71 0.71

download these results as csv


RBMA subset

Algorithm mean fm sum fm BD mean fm BD sum fm SD mean fm SD sum fm TT mean fm TT sum fm HH mean fm HH sum fm CY mean fm CY sum fm RD mean fm RD sum fm CB mean fm CB sum fm CL mean fm CL sum fm
JS4 0.47 0.50 0.71 0.71 0.28 0.28 0.15 0.15 0.43 0.43 0.61 0.61 0.73 0.73 0.82 0.82 0.76 0.76
JS2 0.50 0.54 0.80 0.80 0.28 0.28 0.08 0.08 0.48 0.48 0.33 0.33 0.55 0.55 0.79 0.79 0.76 0.76
JS3 0.50 0.53 0.77 0.77 0.28 0.28 0.10 0.10 0.47 0.47 0.61 0.61 0.67 0.67 0.82 0.82 0.76 0.76
JS1 0.47 0.51 0.77 0.77 0.25 0.25 0.08 0.08 0.48 0.48 0.10 0.10 0.16 0.16 0.73 0.73 0.76 0.76
CS4 0.44 0.48 0.85 0.85 0.34 0.34 0.14 0.14 0.57 0.57 0.02 0.02 0.03 0.03 0.02 0.02 0.05 0.05
CS2 0.40 0.43 0.86 0.86 0.34 0.34 0.13 0.13 0.52 0.52 0.02 0.02 0.04 0.04 0.02 0.02 0.05 0.05
RV3 0.55 0.58 0.85 0.85 0.27 0.27 0.13 0.13 0.48 0.48 0.64 0.64 0.79 0.79 0.82 0.82 0.52 0.52

download these results as csv

MDB subset

Algorithm mean fm sum fm BD mean fm BD sum fm SD mean fm SD sum fm TT mean fm TT sum fm HH mean fm HH sum fm CY mean fm CY sum fm RD mean fm RD sum fm CB mean fm CB sum fm CL mean fm CL sum fm
JS4 0.56 0.53 0.76 0.76 0.52 0.52 0.45 0.45 0.42 0.42 0.33 0.33 0.47 0.47 0.91 0.91 1.00 1.00
JS2 0.61 0.58 0.81 0.81 0.65 0.65 0.27 0.27 0.49 0.49 0.39 0.39 0.49 0.49 0.91 0.91 1.00 1.00
JS3 0.60 0.57 0.80 0.80 0.61 0.61 0.36 0.36 0.47 0.47 0.37 0.37 0.49 0.49 0.91 0.91 1.00 1.00
JS1 0.59 0.56 0.77 0.77 0.67 0.67 0.19 0.19 0.47 0.47 0.24 0.24 0.34 0.34 0.91 0.91 1.00 1.00
CS4 0.47 0.48 0.76 0.76 0.51 0.51 0.09 0.09 0.61 0.61 0.25 0.25 0.15 0.15 0.00 0.00 0.00 0.00
CS2 0.44 0.46 0.71 0.71 0.51 0.51 0.10 0.10 0.60 0.60 0.20 0.20 0.14 0.14 0.00 0.00 0.00 0.00
RV3 0.65 0.60 0.72 0.72 0.63 0.63 0.60 0.60 0.60 0.60 0.27 0.27 0.66 0.66 0.91 0.91 0.73 0.73

download these results as csv

MDB-Drums [2]

MIDI subset

Algorithm mean fm sum fm BD mean fm BD sum fm SD mean fm SD sum fm TT mean fm TT sum fm HH mean fm HH sum fm CY mean fm CY sum fm RD mean fm RD sum fm CB mean fm CB sum fm CL mean fm CL sum fm
JS4 0.58 0.66 0.71 0.71 0.63 0.63 0.26 0.26 0.52 0.52 0.23 0.23 0.30 0.30 0.74 0.74 0.94 0.94
JS2 0.61 0.67 0.75 0.75 0.62 0.62 0.23 0.23 0.58 0.58 0.23 0.23 0.19 0.19 0.70 0.70 0.94 0.94
JS3 0.61 0.67 0.74 0.74 0.64 0.64 0.22 0.22 0.56 0.56 0.24 0.24 0.25 0.25 0.72 0.72 0.94 0.94
JS1 0.57 0.61 0.71 0.71 0.55 0.55 0.18 0.18 0.60 0.60 0.25 0.25 0.12 0.12 0.40 0.40 0.94 0.94
CS4 0.59 0.63 0.76 0.76 0.67 0.67 0.19 0.19 0.67 0.67 0.32 0.32 0.20 0.20 0.05 0.05 0.02 0.02
CS2 0.58 0.63 0.76 0.76 0.63 0.63 0.23 0.23 0.68 0.68 0.33 0.33 0.22 0.22 0.04 0.04 0.02 0.02
RV3 0.66 0.75 0.78 0.78 0.66 0.66 0.37 0.37 0.66 0.66 0.22 0.22 0.52 0.52 0.74 0.74 0.90 0.90

download these results as csv

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2018:Drum_Transcription_Results&oldid=12816"