Difference between revisions of "2017:Drum Transcription Results"

From MIREX Wiki
(→‎MDB subset)
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
== Introduction ==
 
== Introduction ==
  
 +
The drum transcription task was reintroduced this year after it's first edition in 2005.
 +
Two out of the three datasets used in 2005 were available and have been used for evaluation also this year.
 +
For those datasets the results from 2005 may be compared to this years results.
  
−
== What's new ==
+
As in 2005 only the three main drum instruments (kick drum, snare drum, and hi-hat) are considered.
−
 
+
Additionally to the two datasets from 2005, three new datasets were used in the evaluation.
−
Two new datasets.
+
For training the algorithms, the public training set from 2005 plus additional training data taken from the new datasets was provided to the participants.
  
  
 
== Submissions ==
 
== Submissions ==
−
 
−
https://drive.google.com/open?id=0B5QVjGCSDYW1Nk16dkZmYUpEM2c
 
−
 
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 17: Line 17:
 
! Contributors
 
! Contributors
 
|-
 
|-
−
| CS1-CS3 (<em>Chordino</em>)
+
| CS1-CS3
−
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/CS.pdf PDF]
+
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/CS4.pdf PDF]
 
| Carl Southall
 
| Carl Southall
 
|-
 
|-
Line 26: Line 26:
 
|-
 
|-
 
| RV1-RV4
 
| RV1-RV4
−
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/RV.pdf PDF]
+
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/RV1.pdf PDF]
 
| Richard Vogl
 
| Richard Vogl
 
|}
 
|}
 +
 +
  
 
== Results ==
 
== Results ==
 +
 +
The overall results represent the mean values over all datasets.
  
 
=== Overall ===
 
=== Overall ===
 
<csv>2017/dt/eval-set_results_global.csv</csv>
 
<csv>2017/dt/eval-set_results_global.csv</csv>
  
−
2005 baseline: 0.670 (YGO) *1
+
2005 baseline: '''0.670''' (YGO)
 +
 
 +
The best overall result from 2005 is only provided to put the current results into perspective.
 +
Since the overall result form 2005 was calculated on different datasets it is problematic to compare them directly.
  
−
=== IDMT ===
+
=== IDMT subset ===
 
<csv>2017/dt/eval-set_results_IDMT.csv</csv>
 
<csv>2017/dt/eval-set_results_IDMT.csv</csv>
  
−
2005 baseline: 0.753 (CD)
+
2005 baseline: '''0.753''' (CD)
  
−
=== KT ===
+
=== KT subset ===
 
<csv>2017/dt/eval-set_results_KT.csv</csv>
 
<csv>2017/dt/eval-set_results_KT.csv</csv>
  
−
2005 baseline: 0.617 (YGO)
+
2005 baseline: '''0.617''' (YGO)
  
−
=== RBMA ===
+
=== RBMA subset ===
 
<csv>2017/dt/eval-set_results_RBMA.csv</csv>
 
<csv>2017/dt/eval-set_results_RBMA.csv</csv>
  
−
=== MEDLEY ===
+
=== MDB subset ===
 
<csv>2017/dt/eval-set_results_MEDLEY.csv</csv>
 
<csv>2017/dt/eval-set_results_MEDLEY.csv</csv>
  
−
=== GEN ===
+
MDB-Drums [http://www.musicinformatics.gatech.edu/wp-content_nondefault/uploads/2017/10/Wu-et-al_2017_MDB-Drums-An-Annotated-Subset-of-MedleyDB-for-Automatic-Drum-Transcription.pdf]
 +
 
 +
=== GEN subset ===
 
<csv>2017/dt/eval-set_results_GEN.csv</csv>
 
<csv>2017/dt/eval-set_results_GEN.csv</csv>

Latest revision as of 20:47, 23 October 2017

Introduction

The drum transcription task was reintroduced this year after it's first edition in 2005. Two out of the three datasets used in 2005 were available and have been used for evaluation also this year. For those datasets the results from 2005 may be compared to this years results.

As in 2005 only the three main drum instruments (kick drum, snare drum, and hi-hat) are considered. Additionally to the two datasets from 2005, three new datasets were used in the evaluation. For training the algorithms, the public training set from 2005 plus additional training data taken from the new datasets was provided to the participants.


Submissions

Abstract Contributors
CS1-CS3 PDF Carl Southall
CW1-CW3 PDF Chih-Wei Wu
RV1-RV4 PDF Richard Vogl


Results

The overall results represent the mean values over all datasets.

Overall

Algorithm mean fm mean pr mean rc BD mean fm SD mean fm HH mean fm
CW1 0.51 0.46 0.68 0.68 0.48 0.38
CW3 0.53 0.50 0.65 0.67 0.46 0.42
CW2 0.55 0.52 0.66 0.70 0.55 0.40
RV3 0.68 0.74 0.70 0.81 0.64 0.51
RV2 0.67 0.69 0.73 0.78 0.67 0.51
RV1 0.71 0.75 0.74 0.82 0.70 0.53
RV4 0.70 0.74 0.73 0.81 0.70 0.52
CS1 0.61 0.56 0.73 0.79 0.55 0.46
CS3 0.63 0.59 0.75 0.78 0.58 0.49
CS2 0.63 0.61 0.71 0.78 0.57 0.49

download these results as csv

2005 baseline: 0.670 (YGO)

The best overall result from 2005 is only provided to put the current results into perspective. Since the overall result form 2005 was calculated on different datasets it is problematic to compare them directly.

IDMT subset

Algorithm mean fm mean pr mean rc BD mean fm SD mean fm HH mean fm
CW1 0.37 0.30 0.66 0.55 0.45 0.26
CW3 0.42 0.37 0.62 0.53 0.44 0.32
CW2 0.41 0.35 0.67 0.56 0.54 0.28
RV3 0.62 0.73 0.68 0.73 0.70 0.45
RV2 0.66 0.69 0.75 0.74 0.71 0.54
RV1 0.66 0.74 0.73 0.75 0.72 0.53
RV4 0.66 0.74 0.72 0.74 0.73 0.51
CS1 0.51 0.49 0.64 0.62 0.48 0.42
CS3 0.51 0.51 0.63 0.60 0.48 0.43
CS2 0.52 0.54 0.63 0.60 0.51 0.43

download these results as csv

2005 baseline: 0.753 (CD)

KT subset

Algorithm mean fm mean pr mean rc BD mean fm SD mean fm HH mean fm
CW1 0.48 0.40 0.67 0.59 0.44 0.40
CW3 0.48 0.43 0.62 0.58 0.42 0.41
CW2 0.52 0.48 0.64 0.60 0.53 0.41
RV3 0.62 0.73 0.60 0.77 0.63 0.46
RV2 0.63 0.66 0.65 0.76 0.68 0.45
RV1 0.65 0.73 0.64 0.80 0.68 0.47
RV4 0.65 0.72 0.63 0.79 0.68 0.44
CS1 0.53 0.48 0.63 0.71 0.50 0.38
CS3 0.56 0.52 0.65 0.71 0.53 0.40
CS2 0.55 0.52 0.61 0.70 0.52 0.39

download these results as csv

2005 baseline: 0.617 (YGO)

RBMA subset

Algorithm mean fm mean pr mean rc BD mean fm SD mean fm HH mean fm
CW1 0.50 0.46 0.61 0.71 0.35 0.39
CW3 0.54 0.51 0.63 0.75 0.30 0.47
CW2 0.54 0.52 0.61 0.73 0.37 0.46
RV3 0.69 0.71 0.73 0.89 0.49 0.55
RV2 0.70 0.68 0.78 0.91 0.60 0.55
RV1 0.72 0.74 0.75 0.91 0.62 0.56
RV4 0.72 0.73 0.76 0.92 0.64 0.57
CS1 0.66 0.60 0.79 0.88 0.45 0.57
CS3 0.66 0.60 0.81 0.87 0.49 0.58
CS2 0.64 0.60 0.72 0.87 0.43 0.54

download these results as csv

MDB subset

Algorithm mean fm mean pr mean rc BD mean fm SD mean fm HH mean fm
CW1 0.62 0.61 0.68 0.76 0.52 0.53
CW3 0.59 0.56 0.66 0.71 0.49 0.52
CW2 0.62 0.62 0.65 0.75 0.55 0.52
RV3 0.69 0.79 0.66 0.74 0.62 0.57
RV2 0.66 0.77 0.63 0.60 0.62 0.64
RV1 0.73 0.78 0.73 0.75 0.70 0.60
RV4 0.70 0.79 0.68 0.70 0.68 0.61
CS1 0.68 0.65 0.76 0.83 0.60 0.57
CS3 0.74 0.70 0.82 0.84 0.67 0.65
CS2 0.72 0.71 0.76 0.82 0.62 0.65

download these results as csv

MDB-Drums [1]

GEN subset

Algorithm mean fm mean pr mean rc BD mean fm SD mean fm HH mean fm
CW1 0.60 0.53 0.76 0.80 0.64 0.33
CW3 0.63 0.60 0.72 0.79 0.66 0.36
CW2 0.65 0.65 0.72 0.84 0.74 0.33
RV3 0.76 0.73 0.83 0.90 0.77 0.50
RV2 0.70 0.63 0.83 0.89 0.75 0.38
RV1 0.78 0.74 0.86 0.91 0.80 0.50
RV4 0.76 0.72 0.84 0.90 0.79 0.48
CS1 0.68 0.59 0.84 0.90 0.73 0.37
CS3 0.69 0.61 0.85 0.86 0.75 0.39
CS2 0.72 0.65 0.84 0.89 0.74 0.42

download these results as csv

Retrieved from "https://music-ir.org/mirex/w/index.php?title=2017:Drum_Transcription_Results&oldid=12354"