== Introduction ==

The drum transcription task was reintroduced last year after it's first edition in 2005.
Two out of the three original datasets used in 2005 were available and have been used for evaluation also this year.
For those datasets the results from 2005 may be compared to this years results.
Additionally to the two datasets from 2005, three new datasets were used in the evaluation.
For training the algorithms, the public training set from 2005 plus additional training data taken from the new datasets was provided to the participants.
For these datasets only the three main drum instruments (kick drum, snare drum, and hi-hat) are considered.

This year additionally to the 3 instrument class task, a second sub task, an 8 instrument class transcription task, was introduced.
To this new evaluation and training datasets were used this year.

== Submissions ==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| AR5, JAR1-3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/AR5.pdf PDF]
| [https://www.ircam.fr/ Celine Jacques], [https://www.ircam.fr/recherche/equipes-recherche/anasyn/ Achille Aknin], [http://anasynth.ircam.fr/home/english/people/roebel Axel Roebel]
|-
| CS2-4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/CS2.pdf PDF]
| [http://www.bcu.ac.uk/computing-engineering-and-the-built-environment/research/digital-technology Carl Southall]
|-
| JS1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/JS1.pdf PDF]
| [https://www.cardiff.ac.uk/people/research-students/view/999651- Julien Schroeter]
|-
| RV1-6
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2018/RV1.pdf PDF]
| [http://ifs.tuwien.ac.at/user/426 Richard Vogl]
|}

== 3 Class Results ==

The overall results represent the mean values over all datasets.

=== Overall ===
<csv>2018/dt/eval-set_3_results_global.csv</csv>

2005 baseline: '''0.670''' (YGO)

The best overall result from 2005 is only provided to put the current results into perspective.
Since the overall result form 2005 was calculated on different datasets it is problematic to compare them directly.

=== IDMT subset ===
<csv>2018/dt/eval-set_3_results_IDMT.csv</csv>

2005 baseline: '''0.753''' (CD)

=== KT subset ===
<csv>2018/dt/eval-set_3_results_KT.csv</csv>

2005 baseline: '''0.617''' (YGO)

=== RBMA subset ===
<csv>2018/dt/eval-set_3_results_RBMA.csv</csv>

=== MDB subset ===
<csv>2018/dt/eval-set_3_results_MEDLEY.csv</csv>

MDB-Drums [http://www.musicinformatics.gatech.edu/wp-content_nondefault/uploads/2017/10/Wu-et-al_2017_MDB-Drums-An-Annotated-Subset-of-MedleyDB-for-Automatic-Drum-Transcription.pdf]

=== GEN subset ===
<csv>2018/dt/eval-set_3_results_GEN.csv</csv>

== 8 Class Results ==

The overall results represent the mean values over all datasets.

=== Overall ===
<csv>2018/dt/eval-set-8_results_global.csv</csv>

=== RBMA subset ===
<csv>2018/dt/eval-set-8_results_RBMA.csv</csv>

=== MDB subset ===
<csv>2018/dt/eval-set-8_results_MEDLEY.csv</csv>

MDB-Drums [http://www.musicinformatics.gatech.edu/wp-content_nondefault/uploads/2017/10/Wu-et-al_2017_MDB-Drums-An-Annotated-Subset-of-MedleyDB-for-Automatic-Drum-Transcription.pdf]

=== MIDI subset ===
<csv>2018/dt/eval-set-8_results_MIDI.csv</csv>

2018:Drum Transcription

2018-08-09T19:00:48Z

Richard Vogl: /* Submission closing date */

==Description==

Drum transcription is the task of detecting the positions in time and labeling the drum class of drum instrument onsets in polyphonic music.
This information is a prerequisite for several applications and can also be used for other high-level MIR tasks.
Due to several new approaches recently being presented we propose to reintroduce this task.
We will mainly stick to the mode used in the first edition in 2005, but new datasets will be used.
Only the three main drum instruments of drum kits for western pop music are considered.
These are: bass drum, snare drum, and hi-hat (in all variations like open, closed, pedal, etc.).

In addition to the classic three drum instrument task, we will also run a 8 drum instrument classes task, this year.
Separate training and evaluation data will be used for this task.

==Data==

For evaluation 5 different datasets will be used.
We use two of the three datasets from the 2005 drum detection MIREX task as a baseline.

* CD set
* KT set

Additionally three new datasets will be used. They contain polyphonic music of different genres, as well as drum only tracks, and some tracks without drums:
* RBMA set (35 full length, polyphonic tracks, electronically produced and recorded, manually annotated and double-checked)
* MEDLEY set (23 full length tracks, recorded, manually annotated and double-checked)
* GEN set (synthesized MIDI drum tracks and loops without accompaniment)

===Audio Format===

The input for this task is a set of sound files adhering to the format and content requirements mentioned below.

* All audio is 44100 Hz, 16-bit mono, WAV PCM
* All available sound files will be used in their entirety (which can be short excerpts of 30s or full length music tracks of up to 7m)
* Some sound files will be recorded polyphonic music with drums (might be live performances or studio recordings)
* Some sound files will be rendered audio of MIDI files
* Some sound files may not contain any drums
* Both drums mixed with music and solo drums, will be part of the set
* Tracks with only the three drum instruments (or less) as well as tracks with full drum kits (with instruments not expected to be transcribed) will be part of the set
* Drum kit sounds will have a broad range: from natural recorded kits, live kits to sampled drums as well as electronic synthesizers

===Training Data===

* A representative random subset of the data will be made available to all participants in advance of the evaluation - please contact the task captains!
* Training data can be used by the participants as they please.
* Training data will not be used again during the evaluation.
* Usage of additional training data is discouraged. If additional training data is used, please note so in the submission.

==I/O format==

The input will be a directory containing audio files in the audio format specified above.
There might be other files in the directory, so make sure to filter for ‘*.wav’ files.

The output will also be a directory.
The algorithm is expected to process every file and generate an individual *.txt output file for every wav file with the same name.
e.g.:
input:
audio_file_10.wav
output:
audio_file_10.txt

For transcription three drum instrument types are considered:
BD 0 bass drum
SD 1 snare drum
HH 2 hi-hat (any hi-hat like open, half-open, closed, ...)

Drum types are strictly these types only (so: no ride cymbals in the HH, no toms in the BD, no claps nor side sticks/rim shots in the SD, etc...)
This involves the following remapping from other labels to these 3 base labels:

name midi label code
bass drum 36 KD 0
snare drum 38 SD 1
closed hi-hat 42 HH 2
open hi-hat 46 HH 2
pedal hi-hat 44 HH 2
cowbell 56
ride bell 53
low floor tom 41
high floor tom 43
low tom 45
low-mid tom 47
high-mid tom 48
high tom 50
side stick 37
hand clap 39
ride cymbal 51
crash cymbal 49
splash cymbal 55
chinese cymbal 52
shaker, maracas 70
tambourine 54
claves, sticks 75

All annotations are remapped to these three labels in advance (no looking back to the broader labels afterwards).

The annotation files as well as the expected output of the algorithms will have the following format:
A text file (UTF-8 encoding) with no header and footer, one line represents an instrument onset with the following format:
<TTT.TTT> \t <LL> \n
Where <TTT.TTT> is a floating point number with 3 decimals (ms accuracy), followed by a tab and <LL> the label for drum instrument onset as defined above (either number, or string), followed by a newline.
If multiple onsets occur at the exact same time, two separate lines with the same timestamp are expected.

Example of the content of a output file:

[test_file_0.txt]
<start-of-file>
0.125 0
0.125 2
0.250 2
0.375 1
0.375 2
0.500 2
0.625 0
0.625 2
0.750 2
0.875 1
0.875 2
1.000 2
<end-of-file>

Annotation files for the public subset will have the same format

=== 8 class labels ===
In case of the 8 class data, labels are as following:

BD 0 bass drum
SD 1 snare drum
TT 2 any tom-tom
HH 3 hi-hat (any hi-hat like open, half-open, closed, ...)
CY 4 cymbal (any crashed cymbal, e.g.: crash, splash, chinese)
RD 5 ride (not crashed)
CB 6 cowbell (and ride bell)
CL 7 clave / sticks

name midi label code
bass drum 36 KD 0
snare drum 38 SD 1
closed hi-hat 42 HH 3
open hi-hat 46 HH 3
pedal hi-hat 44 HH 3
cowbell 56 CB 6
ride bell 53 CB 6
low floor tom 41 TT 2
high floor tom 43 TT 2
low tom 45 TT 2
low-mid tom 47 TT 2
high-mid tom 48 TT 2
high tom 50 TT 2
side stick 37
hand clap 39
ride cymbal 51 RD 5
crash cymbal 49 CY 4
splash cymbal 55 CY 4
chinese cymbal 52 CY 4
shaker, maracas 70
tambourine 54
claves, sticks 75 CL 7

Note that there might be annotations outside of the duration of the corresponding audio. These are to be ignored. 

==Evaluation==

* F-measure (harmonic mean of the recall rate and the precision rate, beta parameter 1, so equal importance to prec. and recall) is calculated for each of three drum types (BD, SD, and HH), resulting in three F-measure scores.
* Additionally a total F-measure score for all onsets over all instrument classes will be calculated.
* Calculation time measure: the time it takes to do the complete run from the moment your algorithm starts until the moment it stops will be reported

Evaluation parameters:
* The limit of onset-deviation errors in calculating the above F-measure is 30 ms (so a range of [-30 ms, +30 ms] around the true times)
* Any parameter adaptation (e.g. for peak picking) must be done on public data, i.e. in advance.

Conditions:
* The actual drum sounds (sound samples) used in any of the input audio are not public and not used for training.
* Participants are encouraged to only use the provided training data for training and parameter optimization.
If this is not possible, it should explicitly be stated that and which additional data was used. In this case it would be favorable to submit two versions: one trained with the public data only, and one trained using additional data.

==Packaging submissions==

* Participants only send in the application part of their algorithm, not the training part (if there is one)
* Algorithms must adhere to the specifications on the MIREX web page

===Command line calling format===

Python:

python <your_script_name.py> -i <inputfolder> -o <outputfolder>

Matlab:

<path_to_matlab>\matlab.exe" -nodisplay -nosplash -nodesktop -r "try, <your_script_name>(<inputfolder>, <outputfolder>), catch me, fprintf('%s / %s\n',me.identifier,me.message), end, exit"

Sonic Annotator:

Contact task captains to define format.

===Time, Software and Hardware limits===

Max runtime: TBA, we must be able to run it in the time give.

Software: Preferred Python. May be Matlab, Sonic Annotator.

==Submission closing date==

<strike>August 11th 2018</strike> August 18th 2018