Difference between revisions of "2016:Audio Downbeat Estimation Results"

From MIREX Wiki
(Created page with "= Submitted Algorithms = {|class="wikitable" style="text-align: left;" |+ Algorithms submitted to the Automatic Downbeat Estimation task ! width="80" | Submission code ...")
 
(→‎Runtime)
 
(16 intermediate revisions by 2 users not shown)
Line 10: Line 10:
 
         !
 
         !
 
         !
 
         !
 +
|-
 +
! DBDR1
 +
| DB1_no_beatles  ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/DBDR1.pdf PDF] || Simon Durand, Juan Bello, Bertrand David, Gael Richard
 
|-
 
|-
 
! DBDR2
 
! DBDR2
−
| DB1_no_ballroom ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/DBDR2.pdf PDF] || Simon Durand, Juan Bello, Bertrand David, Gael Richard
+
| DB2_no_ballroom ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/DBDR2.pdf PDF] || Simon Durand, Juan Bello, Bertrand David, Gael Richard
−
|-
+
        |-
−
! DBDR3
+
! KB1
−
| DB2_no_beatles ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/DBDR3.pdf PDF] || Simon Durand, Juan Bello, Bertrand David, Gael Richard
+
| babeats13 ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/KBDW1.pdf PDF] || Florian Krebs, Sebastian Böck
 
         |-
 
         |-
−
! FK2
+
! KB2
−
| mixed_no_ballroom ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/FK2.pdf PDF] || Florian Krebs, Sebastian Böck
+
| babeats15 ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/FK3.pdf PDF] || Florian Krebs, Sebastian Böck
 
         |-
 
         |-
−
! FK3
+
! BK4
−
| mixed_no_beatles ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/FK3.pdf PDF] || Florian Krebs, Sebastian Böck
+
| joint_tracker ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/BK4.pdf PDF] || Sebastian Böck, Florian Krebs  
 
         |-
 
         |-
−
! FK4
+
        ! DSR1
−
| carnatic_tracker ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/FK4.pdf PDF] || Florian Krebs, Sebastian Böck
+
| downbeater ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/DSR1.pdf PDF] || Matthew Davies, Adam Stark, Andrew Robertson
 
         |-
 
         |-
−
         ! FK6
+
         ! CD4
−
| 2014_FK4_new_ss ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/FK6.pdf PDF] || Florian Krebs, Sebastian Böck
+
| qm-barbeattracker ||  style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2016/CD4.pdf PDF] || Matthew Davies, Chris Cannam
 
         |-
 
         |-
 
|}
 
|}
Line 41: Line 44:
 
! Recall
 
! Recall
 
|-                                                 
 
|-                                                 
−
| DBDR1
+
| BK4*
 +
| 0.908
 +
| 0.906
 +
| 0.917
 +
|-                                               
 +
| KB1*
 +
| 0.898
 +
| 0.888
 +
| 0.917
 +
|-                                               
 +
| KB2*
 +
| 0.860
 +
| 0.853
 +
| 0.890
 +
|-                                               
 +
| DBDR1*
 
| 0.838  
 
| 0.838  
 
| 0.874  
 
| 0.874  
Line 51: Line 69:
 
| 0.804  
 
| 0.804  
 
|-                                                 
 
|-                                                 
−
| BK4
+
| DSR1
−
| 0.908
+
| 0.463
−
| 0.906
+
| 0.476
−
| 0.917
+
| 0.468
 
|-                                                 
 
|-                                                 
 
| CD4
 
| CD4
Line 60: Line 78:
 
| 0.416  
 
| 0.416  
 
| 0.419  
 
| 0.419  
−
|-                                               
 
−
| DSR1
 
−
| 0.463
 
−
| 0.476
 
−
| 0.468
 
−
|-                                               
 
−
| KB1
 
−
| 0.898
 
−
| 0.888
 
−
| 0.917
 
−
|-                                               
 
−
| KB2
 
−
| 0.860
 
−
| 0.853
 
−
| 0.890
 
 
|}
 
|}
  
Line 85: Line 88:
 
! Recall
 
! Recall
 
|-                                                 
 
|-                                                 
−
| DBDR1
+
| DBDR2*
−
| 0.849
 
−
| 0.861
 
−
| 0.868
 
−
|-                                               
 
−
| DBDR2
 
 
| 0.872  
 
| 0.872  
 
| 0.861  
 
| 0.861  
−
| 0.909  
+
| 0.909
 
|-                                                 
 
|-                                                 
−
| BK4
+
| BK4*
 
| 0.865  
 
| 0.865  
 
| 0.872  
 
| 0.872  
 
| 0.876  
 
| 0.876  
 
|-                                                 
 
|-                                                 
−
| CD4
+
| DBDR1
−
| 0.604
+
| 0.849
−
| 0.586
+
| 0.861
−
| 0.642
+
| 0.868
 
|-                                                 
 
|-                                                 
−
| DSR1
+
| KB2*
−
| 0.665
+
| 0.818
−
| 0.646
+
| 0.799
−
| 0.708
+
| 0.870
 
|-                                                 
 
|-                                                 
 
| KB1
 
| KB1
 
| 0.803  
 
| 0.803  
 
| 0.776  
 
| 0.776  
−
| 0.859  
+
| 0.859
 
|-                                                 
 
|-                                                 
−
| KB2
+
| DSR1
−
| 0.818
+
| 0.665
−
| 0.799
+
| 0.646
−
| 0.870
+
| 0.708
 +
|-                                               
 +
| CD4
 +
| 0.604
 +
| 0.586
 +
| 0.642
 
|-
 
|-
 
|}
 
|}
−
 
  
 
{| class="wikitable sortable" border="1"
 
{| class="wikitable sortable" border="1"
Line 130: Line 132:
 
! Precision
 
! Precision
 
! Recall
 
! Recall
−
|-
+
|-                                              
 +
| BK4*
 +
| 0.369
 +
| 0.290
 +
| 0.566
 +
|-                                               
 +
| KB2*
 +
| 0.330
 +
| 0.263
 +
| 0.487
 +
|-                                               
 +
| KB1
 +
| 0.269
 +
| 0.221
 +
| 0.380
 +
|-                                               
 
| DBDR2
 
| DBDR2
−
| 0.221
+
| 0.231
−
| 0.195
+
| 0.194
−
| 0.297
+
| 0.330
−
|-
+
|-                                              
−
| DBDR3
+
| DBDR1
−
| 0.216
+
| 0.201
−
| 0.193
+
| 0.199
−
| 0.289
+
| 0.240
−
|-
+
|-                                              
−
| FK2
+
| CD4
 +
| 0.186
 
| 0.154  
 
| 0.154  
−
| 0.133
+
| 0.258
−
| 0.205
+
|-                                              
−
|-
+
| DSR1
−
| FK3
+
| 0.184
−
| 0.166
+
| 0.155
−
| 0.144
+
| 0.251
−
| 0.225
 
−
|-
 
−
| FK4
 
−
| 0.474
 
−
| 0.480
 
−
| 0.482
 
−
|-
 
−
| FK6
 
−
| 0.197
 
−
| 0.160
 
−
| 0.285
 
 
|}
 
|}
−
 
  
 
{| class="wikitable sortable" border="1"
 
{| class="wikitable sortable" border="1"
Line 170: Line 176:
 
! Precision
 
! Precision
 
! Recall
 
! Recall
−
|-
+
|-                                              
 +
| BK4*
 +
| 0.537
 +
| 0.468
 +
| 0.729
 +
|-                                               
 
| DBDR2
 
| DBDR2
−
| 0.472
+
| 0.415
−
| 0.445
+
| 0.360
−
| 0.579
+
| 0.554
−
|-
+
|-                                              
−
| DBDR3
+
| KB1
−
| 0.446
+
| 0.352
−
| 0.409
+
| 0.301
−
| 0.558
+
| 0.498
−
|-
+
|-                                              
−
| FK2
+
| KB2*
−
| 0.289
+
| 0.336
−
| 0.247
+
| 0.269
−
| 0.393
+
| 0.513
−
|-
+
|-                                              
−
| FK3
+
| DSR1
−
| 0.298
+
| 0.317
−
| 0.249
+
| 0.281
−
| 0.423
+
| 0.411 
−
|-
+
|-                                              
−
| FK4
+
| DBDR1
−
| 0.142
+
| 0.306
−
| 0.145
+
| 0.292
−
| 0.151
+
| 0.379
−
|-
+
|-                                              
−
| FK6
+
| CD4
−
| 0.284
+
| 0.218
−
| 0.232
+
| 0.186
−
| 0.426
+
| 0.291
 
|}
 
|}
−
 
  
 
{| class="wikitable sortable" border="1"
 
{| class="wikitable sortable" border="1"
−
 
 
|+ Results cretan dataset
 
|+ Results cretan dataset
 
|-
 
|-
Line 211: Line 220:
 
! Precision
 
! Precision
 
! Recall
 
! Recall
−
|-
+
|-                                              
 +
| BK4*
 +
| 0.635
 +
| 0.951
 +
| 0.476
 +
|-                                               
 +
| KB2*
 +
| 0.443
 +
| 0.661
 +
| 0.334
 +
|-                                               
 +
| KB1
 +
| 0.433
 +
| 0.641
 +
| 0.328
 +
|-                                               
 +
| DBDR1
 +
| 0.426
 +
| 0.715
 +
| 0.308
 +
|-                                               
 
| DBDR2
 
| DBDR2
−
| 0.415
+
| 0.418
−
| 0.640
+
| 0.637
−
| 0.307
+
| 0.311
 +
|-                                               
 +
| DSR1
 +
| 0.265
 +
| 0.398
 +
| 0.199 
 +
|-                                               
 +
| CD4
 +
| 0.250
 +
| 0.377
 +
| 0.188
 +
|}
 +
 
 +
{| class="wikitable sortable" border="1"
 +
|+ Results hjdb dataset
 
|-
 
|-
−
| DBDR3
+
! Algorithm
−
| 0.449
+
! F-Measure
 +
! Precision
 +
! Recall
 +
|-                                               
 +
| BK4*
 +
| 0.970
 +
| 0.970
 +
| 0.970
 +
|-                                               
 +
| KB2*
 +
| 0.851
 +
| 0.854
 +
| 0.848
 +
|-                                               
 +
| KB1
 +
| 0.690
 
| 0.693  
 
| 0.693  
−
| 0.332
+
| 0.688
 +
|-                                               
 +
| DBDR2
 +
| 0.629
 +
| 0.628
 +
| 0.638
 +
|-                                               
 +
| DBDR1
 +
| 0.578
 +
| 0.613
 +
| 0.561
 +
|-                                               
 +
| CD4
 +
| 0.334
 +
| 0.341
 +
| 0.329
 +
|-                                               
 +
| DSR1
 +
| 0.208
 +
| 0.232
 +
| 0.196
 +
|}
 +
 
 +
{| class="wikitable sortable" border="1"
 +
|+ Results rwc_classical dataset
 
|-
 
|-
−
| FK2
+
! Algorithm
−
| 0.151
+
! F-Measure
−
| 0.226
+
! Precision
−
| 0.113
+
! Recall
−
|-
+
|-                                               
−
| FK3
+
| BK4*
−
| 0.167
+
| 0.599
−
| 0.249
+
| 0.659
−
| 0.125
+
| 0.598
−
|-
+
|-                                              
−
| FK4
+
| DBDR2*
−
| 0.233
+
| 0.532
−
| 0.382
+
| 0.539
−
| 0.176
+
| 0.574
−
|-
+
|-                                              
−
| FK6
+
| DBDR1*
 +
| 0.527
 +
| 0.570
 
| 0.529  
 
| 0.529  
−
| 0.791
+
|-                                               
−
| 0.398
+
| KB1
 +
| 0.436
 +
| 0.475
 +
| 0.447
 +
|-                                               
 +
| KB2*
 +
| 0.428
 +
| 0.459
 +
| 0.444
 +
|-                                               
 +
| DSR1
 +
| 0.251
 +
| 0.260
 +
| 0.279
 +
|-                                               
 +
| CD4
 +
| 0.174
 +
| 0.189
 +
| 0.185
 
|}
 
|}
−
 
  
 
{| class="wikitable sortable" border="1"
 
{| class="wikitable sortable" border="1"
−
|+ Results hjdb dataset
+
|+ Results gtzan dataset
 
|-
 
|-
 
! Algorithm
 
! Algorithm
Line 251: Line 352:
 
! Precision
 
! Precision
 
! Recall
 
! Recall
−
|-
+
|-                                              
 +
| KB2
 +
| 0.647
 +
| 0.665
 +
| 0.653
 +
|-                                               
 +
| BK4
 +
| 0.638
 +
| 0.636
 +
| 0.669
 +
|-                                               
 +
| KB1
 +
| 0.630
 +
| 0.647
 +
| 0.634
 +
|-                                               
 
| DBDR2
 
| DBDR2
−
| 0.691
+
| 0.619
−
| 0.725
+
| 0.628
−
| 0.676
+
| 0.666
−
|-
+
|-                                              
−
| DBDR3
+
| DBDR1
−
| 0.682
+
| 0.615
−
| 0.705
+
| 0.651
−
| 0.672
+
| 0.631
−
|-
+
|-                                              
−
| FK2
+
| CD4
−
| 0.794
+
| 0.460
−
| 0.795
+
| 0.461
−
| 0.794
+
| 0.482
−
|-
+
|-                                              
−
| FK3
+
| DSR1
−
| 0.824
+
| 0.397
−
| 0.824
+
| 0.397
−
| 0.823
+
| 0.423
−
|-
 
−
| FK4
 
−
| 0.120
 
−
| 0.232
 
−
| 0.096
 
−
|-
 
−
| FK6
 
−
| 0.626
 
−
| 0.627
 
−
| 0.625
 
 
|}
 
|}
  
−
<nowiki>*</nowiki>) Rows marked by an asterisk should be taken with care as in those cases overlapping test and training sets were used.
+
<nowiki>*</nowiki>) Rows marked by an asterisk should be taken with care as in those cases overlapping test and training sets were used. This could lead to overestimated metrics.
  
 
= Runtime =
 
= Runtime =
−
All submissions finished the computations in less than 24 hours.
+
The runtime is measured for the *hjdb* dataset (duration 3h19m) and then extrapolated to the duration of all datasets (total 38h51m). For measuring runtimes, we used a PC with four i5-2400 CPUs @ 3.10GHz.
 +
 
 +
{| class="wikitable" border="1"
 +
|-
 +
! Algorithm
 +
! Multi-core
 +
! Runtime hjdb [mm]
 +
! Extrapolated runtime all [hh]
 +
! x times faster than realtime
 +
|-                                               
 +
| DBDRx
 +
| yes
 +
| 86.8
 +
| 16.9
 +
| 2.29
 +
|-                                                                                           
 +
| KBx
 +
| yes
 +
| 22.8
 +
| 4.45
 +
| 8.72
 +
|-                                                                                           
 +
| BK4
 +
| yes
 +
| 13.1
 +
| 2.55
 +
| 15.2
 +
|-
 +
| DSR1
 +
| no
 +
| 4.5
 +
| 0.88
 +
| 44.2
 +
|-                                                 
 +
| CD4
 +
| no
 +
| 3.1
 +
| 0.6
 +
| 63.5
 +
|-                                                                                             
 +
|}

Latest revision as of 09:00, 24 August 2016

Submitted Algorithms

Algorithms submitted to the Automatic Downbeat Estimation task
Submission code Submission name Abstract Contributors
DBDR1 DB1_no_beatles PDF Simon Durand, Juan Bello, Bertrand David, Gael Richard
DBDR2 DB2_no_ballroom PDF Simon Durand, Juan Bello, Bertrand David, Gael Richard
KB1 babeats13 PDF Florian Krebs, Sebastian Böck
KB2 babeats15 PDF Florian Krebs, Sebastian Böck
BK4 joint_tracker PDF Sebastian Böck, Florian Krebs
DSR1 downbeater PDF Matthew Davies, Adam Stark, Andrew Robertson
CD4 qm-barbeattracker PDF Matthew Davies, Chris Cannam

Results

Results ballroom dataset
Algorithm F-Measure Precision Recall
BK4* 0.908 0.906 0.917
KB1* 0.898 0.888 0.917
KB2* 0.860 0.853 0.890
DBDR1* 0.838 0.874 0.846
DBDR2 0.783 0.808 0.804
DSR1 0.463 0.476 0.468
CD4 0.412 0.416 0.419
Results beatles dataset
Algorithm F-Measure Precision Recall
DBDR2* 0.872 0.861 0.909
BK4* 0.865 0.872 0.876
DBDR1 0.849 0.861 0.868
KB2* 0.818 0.799 0.870
KB1 0.803 0.776 0.859
DSR1 0.665 0.646 0.708
CD4 0.604 0.586 0.642
Results carnatic dataset
Algorithm F-Measure Precision Recall
BK4* 0.369 0.290 0.566
KB2* 0.330 0.263 0.487
KB1 0.269 0.221 0.380
DBDR2 0.231 0.194 0.330
DBDR1 0.201 0.199 0.240
CD4 0.186 0.154 0.258
DSR1 0.184 0.155 0.251
Results turkish dataset
Algorithm F-Measure Precision Recall
BK4* 0.537 0.468 0.729
DBDR2 0.415 0.360 0.554
KB1 0.352 0.301 0.498
KB2* 0.336 0.269 0.513
DSR1 0.317 0.281 0.411
DBDR1 0.306 0.292 0.379
CD4 0.218 0.186 0.291
Results cretan dataset
Algorithm F-Measure Precision Recall
BK4* 0.635 0.951 0.476
KB2* 0.443 0.661 0.334
KB1 0.433 0.641 0.328
DBDR1 0.426 0.715 0.308
DBDR2 0.418 0.637 0.311
DSR1 0.265 0.398 0.199
CD4 0.250 0.377 0.188
Results hjdb dataset
Algorithm F-Measure Precision Recall
BK4* 0.970 0.970 0.970
KB2* 0.851 0.854 0.848
KB1 0.690 0.693 0.688
DBDR2 0.629 0.628 0.638
DBDR1 0.578 0.613 0.561
CD4 0.334 0.341 0.329
DSR1 0.208 0.232 0.196
Results rwc_classical dataset
Algorithm F-Measure Precision Recall
BK4* 0.599 0.659 0.598
DBDR2* 0.532 0.539 0.574
DBDR1* 0.527 0.570 0.529
KB1 0.436 0.475 0.447
KB2* 0.428 0.459 0.444
DSR1 0.251 0.260 0.279
CD4 0.174 0.189 0.185
Results gtzan dataset
Algorithm F-Measure Precision Recall
KB2 0.647 0.665 0.653
BK4 0.638 0.636 0.669
KB1 0.630 0.647 0.634
DBDR2 0.619 0.628 0.666
DBDR1 0.615 0.651 0.631
CD4 0.460 0.461 0.482
DSR1 0.397 0.397 0.423

*) Rows marked by an asterisk should be taken with care as in those cases overlapping test and training sets were used. This could lead to overestimated metrics.

Runtime

The runtime is measured for the *hjdb* dataset (duration 3h19m) and then extrapolated to the duration of all datasets (total 38h51m). For measuring runtimes, we used a PC with four i5-2400 CPUs @ 3.10GHz.

Algorithm Multi-core Runtime hjdb [mm] Extrapolated runtime all [hh] x times faster than realtime
DBDRx yes 86.8 16.9 2.29
KBx yes 22.8 4.45 8.72
BK4 yes 13.1 2.55 15.2
DSR1 no 4.5 0.88 44.2
CD4 no 3.1 0.6 63.5
Retrieved from "https://music-ir.org/mirex/w/index.php?title=2016:Audio_Downbeat_Estimation_Results&oldid=11953"