MIREX Wiki - User contributions [en]

2016:MIREX2016 Results

2016-08-04T16:58:47Z

Peter Organisciak: Struct results

*Train-Test Task Set
** [https://www.music-ir.org/nema_out/mirex2016/results/act/composer_report/ Audio Classical Composer Identification Results ]  
** [https://www.music-ir.org/nema_out/mirex2016/results/act/latin_report/ Audio Latin Genre Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2016/results/act/mood_report/index.html Audio Music Mood Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2016/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2016/results/act/kgenrek_report/ Audio KPOP Genre (Annotated by Korean Annotators) Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2016/results/act/kgenrea_report/ Audio KPOP Genre (Annotated by American Annotators) Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2016/results/act/kmoodk_report/ Audio KPOP Mood (Annotated by Korean Annotators) Classification Results ]  
** [https://www.music-ir.org/nema_out/mirex2016/results/act/kmooda_report/ Audio KPOP Mood (Annotated by American Annotators) Classification Results ]  

* Audio Key Detection Results
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/akd/mrx_05 MIREX 2005 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/akd/gsteps GiantSteps Dataset]  

* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/adc04/ ADC04 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/mrx05/ MIREX05 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/ind08/ INDIAN08 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/mrx09_0db/ MIREX09 0dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ame/orchset/ ORCHSET15 Dataset]  

* Query-by-Singing/Humming Results
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task1_hidden/ Hidden Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task1_jang/ Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task1_thinkit/ ThinkIt Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task1_ioacas/ IOACAS Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/qbsh/qbsh_task2_jang/ Subtask2 Dataset]  

* [[2016:Audio Downbeat Estimation Results|Audio Downbeat Estimation Results]]
* [[2016:Audio Fingerprinting Results|Audio Fingerprinting Results]]

* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/struct/mrx09/ MIREX09 dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/struct/salami/ SALAMI dataset] (partial)  

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/abt/smc/ SMC Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/abt/maz/ MAZ Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2016/results/abt/mck/ MCK Dataset]  

* [https://nema.lis.illinois.edu/nema_out/mirex2016/results/aod/ Audio Onset Detection Results]  
* [https://nema.lis.illinois.edu/nema_out/mirex2016/results/ate/ Audio Tempo Estimation Results]  

* [[2016:Audio_Chord_Estimation_Results | Audio Chord Estimation]]
** [[2016:Audio_Chord_Estimation_Results#Isophonics_2009 | Isophonics 2009 Dataset]]  
** [[2016:Audio_Chord_Estimation_Results#Billboard_2012 | Billboard 2012 Dataset]]  
** [[2016:Audio_Chord_Estimation_Results#Billboard_2013 | Billboard 2013 Dataset]]  
** [[2016:Audio_Chord_Estimation_Results#JayChou_2015 | JayChou 2015 Dataset]]  
** [[2016:Audio_Chord_Estimation_Results#RobbieWilliams_2016 | RobbieWilliams 2016 Dataset]]  

* Multiple Fundamental Frequency Estimation & Tracking Results
** [[2016:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_%2D_MIREX_Dataset | MIREX Dataset]]  

* [[2016:Set List Identification Results | Set List Identification Results]]

* [https://www.music-ir.org/mirex/wiki/2016:Singing_Voice_Separation_Results Singing Voice Separation]

2015:Audio Cover Song Identification Results

2015-10-21T04:06:14Z

Peter Organisciak: /* General Legend */

== Introduction ==

Audio Cover Song (ACS) Identification was evaluated against two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! CT1
| MFCCShapeSSM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CT1.pdf PDF] || [http://www.ctralie.com/ Christopher Tralie]
|-
! YWW1
| YWW || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YWW1.pdf PDF] || [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]
|-
! CYWW1
| CYWW || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CYWW1.pdf PDF] || [http://slam.iis.sinica.edu.tw/ Chuan-Yau Chan], [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2015/acs/coversong1000.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2015/acs/coversong1000.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2015/acs/coversong1000.precision.groups.csv</csv>

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2015/acs/mazurkas.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2015/acs/mazurkas.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2015/acs/mazurkas.precision.groups.csv</csv>

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.coversong1000.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.coversong1000.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.coversong1000.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.coversong1000.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.mazurkas.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.mazurkas.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.mazurkas.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.mazurkas.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 
[[Category: Results]]

2015:Audio Cover Song Identification Results

2015-10-21T04:02:39Z

Peter Organisciak: /* General Legend */

== Introduction ==

Audio Cover Song (ACS) Identification was evaluated against two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! CT1
| MFCCShapeSSM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CT1.pdf PDF] || Christopher Tralie
|-
! YWW1
| YWW || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YWW1.pdf PDF] || [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]
|-
! CYWW1
| CYWW || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CYWW1.pdf PDF] || [http://slam.iis.sinica.edu.tw/ Chuan-Yau Chan], [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2015/acs/coversong1000.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2015/acs/coversong1000.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2015/acs/coversong1000.precision.groups.csv</csv>

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2015/acs/mazurkas.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2015/acs/mazurkas.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2015/acs/mazurkas.precision.groups.csv</csv>

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.coversong1000.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.coversong1000.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.coversong1000.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.coversong1000.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.mazurkas.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.mazurkas.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.mazurkas.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.mazurkas.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 
[[Category: Results]]

2015:Audio Cover Song Identification Results

2015-10-21T04:02:16Z

Peter Organisciak: /* General Legend */

== Introduction ==

Audio Cover Song (ACS) Identification was evaluated against two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! CT1
| MFCCShapeSSM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CT1.pdf PDF] || Christopher Tralie
|-
! YWW1
| YWW || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YWW1.pdf PDF] | [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]
|-
! CYWW1
| CYWW || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CYWW1.pdf PDF] || [http://slam.iis.sinica.edu.tw/ Chuan-Yau Chan], [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2015/acs/coversong1000.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2015/acs/coversong1000.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2015/acs/coversong1000.precision.groups.csv</csv>

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2015/acs/mazurkas.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2015/acs/mazurkas.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2015/acs/mazurkas.precision.groups.csv</csv>

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.coversong1000.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.coversong1000.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.coversong1000.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.coversong1000.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.mazurkas.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.mazurkas.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.mazurkas.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.mazurkas.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 
[[Category: Results]]

2015:Audio Cover Song Identification Results

2015-10-21T04:01:15Z

Peter Organisciak:

== Introduction ==

Audio Cover Song (ACS) Identification was evaluated against two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! CT1
| MFCCShapeSSM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CT1.pdf PDF] || Christopher Tralie
|-
! YWW1
| YWW || style="text-align: center;" | [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]
|-
! CYWW1
| CYWW || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CYWW1.pdf PDF] || [http://slam.iis.sinica.edu.tw/ Chuan-Yau Chan], [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2015/acs/coversong1000.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2015/acs/coversong1000.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2015/acs/coversong1000.precision.groups.csv</csv>

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2015/acs/mazurkas.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2015/acs/mazurkas.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2015/acs/mazurkas.precision.groups.csv</csv>

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.coversong1000.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.coversong1000.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.coversong1000.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.coversong1000.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.mazurkas.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.mazurkas.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.mazurkas.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.mazurkas.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 
[[Category: Results]]

2015:Audio Cover Song Identification Results

2015-10-21T01:49:50Z

Peter Organisciak: Created page with "== Introduction == Audio Cover Song (ACS) Identification was evaluated against two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''. ===Mixed Collection Informa..."

== Introduction ==

Audio Cover Song (ACS) Identification was evaluated against two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! YCW1
| MCY_COVER || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/YCW1.pdf PDF] || [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://slam.iis.sinica.edu.tw/ Chuan-Yau Chan], [http://www.iis.sinica.edu.tw Hsin-Min Wang]
|-
! CYWW1
| CYC_COVER || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2015/CYWW1.pdf PDF] || [http://slam.iis.sinica.edu.tw/ Chuan-Yau Chan], [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2015/acs/coversong1000.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2015/acs/coversong1000.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2015/acs/coversong1000.precision.groups.csv</csv>

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2015/acs/mazurkas.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2015/acs/mazurkas.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2015/acs/mazurkas.precision.groups.csv</csv>

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.coversong1000.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.coversong1000.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.coversong1000.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.coversong1000.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.mazurkas.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.mazurkas.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''CT1''' : [https://music-ir.org/mirex/results/2015/acs/CT1.mazurkas.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2015/acs/CYWW1.mazurkas.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 
[[Category: Results]]

2015:MIREX2015 Results

2015-10-21T00:15:07Z

Peter Organisciak: /* Other Tasks */

==Results by Task ==

==OVERALL RESULTS POSTERS ==

[https://www.music-ir.org/mirex/results/2015/mirex_2015_poster.pdf MIREX 2015 Overall Results Posters (PDF)]

===Train-Test Task Set===
* [https://www.music-ir.org/nema_out/mirex2015/results/act/composer_report/ Audio Classical Composer Identification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/latin_report/ Audio Latin Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/mood_report/index.html Audio Music Mood Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/kgenrek_report/ Audio KPOP Genre (Annotated by Korean Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/kgenrea_report/ Audio KPOP Genre (Annotated by American Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/kmoodk_report/ Audio KPOP Mood (Annotated by Korean Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2015/results/act/kmooda_report/ Audio KPOP Mood (Annotated by American Annotators) Classification Results ]  

===Other Tasks===

* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/struct/mrx09/ MIREX09 dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/struct/sal/ SALAMI dataset]  

* [[2015:Discovery of Repeated Themes & Sections Results | Discovery of Repeated Themes & Sections Results]]

* [https://nema.lis.illinois.edu/nema_out/mirex2015/results/aod/ Audio Onset Detection Results]  

* [[2015:Audio Cover Song Identification Results|Audio Cover Song Identification Results]]

* [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ate/ Audio Tempo Estimation Results]  

* Audio Key Detection Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/akd/mrx_05 MIREX 2015 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/akd/gsteps GiantSteps Dataset]  

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/abt/smc/ SMC Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/abt/maz/ MAZ Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/abt/mck/ MCK Dataset]  

* Audio Chord Estimation
** [[2015:Audio_Chord_Estimation_Results#MIREX_Chord_2009 | MIREX Chord ’09 Dataset]]  
** [[2015:Audio_Chord_Estimation_Results#Billboard_2012 | Billboard ’12 Dataset]]  
** [[2015:Audio_Chord_Estimation_Results#Billboard_2013 | Billboard ’13 Dataset]]  

* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/adc04/ ADC04 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/mrx05/ MIREX05 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/ind08/ INDIAN08 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/mrx09_0db/ MIREX09 0dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/ame/orchset/ Orcheset15 Dataset]  

* Query-by-Singing/Humming Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task1_hidden/ Hidden Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task1_jang/ Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task1_thinkit/ ThinkIt Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task1_ioacas/ IOACAS Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbsh/qbsh_task2_jang/ Subtask2 Dataset]  

* Query-by-Tapping Results
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbt/qbt_task1_jang/ Subtask 1, Jang dataset]
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbt/qbt_task1_hsiao/ Subtask 1, Hsiao dataset]
** Subtask 1, QBT-Extended dataset
** [https://nema.lis.illinois.edu/nema_out/mirex2015/results/qbt/qbt_task2_jang/ Subtask 2, Jang dataset]
** Subtask 3, QBT-Extended dataset

* [[2015:Audio Downbeat Estimation Results|Audio Downbeat Estimation Results]]
* [[2015:Audio Fingerprinting Results|Audio Fingerprinting Results]]

* [https://www.music-ir.org/mirex/wiki/2015:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results#Summary_Results Real-time Audio to Score Alignment (a.k.a. Score Following) Results]  

* [https://www.music-ir.org/mirex/wiki/2015:Audio_Music_Similarity_and_Retrieval_Results Audio Music Similarity and Retrieval Results] 
* [[2015:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]
* [https://www.music-ir.org/mirex/wiki/2015:Singing_Voice_Separation_Results Singing Voice Separation] 

* Multiple Fundamental Frequency Estimation & Tracking Results
** [[2015:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_%2D_MIREX_Dataset | MIREX Dataset]]  
** [[2015:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_%2D_Su_Dataset |Su Dataset]]  

* [https://www.music-ir.org/mirex/wiki/2015:Music/Speech_Classification_and_Detection_Results Music/Speech Classification and Detection]

* [[2015:Set List Identification Results | Set List Identification Results]]

2014:MIREX2014 Results

2014-10-20T16:03:43Z

Peter Organisciak: /* Other Tasks */

==Results by Task ==

===Train-Test Task Set===
* [https://www.music-ir.org/nema_out/mirex2014/results/act/composer_report/ Audio Classical Composer Identification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/latin_report/ Audio Latin Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/mood_report/index.html Audio Music Mood Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/kgenrek_report/ Audio KPOP Genre (Annotated by Korean Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/kgenrea_report/ Audio KPOP Genre (Annotated by American Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/kmoodk_report/ Audio KPOP Mood (Annotated by Korean Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/kmooda_report/ Audio KPOP Mood (Annotated by American Annotators) Classification Results ]  

===Other Tasks===

* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/mrx09/ MIREX09 dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/sal/ SALAMI dataset]  

* [https://nema.lis.illinois.edu/nema_out/mirex2014/results/aod/ Audio Onset Detection Results]  

* [[2014:Audio Cover Song Identification Results|Audio Cover Song Identification Results]]

* [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ate/ Audio Tempo Estimation Results]  
* [https://nema.lis.illinois.edu/nema_out/mirex2014/results/akd/ Audio Key Detection Results]  

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/abt/smc/ SMC Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/abt/maz/ MAZ Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/abt/mck/ MCK Dataset]  

* Audio Chord Estimation
** [[2014:Audio_Chord_Estimation_Results_MIREX_2009 | MIREX ’09 Dataset]]  
** [[2014:Audio_Chord_Estimation_Results_Billboard_2012 | Billboard ’12 Dataset]]  
** [[2014:Audio_Chord_Estimation_Results_Billboard_2013 | Billboard ’13 Dataset]]  

* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/adc04/ ADC04 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/mrx05/ MIREX05 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/ind08/ INDIAN08 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/mrx09_0db/ MIREX09 0dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset]  

* Audio Tag Classification Results
** Major Miner Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)]  
*** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/atg/subtask1_report/aff/ Affinity estimation evaluation]  
** Mood Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)]  
*** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/atg/subtask2_report/aff/ Affinity estimation evaluation]  

* Query-by-Singing/Humming Results
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/qbsh/qbsh_task1_hidden/ Hidden Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/qbsh/qbsh_task1_jang/ Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/qbsh/qbsh_task1_thinkit/ ThinkIt Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/qbsh/qbsh_task1_ioacas/ IOACAS Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/qbsh/qbsh_task2_jang/ Subtask2 Dataset]  

* [[2014:Audio Downbeat Estimation Results|Audio Downbeat Estimation Results]]

* [https://www.music-ir.org/mirex/wiki/2014:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results#Summary_Results Real-time Audio to Score Alignment (a.k.a. Score Following) Results]  

* [https://www.music-ir.org/mirex/wiki/2014:Audio_Music_Similarity_and_Retrieval_Results Audio Music Similarity and Retrieval Results] 
* [[2014:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]
* [https://www.music-ir.org/mirex/wiki/2014:Singing_Voice_Separation_Results#For_the_Music_Accompaniment_.28dB.29 Singing Voice Separation] 

* [https://www.music-ir.org/mirex/wiki/2014:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results Multiple Fundamental Frequency Estimation & Tracking Results] 

* [[2014:Discovery of Repeated Themes & Sections Results | Discovery of Repeated Themes & Sections Results]]

2014:Audio Chord Estimation Results Billboard 2012

2014-10-20T16:02:27Z

Peter Organisciak: /* Results */

==Introduction==

This page contains the results of these new evaluations for an abridged version of the ''Billboard'' dataset from McGill University, including a representative sample of American popular music from the 1950s through the 1990s, as used for MIREX 2012.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2014/ace/billboard12.csv</csv>

2014:Audio Chord Estimation Results Billboard 2013

2014-10-20T16:02:05Z

Peter Organisciak: /* Results */

==Introduction==

This page contains the results of these new evaluations for a special subset of the ''Billboard'' dataset from McGill University that has never been made available to the public. Further subsets have been withheld to support the ACE task through MIREX 2015.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2014/ace/billboard13.csv</csv>

2014:Audio Chord Estimation Results

2014-10-20T16:01:38Z

Peter Organisciak: /* Results */

==Introduction==

This page contains the results of these new evaluations for the Isophonics dataset, a.k.a. the MIREX 2009 dataset. It comprises the collected Beatles, Queen, and Zweieck datasets from Queen Mary, University of London, and has been used for audio chord estimation in MIREX for many years.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2014/ace/mirex09.csv</csv>

2014:Audio Chord Estimation Results

2014-10-20T15:59:44Z

Peter Organisciak: /* Introduction */

==Introduction==

This page contains the results of these new evaluations for the Isophonics dataset, a.k.a. the MIREX 2009 dataset. It comprises the collected Beatles, Queen, and Zweieck datasets from Queen Mary, University of London, and has been used for audio chord estimation in MIREX for many years.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2014/ace/mirex09.csv</csv>

===Comparative Statistics===

* ''coming soon...''

===Complete Results===

More detailed about the performance of the algorithms, including per-song performance and the breakdown of the WCSR calculations, is available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/MirexChord2009.zip MirexChord2009.zip]

===Algorithmic Output===

The recognition output and the ground-truth files are available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/MirexChord2009Output.zip MirexChord2009Output.zip]

We hope to generate a graphical comparison of all algorithms against the ground truth early in 2014.

2014:Audio Chord Estimation Results Billboard 2013

2014-10-20T15:59:28Z

Peter Organisciak: /* Introduction */

==Introduction==

This page contains the results of these new evaluations for a special subset of the ''Billboard'' dataset from McGill University that has never been made available to the public. Further subsets have been withheld to support the ACE task through MIREX 2015.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2014/ace/billboard13.csv</csv>

===Comparative Statistics===

* ''coming soon...''

===Complete Results===

More detailed about the performance of the algorithms, including per-song performance and the breakdown of the WCSR calculations, is available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2013.zip BillboardTest2013.zip]

===Algorithmic Output===

The recognition output and the ground-truth files are available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2013Output.zip BillboardTest2013Output.zip]

We hope to generate a graphical comparison of all algorithms against the ground truth early in 2014.

2014:Audio Chord Estimation Results Billboard 2012

2014-10-20T15:58:39Z

Peter Organisciak: /* Submissions */

==Introduction==

This page contains the results of these new evaluations for an abridged version of the ''Billboard'' dataset from McGill University, including a representative sample of American popular music from the 1950s through the 1990s, as used for MIREX 2012.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2014/ace/billboard12.csv</csv>

===Comparative Statistics===

* ''coming soon...''

===Complete Results===

More detailed about the performance of the algorithms, including per-song performance and the breakdown of the WCSR calculations, is available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2012.zip BillboardTest2012.zip]

===Algorithmic Output===

The recognition output and the ground-truth files are available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2012Output.zip BillboardTest2012Output.zip]

We hope to generate a graphical comparison of all algorithms against the ground truth early in 2014.

2014:Audio Chord Estimation Results Billboard 2013

2014-10-20T15:58:31Z

Peter Organisciak: /* Submissions */

==Introduction==

This year, we have started a new evaluation battery for audio chord estimation. This page contains the results of these new evaluations for a special subset of the ''Billboard'' dataset from McGill University that has never been made available to the public. Further subsets have been withheld to support the ACE task through MIREX 2015.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2014/ace/billboard13.csv</csv>

===Comparative Statistics===

* ''coming soon...''

===Complete Results===

More detailed about the performance of the algorithms, including per-song performance and the breakdown of the WCSR calculations, is available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2013.zip BillboardTest2013.zip]

===Algorithmic Output===

The recognition output and the ground-truth files are available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2013Output.zip BillboardTest2013Output.zip]

We hope to generate a graphical comparison of all algorithms against the ground truth early in 2014.

2014:Audio Chord Estimation Results

2014-10-20T15:58:23Z

Peter Organisciak: /* Submissions */

==Introduction==

This year, we have started a new evaluation battery for audio chord estimation. This page contains the results of these new evaluations for the Isophonics dataset, a.k.a. the MIREX 2009 dataset. It comprises the collected Beatles, Queen, and Zweieck datasets from Queen Mary, University of London, and has been used for audio chord estimation in MIREX for many years.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| KO1 (shineChords)
| style="text-align: center;" |  
| Maksim Khadkevich, Maurizio Omologo
|-
| CB3 (Chordino)
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/CB3.pdf PDF]
| Chris Cannam, Matthias Mauch
|-
| JR2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/JR2.pdf PDF]
| Jean-Baptiste Rolland
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2014/ace/mirex09.csv</csv>

===Comparative Statistics===

* ''coming soon...''

===Complete Results===

More detailed about the performance of the algorithms, including per-song performance and the breakdown of the WCSR calculations, is available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/MirexChord2009.zip MirexChord2009.zip]

===Algorithmic Output===

The recognition output and the ground-truth files are available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/MirexChord2009Output.zip MirexChord2009Output.zip]

We hope to generate a graphical comparison of all algorithms against the ground truth early in 2014.

2014:Audio Chord Estimation Results

2014-10-20T15:52:26Z

Peter Organisciak: /* Summary */

==Introduction==

This year, we have started a new evaluation battery for audio chord estimation. This page contains the results of these new evaluations for the Isophonics dataset, a.k.a. the MIREX 2009 dataset. It comprises the collected Beatles, Queen, and Zweieck datasets from Queen Mary, University of London, and has been used for audio chord estimation in MIREX for many years.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CB3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB3.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CB4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB4.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CF2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CF2.pdf PDF]
| Chris Cannam, Matthias Mauch, Matthew E. P. Davies, Simon Dixon, Christian Landone, Katy Noland, Mark Levy, Massimiliano Zanoni, Dan Stowell & Luís A. Figueira
|-
| KO1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO1.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| KO2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO2.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| NG1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG1.pdf PDF]
| Nikolay Glazyrin
|-
| NG2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG2.pdf PDF]
| Nikolay Glazyrin
|-
| NMSD1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD1.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| NMSD2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD2.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| PP3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP3.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| PP4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP4.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| SB8
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/SB8.pdf PDF]
| Nikolaas Steenbergen & John Ashley Burgoyne
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2014/ace/mirex09.csv</csv>

===Comparative Statistics===

* ''coming soon...''

===Complete Results===

More detailed about the performance of the algorithms, including per-song performance and the breakdown of the WCSR calculations, is available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/MirexChord2009.zip MirexChord2009.zip]

===Algorithmic Output===

The recognition output and the ground-truth files are available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/MirexChord2009Output.zip MirexChord2009Output.zip]

We hope to generate a graphical comparison of all algorithms against the ground truth early in 2014.

2014:Audio Chord Estimation Results Billboard 2013

2014-10-20T15:50:21Z

Peter Organisciak: /* Summary */

==Introduction==

This year, we have started a new evaluation battery for audio chord estimation. This page contains the results of these new evaluations for a special subset of the ''Billboard'' dataset from McGill University that has never been made available to the public. Further subsets have been withheld to support the ACE task through MIREX 2015.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CB3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB3.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CB4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB4.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CF2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CF2.pdf PDF]
| Chris Cannam, Matthias Mauch, Matthew E. P. Davies, Simon Dixon, Christian Landone, Katy Noland, Mark Levy, Massimiliano Zanoni, Dan Stowell & Luís A. Figueira
|-
| KO1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO1.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| KO2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO2.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| NG1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG1.pdf PDF]
| Nikolay Glazyrin
|-
| NG2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG2.pdf PDF]
| Nikolay Glazyrin
|-
| NMSD1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD1.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| NMSD2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD2.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| PP3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP3.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| PP4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP4.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| SB8
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/SB8.pdf PDF]
| Nikolaas Steenbergen & John Ashley Burgoyne
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2014/ace/billboard13.csv</csv>

===Comparative Statistics===

* ''coming soon...''

===Complete Results===

More detailed about the performance of the algorithms, including per-song performance and the breakdown of the WCSR calculations, is available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2013.zip BillboardTest2013.zip]

===Algorithmic Output===

The recognition output and the ground-truth files are available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2013Output.zip BillboardTest2013Output.zip]

We hope to generate a graphical comparison of all algorithms against the ground truth early in 2014.

2014:Audio Chord Estimation Results Billboard 2012

2014-10-20T15:49:56Z

Peter Organisciak: /* Summary */

==Introduction==

This page contains the results of these new evaluations for an abridged version of the ''Billboard'' dataset from McGill University, including a representative sample of American popular music from the 1950s through the 1990s, as used for MIREX 2012.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CB3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB3.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CB4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB4.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CF2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CF2.pdf PDF]
| Chris Cannam, Matthias Mauch, Matthew E. P. Davies, Simon Dixon, Christian Landone, Katy Noland, Mark Levy, Massimiliano Zanoni, Dan Stowell & Luís A. Figueira
|-
| KO1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO1.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| KO2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO2.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| NG1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG1.pdf PDF]
| Nikolay Glazyrin
|-
| NG2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG2.pdf PDF]
| Nikolay Glazyrin
|-
| NMSD1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD1.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| NMSD2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD2.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| PP3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP3.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| PP4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP4.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| SB8
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/SB8.pdf PDF]
| Nikolaas Steenbergen & John Ashley Burgoyne
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2014/ace/billboard12.csv</csv>

===Comparative Statistics===

* ''coming soon...''

===Complete Results===

More detailed about the performance of the algorithms, including per-song performance and the breakdown of the WCSR calculations, is available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2012.zip BillboardTest2012.zip]

===Algorithmic Output===

The recognition output and the ground-truth files are available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2012Output.zip BillboardTest2012Output.zip]

We hope to generate a graphical comparison of all algorithms against the ground truth early in 2014.

2014:Audio Chord Estimation Results Billboard 2012

2014-10-20T15:40:53Z

Peter Organisciak: /* Introduction */

==Introduction==

This page contains the results of these new evaluations for an abridged version of the ''Billboard'' dataset from McGill University, including a representative sample of American popular music from the 1950s through the 1990s, as used for MIREX 2012.

==Why evaluate differently?==

* Researchers interested in automatic chord estimation have been dissatisfied with the traditional evaluation techniques used for this task at MIREX.

* Numerous alternatives have been proposed in the literature (Harte, 2010; Mauch, 2010; Pauwels & Peeters, 2013).

* At ISMIR 2010 in Utrecht, a group discussed alternatives and developed the [[The_Utrecht_Agreement_on_Chord_Evaluation | Utrecht Agreement]] for updating the task, but until this year, nobody had implemented any of the suggestions.

==What’s new?==

===More precise recall estimation===

* MIREX typically uses ''chord symbol recall'' (CSR) to estimate how well the predicted chords match the ground truth: the total duration of segments where the predictions match the ground truth divided by the total duration of the song.

* In previous years, MIREX has used an approximate CSR by sampling both the ground-truth and the automatic annotations every 10 ms.

* Following Harte (2010), we view the ground-truth and estimated annotations instead as continuous segmentations of the audio because (1) this is more precise and also (2) more computationally efficient.

* Moreover, because pieces of music come in a wide variety of lengths, we believe it is better to weight the CSR by the length of the song. This final number is referred to as the ''weighted chord symbol recall'' (WCSR).

===Advanced chord vocabularies===

* We computed WCSR with five different chord vocabulary mappings:
# Chord root note only;
# Major and minor;
# Seventh chords;
# Major and minor with inversions; and
# Seventh chords with inversions.

* With the exception of no-chords, calculating the vocabulary mapping involves examining the root note, the bass note, and the relative interval structure of the chord labels.

* A mapping exists if both the root notes and bass notes match, and the structure of the output label is the largest possible subset of the input label given the vocabulary.

* For instance, in the major and minor case, G:7(#9) is mapped to G:maj because the interval set of G:maj, {1,3,5}, is a subset of the interval set of the G:7(#9), {1,3,5,b7,#9}. In the seventh-chord case, G:7(#9) is mapped to G:7 instead because the interval set of G:7 {1, 3, 5, b7} is also a subset of G:7(#9) but is larger than G:maj.

* Our recommendations are motivated by the frequencies of chord qualities in the ''Billboard'' corpus of American popular music (Burgoyne et al., 2011).
{| class="wikitable"
|+ Most Frequent Chord Qualities in the ''Billboard'' Corpus
|-
! Quality
! Freq.
! Cum. Freq.
|-
| maj
| align="right"| 52
| align="right"| 52
|-
| min
| align="right"| 13
| align="right"| 65
|-
| 7
| align="right"| 10
| align="right"| 75
|-
| min7
| align="right"| 8
| align="right"| 83
|-
| maj7
| align="right"| 3
| align="right"| 86
|}

===Evaluation of segmentation===

* The chord transcription literature includes several other evaluation metrics, which mainly focus on the segmentation of the transcription.

* We propose to include the directional Hamming distance in the evaluation. The directional Hamming distance is calculated by finding for each annotated segment the maximally overlapping segment in the other annotation, and then summing the differences (Abdallah et al., 2005; Mauch, 2010).

* Depending on the order of application, the directional Hamming distance yields a measure of over- or under-segmentation. To keep the scaling consistent with WCSR values (1.0 is best and 0.0 is worst), we report 1 – over-segmentation and 1 – under-segmentation, as well as the harmonic mean of these values (cf. Harte, 2010).

===Comparative Statistics===

* ''coming soon...''

==Submissions==

{| class="wikitable"
!
! Abstract
! Contributors
|-
| CB3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB3.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CB4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CB4.pdf PDF]
| Taemin Cho & Juan P. Bello
|-
| CF2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CF2.pdf PDF]
| Chris Cannam, Matthias Mauch, Matthew E. P. Davies, Simon Dixon, Christian Landone, Katy Noland, Mark Levy, Massimiliano Zanoni, Dan Stowell & Luís A. Figueira
|-
| KO1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO1.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| KO2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/KO2.pdf PDF]
| Maksim Khadkevich & Maurizio Omologo
|-
| NG1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG1.pdf PDF]
| Nikolay Glazyrin
|-
| NG2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NG2.pdf PDF]
| Nikolay Glazyrin
|-
| NMSD1
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD1.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| NMSD2
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/NMSD2.pdf PDF]
| Yizhao Ni, Matt Mcvicar, Raul Santos-Rodriguez & Tijl De Bie
|-
| PP3
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP3.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| PP4
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/PP4.pdf PDF]
| Johan Pauwels & Geoffroy Peeters
|-
| SB8
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/SB8.pdf PDF]
| Nikolaas Steenbergen & John Ashley Burgoyne
|}

==Results==

===Summary===

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best). The table is sorted on WCSR for the major-minor vocabulary. Algorithms that conducted training are marked with an asterisk; all others were submitted pre-trained.

<csv>2013/ace/billboard12.csv</csv>

===Comparative Statistics===

* ''coming soon...''

===Complete Results===

More detailed about the performance of the algorithms, including per-song performance and the breakdown of the WCSR calculations, is available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2012.zip BillboardTest2012.zip]

===Algorithmic Output===

The recognition output and the ground-truth files are available from this archive:

* [https://music-ir.org/mirex/results/2013/ace/BillboardTest2012Output.zip BillboardTest2012Output.zip]

We hope to generate a graphical comparison of all algorithms against the ground truth early in 2014.

2014:MIREX2014 Results

2014-10-20T14:57:02Z

Peter Organisciak: /* Results by Task */

==Results by Task ==

===Train-Test Task Set===
* [https://www.music-ir.org/nema_out/mirex2014/results/act/composer_report/ Audio Classical Composer Identification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/latin_report/ Audio Latin Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/mood_report/index.html Audio Music Mood Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/kgenrek_report/ Audio KPOP Genre (Annotated by Korean Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/kgenrea_report/ Audio KPOP Genre (Annotated by American Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/kmoodk_report/ Audio KPOP Mood (Annotated by Korean Annotators) Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2014/results/act/kmooda_report/ Audio KPOP Mood (Annotated by American Annotators) Classification Results ]  

===Other Tasks===

* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/mrx09/ MIREX09 dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/sal/ SALAMI dataset]  

* [https://nema.lis.illinois.edu/nema_out/mirex2014/results/aod/ Audio Onset Detection Results]  

* [[2014:Audio Cover Song Identification Results|Audio Cover Song Identification Results]]

* [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ate/ Audio Tempo Estimation Results]  

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/abt/smc/ SMC Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/abt/maz/ MAZ Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/abt/mck/ MCK Dataset]  

* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/adc04/ ADC04 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/mrx05/ MIREX05 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/ind08/ INDIAN08 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/mrx09_0db/ MIREX09 0dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset]  

* Audio Tag Classification Results
** Major Miner Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)]  
*** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/atg/subtask1_report/aff/ Affinity estimation evaluation]  
** Mood Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)]  
*** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/atg/subtask2_report/aff/ Affinity estimation evaluation]  

* [[2014:Audio Downbeat Estimation Results|Audio Downbeat Estimation Results]]

* [https://www.music-ir.org/mirex/wiki/2014:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results#Summary_Results Real-time Audio to Score Alignment (a.k.a. Score Following) Results]  

* [https://www.music-ir.org/mirex/wiki/2014:Audio_Music_Similarity_and_Retrieval_Results Audio Music Similarity and Retrieval Results] 
* [[2014:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]
* [https://www.music-ir.org/mirex/wiki/2014:Singing_Voice_Separation_Results#For_the_Music_Accompaniment_.28dB.29 Singing Voice Separation] 

* [https://www.music-ir.org/mirex/wiki/2014:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results Multiple Fundamental Frequency Estimation & Tracking Results] 

* [[2014:Discovery of Repeated Themes & Sections Results | Discovery of Repeated Themes & Sections Results]]

2014:Audio Cover Song Identification Results

2014-10-08T17:35:26Z

Peter Organisciak: /* Sapp's Mazurka Collection */

== Introduction ==

Audio Cover Song (ACS) Identification was evaluated against two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
!NC2
| NC2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/NC2.pdf PDF] || Ning Chen
|-
! NC3
| NC3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/NC2.pdf PDF] || Ning Chen
|-

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2014/acs/coversong1000.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2014/acs/coversong1000.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2014/acs/coversong1000.precision.groups.csv</csv>

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2014/acs/mazurkas.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2014/acs/mazurkas.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2014/acs/mazurkas.precision.groups.csv</csv>

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''NC2''' : [https://music-ir.org/mirex/results/2014/acs/NC2.coversong1000.avgprec.txt Ning Chen] 

'''NC3''' : [https://music-ir.org/mirex/results/2014/acs/NC3.coversong1000.avgprec.txt Ning Chen] 

'''Rank Lists'''

'''NC2''' : [https://music-ir.org/mirex/results/2014/acs/NC2.coversong1000.ranklist.txt Ning Chen] 

'''NC3''' : [https://music-ir.org/mirex/results/2014/acs/NC3.coversong1000.ranklist.txt Ning Chen] 

===Mixed Collection===
'''Average Precision by Query'''

'''NC2''' : [https://music-ir.org/mirex/results/2014/acs/NC2.mazurkas.avgprec.txt Ning Chen] 

'''NC3''' : [https://music-ir.org/mirex/results/2014/acs/NC3.mazurkas.avgprec.txt Ning Chen] 

'''Rank Lists'''

'''NC2''' : [https://music-ir.org/mirex/results/2014/acs/NC2.mazurkas.ranklist.txt Ning Chen] 

'''NC3''' : [https://music-ir.org/mirex/results/2014/acs/NC3.mazurkas.ranklist.txt Ning Chen]

2014:Audio Cover Song Identification Results

2014-10-08T17:28:46Z

Peter Organisciak: /* Mixed Collection */

== Introduction ==

Audio Cover Song (ACS) Identification was evaluated against two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
!NC2
| NC2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/NC2.pdf PDF] || Ning Chen
|-
! NC3
| NC3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/NC2.pdf PDF] || Ning Chen
|-

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2014/acs/coversong1000.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2014/acs/coversong1000.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2014/acs/coversong1000.precision.groups.csv</csv>

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2014/acs/mazurkas.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2014/acs/mazurkas.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2014/acs/mazurkas.precision.groups.csv</csv>

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''NC2''' : [https://music-ir.org/mirex/results/2014/acs/NC2.coversong1000.avgprec.txt Ning Chen] 

'''NC3''' : [https://music-ir.org/mirex/results/2014/acs/NC3.coversong1000.avgprec.txt Ning Chen] 

'''Rank Lists'''

'''NC2''' : [https://music-ir.org/mirex/results/2014/acs/NC2.coversong1000.ranklist.txt Ning Chen] 

'''NC3''' : [https://music-ir.org/mirex/results/2014/acs/NC3.coversong1000.ranklist.txt Ning Chen] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.mazurkas.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.mazurkas.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.mazurkas.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.mazurkas.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 
[[Category: Results]]

2014:Audio Cover Song Identification Results

2014-10-08T17:26:45Z

Peter Organisciak: /* Sapp's Mazuraka Collection */

== Introduction ==

Audio Cover Song (ACS) Identification was evaluated against two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
!NC2
| NC2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/NC2.pdf PDF] || Ning Chen
|-
! NC3
| NC3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/NC2.pdf PDF] || Ning Chen
|-

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2014/acs/coversong1000.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2014/acs/coversong1000.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2014/acs/coversong1000.precision.groups.csv</csv>

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2014/acs/mazurkas.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2014/acs/mazurkas.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2014/acs/mazurkas.precision.groups.csv</csv>

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.coversong1000.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.coversong1000.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.coversong1000.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.coversong1000.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.mazurkas.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.mazurkas.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.mazurkas.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.mazurkas.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 
[[Category: Results]]

2014:Audio Cover Song Identification Results

2014-10-08T17:26:08Z

Peter Organisciak: /* Mixed Collection */

== Introduction ==

Audio Cover Song (ACS) Identification was evaluated against two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
!NC2
| NC2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/NC2.pdf PDF] || Ning Chen
|-
! NC3
| NC3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/NC2.pdf PDF] || Ning Chen
|-

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2014/acs/coversong1000.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2014/acs/coversong1000.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2014/acs/coversong1000.precision.groups.csv</csv>

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2013/acs/mazurkas.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2013/acs/mazurkas.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2013/acs/mazurkas.precision.groups.csv</csv>

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.coversong1000.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.coversong1000.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.coversong1000.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.coversong1000.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.mazurkas.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.mazurkas.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.mazurkas.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.mazurkas.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 
[[Category: Results]]

2014:Audio Cover Song Identification Results

2014-10-08T16:38:38Z

Peter Organisciak: /* General Legend */

== Introduction ==

Audio Cover Song (ACS) Identification was evaluated against two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
!NC2
| NC2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/NC2.pdf PDF] || Ning Chen
|-
! NC3
| NC3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/NC2.pdf PDF] || Ning Chen
|-

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2013/acs/coversong1000.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2013/acs/coversong1000.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2013/acs/coversong1000.precision.groups.csv</csv>

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2013/acs/mazurkas.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2013/acs/mazurkas.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2013/acs/mazurkas.precision.groups.csv</csv>

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.coversong1000.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.coversong1000.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.coversong1000.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.coversong1000.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.mazurkas.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.mazurkas.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.mazurkas.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.mazurkas.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 
[[Category: Results]]

2014:Audio Cover Song Identification Results

2014-10-08T16:37:54Z

Peter Organisciak:

== Introduction ==

Audio Cover Song (ACS) Identification was evaluated against two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
!NC2
| NC2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/NC2.pdf PDF] || Ning Chen
! NC3
| NC3 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2014/NC2.pdf PDF] || Ning Chen

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2013/acs/coversong1000.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2013/acs/coversong1000.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2013/acs/coversong1000.precision.groups.csv</csv>

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2013/acs/mazurkas.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2013/acs/mazurkas.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2013/acs/mazurkas.precision.groups.csv</csv>

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.coversong1000.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.coversong1000.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.coversong1000.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.coversong1000.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.mazurkas.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.mazurkas.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.mazurkas.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.mazurkas.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 
[[Category: Results]]

2014:MIREX2014 Results

2014-10-08T16:30:03Z

Peter Organisciak: /* Other Tasks */

==Results by Task ==

===Other Tasks===

* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/mrx09/ MIREX09 dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2014/results/struct/sal/ SALAMI dataset]  

* [https://nema.lis.illinois.edu/nema_out/mirex2014/results/aod/ Audio Onset Detection Results]  

* [[2014:Audio Cover Song Identification Results]]

2014:MIREX2014 Results

2014-09-24T18:20:36Z

Peter Organisciak:

MIREX 2014 Submission Instructions

2014-08-14T22:56:51Z

Peter Organisciak: /* Submission System URL */

==Some Reminders==
* Be sure to read through the rest of this page
* Be sure to read through the [[2014:Main_Page| 2014 MIREX Home]] page
* Be sure to read through the task pages for which you are submitting
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]

==Begin with the Video Tutorial==
Go watch the [https://www.music-ir.org//mirex/2010/submission_tutorial/ MIREX 2012 Submission System Video Tutorial]

==Basic Steps==

# Tell us about yourself by creating an identity profile. If you are participating under multiple affiliations, repeat this step for each affiliation.
# Create a submission record. You'll need to add all your contributors. This is easiest if they have also registered and completed step 1, but you can create profiles for them when you create your submission.
# Upload your submission via SFTP to the dropbox. Specific instructions are given after you complete your submission.
# Upload your abstract via the webform.

==Very Important Things to Note==

# NOTA BENE: We are REQUIRING that EACH person involved in a MIREX 2014 submission MUST create an identity for themselves on the submission system. Identities are important to us as they help us better manage the submissions. Even if a colleague of yours is going to do the actual submitting, you still need to create an identity for yourself in the system.
# When you create your personal identity in the system, review your input carefully for errors! Once your personal identity is created and the "submit" button is pressed, it is not possible for you to edit your identity information.
# If you are submitting on behalf of a team you will need to make sure that the identity for each team member is associated with your submission. Your first job is to find out if they have already created identities in the system by using the search tool. If they have, simply click on the identity to add them.
# If you cannot find an identity for one or more of your colleagues, the best way to proceed is get them to create an identity for themselves on the system. This way, they are responsible for the accuracy of their information.
# If your colleague, for some reason, cannot create an identity for themselves, you will need to create an identity for them. Do your best to create as accurate an identity for them as possible.
# If you plan to submit more than one algorithm or algorithm variant to a given task, EACH algorithm or variant needs its own complete submission to be made including the README and binary bundle upload. Each package will be given its own unique identifier. Tell us in the README the priority of a given algorithm in case we have to limit a task to only one or two algorithms/variants per submitter/team.

==Getting Help==
If things do not work or if you make a major mistake or if you are simply confused, please contact the MIREX team at mirex [at] imirsel.org.

==Participant Identity Information Fields==
(* = required field)
*First name*:
*Last name*:
*Organization*:
*Department:
*Unit/Lab:
*URL*:
*Title*:
*From (year)*: To:
*Email:
*Street Address:
*Street Address 2:
*Street Address 3:
*City:
*State, Region:
*Postal Code:
*Country:

==Extended Abstract Details==
The extended abstracts provide the outside world with a general understanding of what each submission is trying to accomplish. The extended abstracts need NOT be cutting edge/never-before-published materials. The extended abstracts will be revised by the authors after the data has been collected (to allow for commentary on results data); however, we at MIREX still need the first-pass drafts at submission time to help us understand what is happening in the submission. Like last year we will post the final versions of the extended abstracts as part of the MIREX 2014 results page.

The MIREX 2014 extended abstracts:
# Are two to four pages long.
# Must conform to the guidelines in the following templates: [https://www.music-ir.org/mirex/templates/2010/MIREX2010_tex_template.zip LaTeX template] [https://www.music-ir.org/mirex/templates/2010/MIREX2010_doc_template.zip Word template]
# Must be submitted in PDF format.
# Should include, if exists, references to other publications about your work (yes, self-reference is encouraged!)
# Should have the same general look and feel as these examples from last year:

==Submission System URL==
The MIREX 2014 Submission System can be found at: https://www.music-ir.org/mirex/sub/ .

MIREX 2014 Submission Instructions

2014-08-14T22:56:43Z

Peter Organisciak: /* Submission System URL */

2014:GC14UX

2014-07-01T02:24:00Z

Peter Organisciak: /* Task */

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.
=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.
=Evaluation=
==Task==

Evaluators are given the following task:

''You are creating a short video about a memorable occasion that happened to you recently, and you need to find some songs to use as background music.''

==Criteria==

''Note that the evaluation criteria or its descriptions may change in the months leading up to the submission deadline, as we test it and work to improve it.''

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How good is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for users to give feedback if they wish to do so.

This simplicity is because: 1) the GC14UX is all about how users perceive their experiences of the systems. We intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. 2) more data capturing opportunities will distract from the real user experience.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T02:15:47Z

Peter Organisciak:

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.
=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.
=Evaluation=
==Task==

Evaluators are given the following task:

''You are creating a short video about what you did this summer and you need to find some songs to use as background music.''

==Criteria==

''Note that the evaluation criteria or its descriptions may change in the months leading up to the submission deadline, as we test it and work to improve it.''

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* '''Affordances''': How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How good is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for users to give feedback if they wish to do so.

This simplicity is because: 1) the GC14UX is all about how users perceive their experiences of the systems. We intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. 2) more data capturing opportunities will distract from the real user experience.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

2014:GC14UX

2014-07-01T02:14:20Z

Peter Organisciak: Update to newest version

{{DISPLAYTITLE:Grand Challenge on User Experience 2014}}
=Welcome to GC14UX=
Grand Challenge on User Experience 2014
=Purpose=
Holistic evaluation of user experience in interacting with user-serving MIR systems.
=Goals=
# to inspire the development of complete MIR systems.
# to promote the notion of user experience as a first-class research objective in the MIR community.
=Dataset=
A set of music 10,000 music audio tracks is provided for the GC14UX. It will be a subset of tracks drawn from the [http://www.jamendo.com/en/welcome Jamendo collection's] CC-BY licensed works.

The Jamendo collection contains music in a variety of genres and moods, but is mostly unknown to most listeners. This will mitigate against the possible user experience bias induced by the differential presence (or absence) of popular or known music within the participating systems.

As of May 20, 2014, the Jamendo collection contains 14742 tracks with the [http://creativecommons.org/licenses/by/3.0/ CC-BY license]. The CC-BY license allows others to distribute, modify, optimize and use your work as a basis, even commercially, as long as you give credit for the original creation. This is one of the most permissive licenses possible.

The 10,000 tracks in GC14UX will be sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems.
=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

==Potential Participants==
{| class="wikitable"
|-
! (Cool) Team Name
! Name(s)
! Email(s)
|-
| The MIR UX Master
| Dr. MIR
| mir@domain.com
|-
|}

=Participating Systems=
Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC14UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: '''1024x768'''. Please test your system for this screen size.

=Evaluation=
==Task==

Evaluators are given the following task:

''You are creating a short video about what you did this summer and you need to find some songs to use as background music.''

==Criteria==

''Note that the evaluation criteria or its descriptions may change in the months leading up to the submission deadline, as we test it and work to improve it.''

* '''Overall satisfaction''': Overall, how pleasurable do you find the experience of using this system?
Extremely unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Extremely satisfactory

* '''Learnability''': How easy was it to figure out how to use the system?
Extremely difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Extremely easy

* '''Robustness''': How good is the system’s ability to warn you when you’re about to make a mistake and allow you to recover?
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good ||| Not Applicable

* ‘’’Affordances’’’: How well does the system allow you to perform what you want to do?

* '''Presentation''': How well does the system communicate what’s going on? (How well do you feel the system informs you of its status? Can you clearly understand the labels and words used in the system? How visible are all of your options and menus when you use this system?)

* '''Aesthetics''': How good is the design? (Is it aesthetically pleasing?)
Extremely Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Extremely Good

* '''Feedback''' (Optional): An open-ended question is provided but is optional for users to give feedback if they wish to do so.

This simplicity is because: 1) the GC14UX is all about how users perceive their experiences of the systems. We intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. 2) more data capturing opportunities will distract from the real user experience.

==Evaluation mechanism==
The GC14UX team will provide a set of evaluation forms which wrap around the participating system. In other words, the evaluation system will offer forms for scoring the participating system, and embed the system within an iframe.

==Evaluators==
Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The evaluation webforms developed by the GC14UX team will ensure all participating systems will get equal number of evaluators.

==Evaluation results==
Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

=Wireframes=
[[File:GCUX wireframe evaluation.png|900px]]

=Important Dates=

*July 1: announce the GC
*Sep. 21st: deadline for system submission
*Sep. 28th: start the evaluation
*Oct. 20th: close the evaluation system
*Oct. 27th: announce the results
*Oct. 31st: MIREX and GC session in ISMIR2014

File:GCUX wireframe 2014 06 16.png

2014-06-25T19:36:47Z

Peter Organisciak: uploaded a new version of "File:GCUX wireframe 2014 06 16.png"

2013:MIREX2013 Results

2013-11-04T22:47:44Z

Peter Organisciak:

==OVERALL RESULTS POSTERS ==

This page is under construction.

[https://www.music-ir.org/mirex/results/2013/mirex_2013_poster.pdf MIREX 2013 Overall Results Posters (PDF)]

==Results by Task ==

===Train-Test Task Set===
* [https://www.music-ir.org/nema_out/mirex2013/results/act/composer_report/ Audio Classical Composer Identification Results ]  
* [https://www.music-ir.org/nema_out/mirex2013/results/act/latin_report/ Audio Latin Genre Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2013/results/act/mood_report/index.html Audio Music Mood Classification Results ]  
* [https://www.music-ir.org/nema_out/mirex2013/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]  
===Other Tasks===

* Audio Beat Tracking Results
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/abt/dav/ DAV Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/abt/maz/ MAZ Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/abt/mck/ MCK Dataset]  
* Audio Chord Detection Results (will add more results soon)
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ace/mrx09/index.html MIREX Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ace/bill/index.html McGill Dataset]  
* [https://nema.lis.illinois.edu/nema_out/mirex2013/results/akd/ Audio Key Detection Results]  
* Audio Melody Extraction Results
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ame/adc04/ ADC04 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ame/mrx05/ MIREX05 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ame/ind08/ INDIAN08 Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ame/mrx09_0db/ MIREX09 0dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset]  
* [[2013:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]
* [https://nema.lis.illinois.edu/nema_out/mirex2013/results/aod/ Audio Onset Detection Results]  
* Audio Tag Classification Results
** Major Miner Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)]  
*** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/atg/subtask1_report/aff/ Affinity estimation evaluation]  
** Mood Tag dataset
*** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)]  
*** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/atg/subtask2_report/aff/ Affinity estimation evaluation]  
* [https://nema.lis.illinois.edu/nema_out/mirex2013/results/ate/ Audio Tempo Estimation Results]  
* [[2013:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]
* Music Structure Segmentation Results
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/struct/mrx09/ MIREX09 dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/struct/sal/ SALAMI dataset]  
* Query-by-Singing/Humming Results
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbsh/qbsh_task1_hidden/ Hidden Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbsh/qbsh_task1a_jang/ Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbsh/qbsh_task1b_thinkit/ ThinkIt Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbsh/qbsh_task1c_ioacas/ IOACAS Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbsh/qbsh_task2_jang/ Subtask2 Dataset]  
* Query-by-Tapping Results
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbt/qbt_task1_jang/ Jang Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbt/qbt_task1_hsiao/ HSIAO Dataset]  
** [https://nema.lis.illinois.edu/nema_out/mirex2013/results/qbt/qbt_task2_jang/ Subtask2 Dataset]  
*[[2013:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]
* [[2013:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]
* [[2013:Discovery of Repeated Themes & Sections Results | Discovery of Repeated Themes & Sections Results]]
* [[2013:Audio Cover Song Identification Results]]

2013:Audio Cover Song Identification Results

2013-11-04T22:35:54Z

== Introduction ==

Audio Cover Song (ACS) Identification was evaluated against two datasets: ''Mixed Collection'' and ''Sapp's Mazurka Collection''.

===Mixed Collection Information===
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.

Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.

===Sapp's Mazurka Collection Information ===
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.

== General Legend ==
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"
|- style="background: yellow;"
! width="80" | Sub code
! width="200" | Submission name
! width="80" style="text-align: center;" | Abstract
! width="440" | Contributors
|-
! YCW1
| MCY_COVER || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/YCW1.pdf PDF] || [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://slam.iis.sinica.edu.tw/ Chuan-Yau Chan], [http://www.iis.sinica.edu.tw Hsin-Min Wang]
|-
! CYWW1
| CYC_COVER || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2013/CYWW1.pdf PDF] || [http://slam.iis.sinica.edu.tw/ Chuan-Yau Chan], [http://slam.iis.sinica.edu.tw/index.htm Ming-Chi Yen], [http://sovideo.iis.sinica.edu.tw/SLG/index.html Ju-Chiang Wang], [http://www.iis.sinica.edu.tw Hsin-Min Wang]

|}

== Results ==

===Mixed Collection===

====Summary Results====
<csv p=2>2013/acs/coversong1000.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2013/acs/coversong1000.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2013/acs/coversong1000.precision.groups.csv</csv>

===Sapp's Mazuraka Collection===

====Summary results====
<csv p=2>2013/acs/mazurkas.summary.csv</csv>

====Number of Correct Covers at Rank X Returned in Top Ten====
<csv>2013/acs/mazurkas.toptendist.csv</csv>

====Average Performance per Query Group====
<csv p=2>2013/acs/mazurkas.precision.groups.csv</csv>

==Individual Results Files==

===Mixed Collection===
'''Average Precision by Query'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.coversong1000.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.coversong1000.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.coversong1000.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.coversong1000.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

===Sapp's Mazurka Collection===

'''Average Precision by Query'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.mazurkas.avgprec.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.mazurkas.avgprec.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 

'''Rank Lists'''

'''YCW1''' : [https://music-ir.org/mirex/results/2013/acs/YCW1.mazurkas.ranklist.txt Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang] 

'''CYWW1''' : [https://music-ir.org/mirex/results/2013/acs/CYWW1.mazurkas.ranklist.txt Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang] 
[[Category: Results]]