2010:Audio Chord Estimation - Revision history

Kahyun Choi: /* Segmentation Score */

2011-12-15T06:37:23Z

‎Segmentation Score

MertBay: /* I/O Format */

2010-06-14T17:29:31Z

‎I/O Format

MertBay at 17:28, 14 June 2010

2010-06-14T17:28:54Z

Kriswest: /* Submission opening date */

2010-06-05T09:26:23Z

‎Submission opening date

Kriswest: Proofing and reordering full proposal text

2010-05-28T16:30:59Z

Proofing and reordering full proposal text

Show changes

Kriswest: /* Data */

2010-05-28T16:15:39Z

‎Data

MertBay at 21:33, 26 May 2010

2010-05-26T21:33:40Z

MertBay at 21:32, 26 May 2010

2010-05-26T21:32:29Z

MertBay: /* Evaluation */

2010-05-26T21:32:11Z

‎Evaluation

MertBay: /* Discussions from 2009 */

2010-03-23T20:03:48Z

‎Discussions from 2009

@@ Line 40: / Line 40: @@
 segmentation score = 1 - max(m,f)
-m and f are not independent of each other so combining them this way ensures that a good score in one does not hide a bad score in the other. The combined segmentation score can take values between 0 and 1 with 0 being the worst and 1 being the best result.--[[User:Chrish|Chrish]] 17:05, 9 September 2009 (UTC)
+m and f are not independent of each other so combining them this way ensures that a good score in one does not hide a bad score in the other. The combined segmentation score can take values between 0 and 1 with 0 being the worst and 1 being the best result.-- Chrish 17:05, 9 September 2009 (UTC)
 === Frame-based recall ===

@@ Line 270: / Line 270: @@
 We still accept participants who would only like to be evaluated on major/minor chords and want to use the number format which is an integer chord id on range 0-24, where values 0-11  denote the C major, C# major, ..., B major  and  12-23 denote the C minor, C# minor, ..., B minor and         24    denotes silence or no-chord segments. '''Please note that the format is still the same'''
-  start_time end_time chord_label
+  start_time end_time chord_number
 === Command line calling format ===

← Older revision		Revision as of 17:28, 14 June 2010
Line 268:		Line 268:
	Please note that two things have changed in the syntax since it was originally described in [6]. The first change is that the root is no longer implied as a voiced element of a chord so a C major chord (notes C, E and G) should be written C:(1,3,5) instead of just C:(3,5) if using the interval list representation. As before, the labels C and C:maj are equivalent to C:(1,3,5). The second change is that the shorthand label "sus2" (intervals 1,2,5) has been added to the available shorthand list.--[[User:Chrish\|Chrish]] 17:05, 9 September 2009 (UTC)		Please note that two things have changed in the syntax since it was originally described in [6]. The first change is that the root is no longer implied as a voiced element of a chord so a C major chord (notes C, E and G) should be written C:(1,3,5) instead of just C:(3,5) if using the interval list representation. As before, the labels C and C:maj are equivalent to C:(1,3,5). The second change is that the shorthand label "sus2" (intervals 1,2,5) has been added to the available shorthand list.--[[User:Chrish\|Chrish]] 17:05, 9 September 2009 (UTC)

−	~~However, we~~ still accept participants who would only like to be evaluated on major/minor chords and want to use the ~~MIREX 2008~~ format which is an integer chord id on range 0-24, where values 0-11 denote the C major, C# major, ..., B major and 12-23 denote the C minor, C# minor, ..., B minor and 24 denotes silence or no-chord segments	+	We still accept participants who would only like to be evaluated on major/minor chords and want to use the number format which is an integer chord id on range 0-24, where values 0-11 denote the C major, C# major, ..., B major and 12-23 denote the C minor, C# minor, ..., B minor and 24 denotes silence or no-chord segments. '''Please note that the format is still the same'''
		+
		+	start_time end_time chord_label
		+
		+	Systems are supposed to print out the onset-offset times as opposed to MIREX 2008 chord output format where only onset were used.
		+

@@ Line 308: / Line 308: @@
 == Submission opening date ==
-TBA
+Friday 4th June 2010
 == Submission closing date ==

@@ Line 14: / Line 14: @@
 == Data ==
-Christopher Harte`s Beatles dataset is used for the evaluations last year. This dataset consists of 12 Beatles albums [6].
+=== Beatles dataset ===
-An approach for text annotation of musical chords is presented in [6].
+Christopher Harte`s Beatles dataset consisting of annotations of 12 Beatles albums.
-.2631021 44.2456460 B
+The text annotation procedure of musical chords that was used to produce this dataset is presented in [6].
-.2456460 45.7201130 E
+=== Queen and Zweieck dataset ===
+Matthias Mauch's Queen and Zweieck dataset consisting of 38 songs from Queen and Zweieck.
 == I/O Format ==

@@ Line 281: / Line 281: @@
 Programs can use their working directory if they need to keep temporary cache files or internal debuggin info. Stdout and stderr will be logged.
-== Discussions from 2009 ==
+All submissions should include a README file including the following the
-https://www.music-ir.org/mirex/2009/index.php/Audio_Chord_Detection#Discussions
+* Command line calling format for all executables and an example formatted set of commands
 == Bibliography ==

← Older revision		Revision as of 21:32, 26 May 2010
Line 161:		Line 161:

	However, we still accept participants who would only like to be evaluated on major/minor and want to use last year`s format which is an integer chord id on range 0-24, where values 0-11 denote the C major, C# major, ..., B major and 12-23 denote the C minor, C# minor, ..., B minor and 24 denotes silence or no-chord segments		However, we still accept participants who would only like to be evaluated on major/minor and want to use last year`s format which is an integer chord id on range 0-24, where values 0-11 denote the C major, C# major, ..., B major and 12-23 denote the C minor, C# minor, ..., B minor and 24 denotes silence or no-chord segments
		+
		+
		+	== Evaluation ==
		+
		+	Algorithms should output text files with a similar format to that used in the ground truth transcriptions. That is to say, they should be flat text files with chord segment labels and times arranged thus:
		+
		+	start_time end_time chord_label
		+
		+	with elements separated by white spaces, times given in seconds, chord labels corresponding to the syntax described in [6] and one chord segment per line.
		+
		+	Please note that two things have changed in the syntax since it was originally described in [6]. The first change is that the root is no longer implied as a voiced element of a chord so a C major chord (notes C, E and G) should be written C:(1,3,5) instead of just C:(3,5) if using the interval list representation. As before, the labels C and C:maj are equivalent to C:(1,3,5). The second change is that the shorthand label "sus2" (intervals 1,2,5) has been added to the available shorthand list.--[[User:Chrish\|Chrish]] 17:05, 9 September 2009 (UTC)
		+
		+
		+	=== Segmentation Score ===
		+
		+	The segmentation score will be calculated using directional hamming distance as described in [8]. An over-segmentation value (m) and an under-segmentation value (f) will be calculated and the final segmentation score will be calculated using the worst case from these two i.e:
		+
		+	segmentation score = 1 - max(m,f)
		+
		+	m and f are not independent of each other so combining them this way ensures that a good score in one does not hide a bad score in the other. The combined segmentation score can take values between 0 and 1 with 0 being the worst and 1 being the best result.--[[User:Chrish\|Chrish]] 17:05, 9 September 2009 (UTC)
		+
		+	=== Frame-based recall ===
		+
		+
		+	For recall evaluation, we may define a different chord dictionary for each level of evaluation (dyads, triads, tetrads etc). Each dictionary is a text file containing chord shorthands / interval lists of the chords that will be considered in that evaluation. The following dictionaries are proposed:
		+
		+	For dyad comparison of major/minor chords only:
		+
		+	N<br>
		+	X:maj<br>
		+	X:min<br>
		+
		+	For comparison of standard triad chords:
		+
		+	N<br>
		+	X:maj<br>
		+	X:min<br>
		+	X:aug<br>
		+	X:dim<br>
		+	X:sus2<br>
		+	X:sus4<br>
		+
		+	For comparison of tetrad (quad) chords:
		+
		+	N <br>
		+	X:maj <br>
		+	X:min<br>
		+	X:aug<br>
		+	X:dim<br>
		+	X:sus2<br>
		+	X:sus4<br>
		+	X:maj7<br>
		+	X:7<br>
		+	X:maj(9)<br>
		+	X:aug(7) <br>
		+	X:min(7)<br>
		+	X:min7<br>
		+	X:min(9)<br>
		+	X:dim(7)<br>
		+	X:hdim7 <br>
		+	X:sus4(7)<br>
		+	X:sus4(b7)<br>
		+	X:dim7<br>
		+
		+
		+	For each evaluation level, the ground truth annotation is compared against the dictionary. Any chord label not belonging to the current dictionary will be replaced with an "X" in a local copy of the annotation and will not be included in the recall calculation.
		+
		+	Note that the level of comparison in terms of intervals can be varied. For example, in a triad evaluation we can consider the first three component intervals in the chord so that a major (1,3,5) and a major7 (1,3,5,7) will be considered the same chord. For a tetrad (quad) evaluation, we would consider the first 4 intervals so major and major7 would then be considered to be different chords.
		+
		+	For the maj/min evaluation (using the first example dictionary), using an interval comparison of 2 (dyad) will compare only the first two intervals of each chord label. This would map augmented and diminished chords to major and minor respectively (and any other symbols that had a major 3rd or minor 3rd as their first interval). Using an interval comparison of 3 with the same dictionary would keep only those chords that have major and minor triads as their first 3 intervals so augmented and diminished chords would be removed from the evaluation.
		+
		+	After the annotation has been "filtered" using a given dictionary, it can be compared against the machine generated estimates output by the algorithm under test. The chord sequences described in the annotation and estimate text files are sampled at a given frame rate (in this case 10ms per frame) to give two sequences of chord frames which may be compared directly with each other. For calculating a hit or a miss, the chord labels from the current frame in each sequence will be compared. Chord comparison is done by converting each chord label into an ordered list of pitch classes then comparing the two lists element by element. If the lists match to the required number of intervals then a hit is recorded, otherwise the estimate is considered a miss. It should be noted that, by converting to pitch classes in the comparison, this evaluation ignores enharmonic pitch and interval spellings so the following chords (slightly silly example just for illustration) will all evaluate as identical:
		+
		+	C:maj = Dbb:maj = C#:(b1,b3,#4)
		+
		+
		+	Basic recall calculation algorithm:
		+
		+	1) filter annotated transcription using chord dictionary for a defined number of intervals
		+
		+	2) sample annotated transcription and machine estimated transcription at 10ms intervals to create a sequence of annotation frames and estimate frames
		+
		+	3) start at the first frame
		+
		+	4) get chord label for current annotation frame and estimate frame
		+
		+	5) check annotation label:<br>
		+
		+	IF symbol is 'X' (i.e. non-dictionary) <br>
		+
		+	THEN ignore frame (record number of ignored frames)<br>
		+
		+	ELSE compare annotated/estimated chords for the predefined number of intervals <br>
		+	increment hit count if chords match<br>
		+
		+	ENDIF
		+
		+	6) increment frame count
		+
		+	7) go back to 4 until final chord frame
		+	--[[User:Chrish\|Chrish]] 17:05, 9 September 2009 (UTC)
		+

	== Submission Format ==		== Submission Format ==

← Older revision		Revision as of 21:32, 26 May 2010
Line 178:		Line 178:

	Programs can use their working directory if they need to keep temporary cache files or internal debuggin info. Stdout and stderr will be logged.		Programs can use their working directory if they need to keep temporary cache files or internal debuggin info. Stdout and stderr will be logged.
−
−	~~== Evaluation ==~~
−
−	Algorithms should output text files with a similar format to that used in the ground truth transcriptions. That is to say, they should be flat text files with chord segment labels and times arranged thus:
−
−	~~start_time end_time chord_label~~
−
−	~~with elements separated by white spaces, times given in seconds, chord labels corresponding to the syntax described in [6] and one chord segment per line.~~
−
−	Please note that two things have changed in the syntax since it was originally described in [6]. The first change is that the root is no longer implied as a voiced element of a chord so a C major chord (notes C, E and G) should be written C:(1,3,5) instead of just C:(3,5) if using the interval list representation. As before, the labels C and C:maj are equivalent to C:(1,3,5). The second change is that the shorthand label "sus2" (intervals 1,2,5) has been added to the available shorthand list.--[[User:Chrish\|Chrish]] 17:05, 9 September 2009 (UTC)
−
−
−	~~=== Segmentation Score ===~~
−
−	The segmentation score will be calculated using directional hamming distance as described in [8]. An over-segmentation value (m) and an under-segmentation value (f) will be calculated and the final segmentation score will be calculated using the worst case from these two i.e:
−
−	~~segmentation score = 1 - max(m,f)~~
−
−	m and f are not independent of each other so combining them this way ensures that a good score in one does not hide a bad score in the other. The combined segmentation score can take values between 0 and 1 with 0 being the worst and 1 being the best result.--[[User:Chrish\|Chrish]] 17:05, 9 September 2009 (UTC)
−
−	~~=== Frame-based recall ===~~
−
−
−	For recall evaluation, we may define a different chord dictionary for each level of evaluation (dyads, triads, tetrads etc). Each dictionary is a text file containing chord shorthands / interval lists of the chords that will be considered in that evaluation. The following dictionaries are proposed:
−
−	~~For dyad comparison of major/minor chords only:~~
−
−	~~N<br>~~
−	~~X:maj<br>~~
−	~~X:min<br>~~
−
−	~~For comparison of standard triad chords:~~
−
−	~~N<br>~~
−	~~X:maj<br>~~
−	~~X:min<br>~~
−	~~X:aug<br>~~
−	~~X:dim<br>~~
−	~~X:sus2<br>~~
−	~~X:sus4<br>~~
−
−	~~For comparison of tetrad (quad) chords:~~
−
−	~~N <br>~~
−	~~X:maj <br>~~
−	~~X:min<br>~~
−	~~X:aug<br>~~
−	~~X:dim<br>~~
−	~~X:sus2<br>~~
−	~~X:sus4<br>~~
−	~~X:maj7<br>~~
−	~~X:7<br>~~
−	~~X:maj(9)<br>~~
−	~~X:aug(7) <br>~~
−	~~X:min(7)<br>~~
−	~~X:min7<br>~~
−	~~X:min(9)<br>~~
−	~~X:dim(7)<br>~~
−	~~X:hdim7 <br>~~
−	~~X:sus4(7)<br>~~
−	~~X:sus4(b7)<br>~~
−	~~X:dim7<br>~~
−
−
−	For each evaluation level, the ground truth annotation is compared against the dictionary. Any chord label not belonging to the current dictionary will be replaced with an "X" in a local copy of the annotation and will not be included in the recall calculation.
−
−	Note that the level of comparison in terms of intervals can be varied. For example, in a triad evaluation we can consider the first three component intervals in the chord so that a major (1,3,5) and a major7 (1,3,5,7) will be considered the same chord. For a tetrad (quad) evaluation, we would consider the first 4 intervals so major and major7 would then be considered to be different chords.
−
−	For the maj/min evaluation (using the first example dictionary), using an interval comparison of 2 (dyad) will compare only the first two intervals of each chord label. This would map augmented and diminished chords to major and minor respectively (and any other symbols that had a major 3rd or minor 3rd as their first interval). Using an interval comparison of 3 with the same dictionary would keep only those chords that have major and minor triads as their first 3 intervals so augmented and diminished chords would be removed from the evaluation.
−
−	After the annotation has been "filtered" using a given dictionary, it can be compared against the machine generated estimates output by the algorithm under test. The chord sequences described in the annotation and estimate text files are sampled at a given frame rate (in this case 10ms per frame) to give two sequences of chord frames which may be compared directly with each other. For calculating a hit or a miss, the chord labels from the current frame in each sequence will be compared. Chord comparison is done by converting each chord label into an ordered list of pitch classes then comparing the two lists element by element. If the lists match to the required number of intervals then a hit is recorded, otherwise the estimate is considered a miss. It should be noted that, by converting to pitch classes in the comparison, this evaluation ignores enharmonic pitch and interval spellings so the following chords (slightly silly example just for illustration) will all evaluate as identical:
−
−	~~C:maj = Dbb:maj = C#:(b1,b3,#4)~~
−
−
−	~~Basic recall calculation algorithm:~~
−
−	~~1) filter annotated transcription using chord dictionary for a defined number of intervals~~
−
−	~~2) sample annotated transcription and machine estimated transcription at 10ms intervals to create a sequence of annotation frames and estimate frames~~
−
−	~~3) start at the first frame~~
−
−	~~4) get chord label for current annotation frame and estimate frame~~
−
−	~~5) check annotation label:<br>~~
−
−	~~IF symbol is 'X' (i.e. non-dictionary) <br>~~
−
−	~~THEN ignore frame (record number of ignored frames)<br>~~
−
−	~~ELSE compare annotated/estimated chords for the predefined number of intervals <br>~~
−	~~increment hit count if chords match<br>~~
−
−	~~ENDIF~~
−
−	~~6) increment frame count~~
−
−	~~7) go back to 4 until final chord frame~~
−	~~--[[User:Chrish\|Chrish]] 17:05, 9 September 2009 (UTC)~~

	== Discussions for 2010 ==		== Discussions for 2010 ==

@@ Line 284: / Line 284: @@
 == Discussions from 2009 ==
-Points to discuss:
+https://www.music-ir.org/mirex/2009/index.php/Audio_Chord_Detection#Discussions
 == Potential Participants ==