2009:Structural Segmentation - Revision history

Kahyun Choi: /* Issues and Discussion */

2011-12-15T06:35:11Z

‎Issues and Discussion

Kahyun Choi: /* Issues and Discussion */

2011-12-15T06:33:37Z

‎Issues and Discussion

Kahyun Choi: /* Potential Participants */

2011-12-15T06:32:39Z

‎Potential Participants

Matthiasmauch: /* Issues and Discussion */

2009-09-15T15:36:55Z

‎Issues and Discussion

Paulus: /* Ground Truth */ factual correction

2009-09-15T08:30:19Z

‎Ground Truth: factual correction

Jordan: /* Evaluation Measures */

2009-09-11T16:53:45Z

‎Evaluation Measures

Jordan: /* Ground Truth */

2009-09-11T16:53:21Z

‎Ground Truth

Paulus: /* Evaluation Measures */

2009-09-09T08:11:13Z

‎Evaluation Measures

Paulus: /* Evaluation Measures */ normalised conditional entropy added

2009-09-09T07:53:54Z

‎Evaluation Measures: normalised conditional entropy added

Paulus: added section on evaluation measures

2009-09-09T07:32:17Z

added section on evaluation measures

@@ Line 93: / Line 93: @@
 Defining the segment: In my opinion a segment would be a state with similar acoustical content (like in Lukashevich). I just want to make clear what the algo should do. --[[User:Timezone|Stephan]] 10:04, 10 August 2009 (UTC)
-Some notes: The proposed output with Wavesurfer -like format is probably the best at this first go at the task. For the evaluation metric: I'd propose using both the F-measure for frame pairs (as per [http://dx.doi.org/10.1109/TASL.2007.910781 Levy&Sandler]) and the over/under segmentation measure by Lukashevich because they provide slightly different information. Both of these assume a "state" based description of the structure, so the hierarchical differences will not be handled very gracefully (hierarchical differences do exist if different persons annotate the same piece and a better metric should perhaps be developed at some point). Still, for the sake of simplicity the would be adequate for the task. The question of the data is bit more interesting. We used three different data sets in a recent [http://dx.doi.org/10.1109/TASL.2009.2020533 publication]: a large in-house set that can't be distributed even for MIREX, 174 songs by The Beatles [http://www.iua.upf.edu/~perfe/annotations/sections/license.html from UPF] and [http://www.cs.tut.fi/sgn/arg/paulus/structure.html#beatles_data from TUT], and [http://staff.aist.go.jp/m.goto/RWC-MDB/AIST-Annotation/ RWC Pop]. The last two of these are publicly available, so basically anybody could train the system with them (if there is something to train). --[[User:Paulus|Paulus]] 13:21, 10 August 2009 (UTC)
+Some notes: The proposed output with Wavesurfer -like format is probably the best at this first go at the task. For the evaluation metric: I'd propose using both the F-measure for frame pairs (as per [http://dx.doi.org/10.1109/TASL.2007.910781 Levy&Sandler]) and the over/under segmentation measure by Lukashevich because they provide slightly different information. Both of these assume a "state" based description of the structure, so the hierarchical differences will not be handled very gracefully (hierarchical differences do exist if different persons annotate the same piece and a better metric should perhaps be developed at some point). Still, for the sake of simplicity the would be adequate for the task. The question of the data is bit more interesting. We used three different data sets in a recent [http://dx.doi.org/10.1109/TASL.2009.2020533 publication]: a large in-house set that can't be distributed even for MIREX, 174 songs by The Beatles [http://www.iua.upf.edu/~perfe/annotations/sections/license.html from UPF] and [http://www.cs.tut.fi/sgn/arg/paulus/structure.html#beatles_data from TUT], and [http://staff.aist.go.jp/m.goto/RWC-MDB/AIST-Annotation/ RWC Pop]. The last two of these are publicly available, so basically anybody could train the system with them (if there is something to train). Paulus 13:21, 10 August 2009 (UTC)
 Some comments:
@@ Line 105: / Line 105: @@
 ) Defining the number of labels would help a lot (some test-set used a very restricted vocabulary, some others a very large one); or giving the possibility to output an estimated  hierarchical structure
---[[User:GPeeters|Peeters]] 17:28, 17 August 2009 (UTC)
+-- Peeters 17:28, 17 August 2009 (UTC)
@@ Line 116: / Line 116: @@
 One obvious drawback to this: this kind of output is dramatically different (I think) from most people's current algorithms, and to redesign these in less than a month is perhaps unfeasible. The alternative is to get multi-scale annotations, but I don't think we have that either. Which leaves the 3rd alternative: forgetting hierarchical concerns altogether and just getting something that works well enough.
-It seems like this is a very deep and far-reaching issue, one that probably lies at the heart of the structural segmentation task. Maybe we need a few more follow-ups to Lukashevich before this gets ironed out at MIREX! --[[User:jordan|Jordan]] 23:12, 17 August 2009 (UTC)
+It seems like this is a very deep and far-reaching issue, one that probably lies at the heart of the structural segmentation task. Maybe we need a few more follow-ups to Lukashevich before this gets ironed out at MIREX! -- Jordan 23:12, 17 August 2009 (UTC)
@@ Line 125: / Line 125: @@
 * The label assigned to a segment serves only to indicate the grouping, and the evaluation should handle this.
 * For each song, only the acoustic data is provided, no any additional information. It is the job of the algorithm to do the analysis.
---[[User:Paulus|Paulus]] 14:53, 2 September 2009 (UTC)
+-- Paulus 14:53, 2 September 2009 (UTC)
 I agree, sounds much more practical. Thanks Paulus. --[[User:Matthiasmauch|Matthias]] 15:36, 15 September 2009 (UTC)

@@ Line 87: / Line 87: @@
 Yes, sorry, forgot to specify that. I'm mainly interested in audio, so I changed that above. --[[User:Matthiasmauch|Matthias]] 11:04, 30 June 2009 (UTC)
-The more the merrier: I could as well throw in the algo I implemented 2 years ago for my thesis [http://www.ifs.tuwien.ac.at/mir/audiosegmentation.html]. I'm also curious about the annotated data mentioned. Thanks for your effort! --[[User:Ewald|Ewald]] 17:33, 1 July 2009 (UTC)
+The more the merrier: I could as well throw in the algo I implemented 2 years ago for my thesis [http://www.ifs.tuwien.ac.at/mir/audiosegmentation.html]. I'm also curious about the annotated data mentioned. Thanks for your effort! --Ewald 17:33, 1 July 2009 (UTC)
 Regarding ground truth: at Queen Mary we have the complete Beatles segmentations (with starts at bar beginnings), plus tens of other songs by Carole King, Queen, and Zweieck. We could leave the latter three untouched (i.e. I would not train my own algorithm on them), or publish them soon, so everyone can train their method on them. --[[User:Matthiasmauch|Matthias]] 16:07, 7 August 2009 (UTC)

@@ Line 73: / Line 73: @@
 Matthias Mauch, Queen Mary, University of London --[[User:Matthiasmauch|Matthias]] 08:49, 30 June 2009 (UTC)
-Maarten Grachten, Johannes Kepler University, Linz, Austria -- [[User:MaartenGrachten|Maarten]]
+Maarten Grachten, Johannes Kepler University, Linz, Austria
 Geoffroy Peeters, IRCAM, Paris, France (depending on the kind of annotations)

← Older revision		Revision as of 15:36, 15 September 2009
Line 126:		Line 126:
	* For each song, only the acoustic data is provided, no any additional information. It is the job of the algorithm to do the analysis.		* For each song, only the acoustic data is provided, no any additional information. It is the job of the algorithm to do the analysis.
	--[[User:Paulus\|Paulus]] 14:53, 2 September 2009 (UTC)		--[[User:Paulus\|Paulus]] 14:53, 2 September 2009 (UTC)
		+
		+	I agree, sounds much more practical. Thanks Paulus. --[[User:Matthiasmauch\|Matthias]] 15:36, 15 September 2009 (UTC)

@@ Line 22: / Line 22: @@
 Ground truth data on audio is available for more than 200 songs, so given a quality measure everyone agrees on, evaluation wouldn't be harder than on other MIREX tasks.
-Jouni Paulus's [http://www.cs.tut.fi/sgn/arg/paulus/structure.html structure analysis page] links to a corpus of  177 Beatles songs ([http://www.cs.tut.fi/sgn/arg/paulus/beatles_sections_TUT.zip zip file])ΓÇöthe only publicly available portion of the TUTstructure07 dataset. That dataset contains 557 songs, a list of which is available [http://www.cs.tut.fi/sgn/arg/paulus/TUTstructure07_files.html here].
+Jouni Paulus's [http://www.cs.tut.fi/sgn/arg/paulus/structure.html structure analysis page] links to a corpus of 177 Beatles songs ([http://www.cs.tut.fi/sgn/arg/paulus/beatles_sections_TUT.zip zip file]). The Beatles annotations are not a part of the TUTstructure07 dataset. That dataset contains 557 songs, a list of which is available [http://www.cs.tut.fi/sgn/arg/paulus/TUTstructure07_files.html here].
 Ewald Peiszer's [http://www.ifs.tuwien.ac.at/mir/audiosegmentation.html thesis page] links to a portion of the corpus he used: 43 non-Beatles pop songs (including 10 J-pop songs) ([http://www.ifs.tuwien.ac.at/mir/audiosegmentation/dl/ep_groundtruth_excl_Paulus.zip zip file]).

@@ Line 29: / Line 29: @@
 == Evaluation Measures ==
-Because of the complexity of the structural segmentation task definition, several different evaluation measures will be employed to address different aspects. It should be noted that none of the evaluation measures cares about the true labels of the sections: they only denote the clustering. This means that it does not matter if the systems produce true labels such as "chorus" and "verse", or arbitrary labels such as "A" and "B".
+At the last ISMIR conference [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich] proposed a measure for segmentation evaluation. Because of the complexity of the structural segmentation task definition, several different evaluation measures will be employed to address different aspects. It should be noted that none of the evaluation measures cares about the true labels of the sections: they only denote the clustering. This means that it does not matter if the systems produce true labels such as "chorus" and "verse", or arbitrary labels such as "A" and "B".
 === Boundary retrieval ===

@@ Line 20: / Line 20: @@
 == Ground Truth ==
-Ground truth data on audio is available for more than 200 songs, so given a quality measure everyone agrees on, evaluation wouldn't be harder than on other MIREX tasks. At the last ISMIR conference [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich] proposed a measure for segmentation evaluation.
+Ground truth data on audio is available for more than 200 songs, so given a quality measure everyone agrees on, evaluation wouldn't be harder than on other MIREX tasks.
 == Evaluation Measures ==

@@ Line 31: / Line 31: @@
 === Frame clustering ===
-Both the result and the ground truth are handled in short frames (e.g., beat or fixed 100ms). All frame pairs in a structure description are handled. The pairs in which both frames are assigned to the same cluster (i.e., have the same label) form the sets P_E (for the system result) and P_A (for the ground truth). The ''pairwise precision rate'' can be calculated by P = \frac{|P_E \cap P_A|}{|P_E|}, ''pairwise recall rate'' by R = \frac{|P_E \cap P_A|}{|P_A|}, and ''pairwise F-measure'' by F=\frac{2 P R}{P + R}. [http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008]
+Both the result and the ground truth are handled in short frames (e.g., beat or fixed 100ms). All frame pairs in a structure description are handled. The pairs in which both frames are assigned to the same cluster (i.e., have the same label) form the sets P_E (for the system result) and P_A (for the ground truth). The ''pairwise precision rate'' can be calculated by P = \frac{|P_E \cap P_A|}{|P_E|}, ''pairwise recall rate'' by R = \frac{|P_E \cap P_A|}{|P_A|}, and ''pairwise F-measure'' by F=\frac{2 P R}{P + R}. ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008])
 == Potential Participants ==

← Older revision		Revision as of 07:32, 9 September 2009
Line 22:		Line 22:
	Ground truth data on audio is available for more than 200 songs, so given a quality measure everyone agrees on, evaluation wouldn't be harder than on other MIREX tasks. At the last ISMIR conference [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich] proposed a measure for segmentation evaluation.		Ground truth data on audio is available for more than 200 songs, so given a quality measure everyone agrees on, evaluation wouldn't be harder than on other MIREX tasks. At the last ISMIR conference [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich] proposed a measure for segmentation evaluation.

		+	== Evaluation Measures ==
		+	Because of the complexity of the structural segmentation task definition, several different evaluation measures will be employed to address different aspects.
		+
		+	=== Boundary retrieval ===
		+	'''Hit rate''' Found segment boundaries are accepted to be correct if they are within 0.5s ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007]) or 3s ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008]) from a border in the ground truth. Based on the matched hits, ''boundary retrieval recall rate'', ''boundary retrieval precision rate'', and ''boundary retrieval F-measure'' are be calculated.
		+
		+	'''Median deviation''' Two median deviation measure between boundaries in the result and ground truth are calculated: ''median true-to-guess'' is the median time from boundaries in ground truth to the closest boundaries in the result, and ''median guess-to-true'' is similarly the median time from boundaries in the result to boundaries in ground truth. ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007])
		+
		+	=== Frame clustering ===
		+	Both the result and the ground truth are handled in short frames (e.g., beat or fixed 100ms). All frame pairs in a structure description are handled. The pairs in which both frames are assigned to the same cluster (i.e., have the same label) form the sets P_E (for the system result) and P_A (for the ground truth). The ''pairwise precision rate'' can be calculated by P = \frac{\|P_E \cap P_A\|}{\|P_E\|}, ''pairwise recall rate'' by R = \frac{\|P_E \cap P_A\|}{\|P_A\|}, and ''pairwise F-measure'' by F=\frac{2 P R}{P + R}. [http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008]

	== Potential Participants ==		== Potential Participants ==