Difference between revisions of "2007:Previous Discussion on Ground Truth"
(New page: == Ground Truth == Corresponding to how the mood taxonomy is going to be set up, there are two ways to obtain ground truth for evaluation purpose. 1. human judgment: we can elicit subject...) |
IMIRSELBot (talk | contribs) m (Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2007:\1)) |
||
| (One intermediate revision by one other user not shown) | |||
| Line 8: | Line 8: | ||
Yet another (good) way of obtaining ground truth: | Yet another (good) way of obtaining ground truth: | ||
3. obtain datasets used in existing research. Those datasets have been labeled by individual reseachers. | 3. obtain datasets used in existing research. Those datasets have been labeled by individual reseachers. | ||
| + | |||
| + | |||
| + | Link to task proposal: [[2007:Audio_Music_Mood_Classification]] | ||
Latest revision as of 12:20, 13 May 2010
Ground Truth
Corresponding to how the mood taxonomy is going to be set up, there are two ways to obtain ground truth for evaluation purpose.
1. human judgment: we can elicit subjective judgments by human evaluators by using an online application comparable to IMIRSEL's Evalutron 6000. Details need to be further discussed. To start, we propose human evaluators to choose one mood label from a set for each music piece. Each piece may get at least 3 eyeballs and a label with at least 2 votes will be assigned to this piece as ground truth. Of cause there will be disagreement and depending on the number of available categories, votes to some pieces may be too scattered and thus invalidate judgments on those pieces.
2. collect labels from popular music websites. A problem is AMG only provide labels for albums. And even if labels for tracks are available, they might not be available for the pieces in our contest dataset.
Yet another (good) way of obtaining ground truth: 3. obtain datasets used in existing research. Those datasets have been labeled by individual reseachers.
Link to task proposal: 2007:Audio_Music_Mood_Classification
