2009:SpecialTagatuneEvaluation

What is Tagatune?

Tagatune is a two-player game designed to extract information about music. In each round of the game, two players are is shown either same songs or different songs. Each player describes his given song by typing in any number of tags, which are revealed to the partner. After reviewing each otherΓÇÖs tags, the players must each decide whether he is given the same piece of music as his partner by selecting the same or different button. After both players have voted, the game reveals the true answer (whether the songs given to the pair of players are the same or different) and prepares the next round. Tagatune is live at www.gwap.com

Since Tagatune is a two-player game, when no partner is available for a player, a bot (a computer program or algorithm) is instituted to play against that player. In each round of the game, the bot generates a set of appropriate tags for a song and reveals these tags to the player. The player then decides his votes for same or different by comparing what he is listening to and the tags revealed by his bot partner. If the songs given to the bot and the player are identical, and the tags generated by the bot are accurate for the song, then the player will have a high probability of guessing correctly that the songs are the same. Otherwise, we would expect the player to make more mistakes in making this judgment. In short, the hypothesis is that better algorithms generate tags that are more fitting descriptions of songs, which in turn, allows players to have a higher chance of guessing correctly.

What is the goal of the MIREX Special Tagatune Evaluation?

The goal of the MIREX Special Tagatune Evaluation competition is to investigate a new method of evaluating music tagging algorithms, by using them as bots in Tagatune, and measuring the number of mistakes players make in guessing whether they are listening to the same or different songs (we will call this the Tagatune metric) when paired against different algorithm bots. We are particularly interested in whether there is a statistical correlation between the ranking of the algorithms induced by the Tagatune metric versus the classical metrics used in MIREX.

There are three main steps to this evaluation.

Step 1: Algorithm to Tags

All submitted algorithms will be

(a) trained using the Tagatune training set and tested on the Tagatune test set,

(b) trained using the MIREX 2008 training set (MajorMiner data) and tested on the Tagatune test set.

The trained algorithm must generate a set of tags for each of the songs in the test set, and rank the tags in a particular order (e.g. by confidence, saliency, relevance etc).

Step 2: Algorithm to Tags

These tags will subsequently be displayed to players of Tagatune in a controlled experiment as well as an internet-wide experiment. The number of mistakes players make in guessing whether the songs are same or different is recorded for each algorithm.

Step 3: Ranking

All submitted algorithms will receive two ranking:

(1) ranking using the MIREX metrics

(2) ranking using the Tagatune metric

The Tagatune Dataset

The Tagatune training and test set consists music clip that are 29 seconds long, and are associated with 6622 tracks, 517 albums and 270 artists. The genres include classical, new age, electronica, rock, pop, world, jazz, blues, metal, punk etc. The tags used in the experiments are each associated with more than fifty songs, where each song is associated with a tag by more than two players independently. The following table shows the minimum, maximum and average number of songs associated with any tags in the training set, test set and the complete set used in this evaluation.

	Training Set	Test Set	Complete Set
MIN	18	15	50
MAX	2103	3767	5870
AVG	212	286	499

The following is a list of tags found in the Tagatune training and test set.

no voice singer duet hard rock world harpsichord sitar chorus female opera male vocal vocals clarinet heavy silence beats funky no strings chimes foreign no piano horns classical female spacey jazz guitar quiet no beat banjo electric solo violins folk female voice wind ambient new age synth funk no singing middle eastern trumpet percussion drum airy voice repetitive birds strings bass harpsicord medieval male voice girl acoustic loud classic string drums electronic not classical chanting no violin not rock no guitar organ no vocal talking choral weird opera fast electric guitar male singer man singing classical guitar country violin electro tribal dark male opera no vocals irish electronica horn operatic arabic low instrumental trance chant strange heavy metal modern bells man deep fast beat hard harp no flute pop lute female vocal oboe mellow orchestral light piano celtic male vocals orchestra eastern old flutes punk spanish sad sax slow male blues vocal indian india woman woman singing rock dance piano solo guitars no drums jazzy singing cello calm female vocals voices different techno clapping house flute not opera not english oriental beat upbeat soft noise choir female singer rap metal hip hop water baroque women fiddle english

An interesting effect of Tagatune is that we have collected many negative tags, which indicates the absence of an instrument (e.g. no piano, no guitar) or the genre that the song does not belong to (e.g. not classical, not rock). Participants of this evaluation might want to tailor their algorithms to take advantage of these negative tags that are not available on the MIREX 2008 dataset.

Submission Format

The submission format is identical to the one for Audio Tag Classification task in MIREX 2008 except for the audio formats, detailed descriptions to be found here: https://www.music-ir.org/mirex/2008/index.php/Audio_Tag_Classification.

Audio Formats

Participating algorithms will have to read audio in the following format:

Γû¬ Sample rate: 44 KHz

Γû¬ Sample size: 16 bit

Γû¬ Number of channels: 2 (stereo)

Γû¬ Encoding: WAV (decoded from MP3 files by IMIRSEL)

Γû¬ Duration: 10 or 29 second clips

Deadlines and Timeline

Submission opening date: Dec 15, 2008

Submission closing date: Jan 30, 2009