2009:Audio Tag Classification Tagatune Results

Introduction

This task compares various algorithms' abilities to associate tags with 29-second audio clips of songs. The tags used were collected by the Tagatune game and algorithms were evaluated using the previously collected tags (using the same statistical procedures as the the other MIREX 2009 tag classification tasks) and in the Tagatune game itself (the Tagatune metric).

What is Tagatune?

Tagatune is a two-player game designed to extract information about music. In each round of the game, two players are each shown a song, either they are shown the same song or two different songs. Each player describes his given song by typing in any number of tags, which are immediately revealed to the partner. After reviewing each other's tags, the players must each decide whether they have been given the same piece of music as their partner. After both players have voted, the game reveals the true answer (whether the songs given to the pair of players are the same or different) and prepares the next round. Tagatune is live at www.gwap.com

http://www.cs.cmu.edu/~elaw/tagatune.jpg

Since Tagatune is a two-player game, when no partner is available for a player, a bot (a computer program or algorithm) is instituted to play against that player. In each round of the game, the bot generates a set of appropriate tags for a song and reveals these tags to the player. The player then decides his votes for same or different by comparing what he is listening to and the tags revealed by his bot partner. If the songs given to the bot and the player are identical, and the tags generated by the bot are accurate for the song, then the player will have a high probability of guessing correctly that the songs are the same. Otherwise, we would expect the player to make more mistakes in making this judgment. In short, the hypothesis is that better algorithms generate tags that are more fitting descriptions of songs, which in turn, allows players to have a higher chance of guessing correctly.

What is the goal of the MIREX Special Tagatune Evaluation?

The goal of the MIREX Special Tagatune Evaluation competition is to investigate a new method of evaluating music tagging algorithms, by using them as bots in Tagatune, and measuring the number of mistakes players make in guessing whether they are listening to the same or different songs (we will call this the Tagatune metric) when paired against different algorithm bots. We are particularly interested in whether there is a statistical correlation between the ranking of the algorithms induced by the Tagatune metric versus the classical metrics used in MIREX. For the motivation behind this evaluation, see this paper.

There are three main steps to this evaluation.

Step 1: Algorithm to Tags

All submitted algorithms are trained using the Tagatune training set and tested on the Tagatune test set. Artist filtering was used in the production of the test and training split, I.e. the training and test sets contained different artists. The trained algorithm must generate a set of tags for each of the songs in the test set, and rank the tags in a particular order (e.g. by confidence, saliency, relevance etc). This part of the evaluation is very similar, if not identical, to the MIREX 2009 Audio Tag Classification tasks where two outputs are produced by each algorithm:

• a set of binary classifications indicating which tags are relevant to each example,
• a set of 'affinity' scores which indicate the degree to which each tag applies to each track.

These different outputs allow the algorithms to be evaluated both on tag 'classification' and tag 'ranking' (where the tags may be ranked for each track and tracks ranked for each tag).

Step 2: Tagatune Experiments

The tags returned as 'relevant' by each algorithm were subsequently displayed to players of Tagatune in an internet-wide experiment. The number of mistakes players make in guessing whether the songs were the same or different was recorded for each algorithm.

Step 3: Ranking

The submitted algorithm were then evaluated by two methods:

(1) ranking using the MIREX metrics

(2) ranking using the Tagatune metric

The Tagatune Dataset

The Tagatune training and test set consist of music clips that are 29 seconds long, and are associated with 6622 tracks, 517 albums and 270 artists. The genres include classical, new age, electronica, rock, pop, world, jazz, blues, metal, punk etc. The tags used in the experiments are each associated with more than fifty songs, where each song is associated with a tag by more than two players independently. The following table shows the minimum, maximum and average number of songs associated with any tags in the training set, test set and the complete set used in this evaluation.

	Training Set	Test Set	Complete Set
MIN	18	15	50
MAX	2103	3767	5870
AVG	212	288	502

Number of samples in training set: 9598

Number of samples in test set: 13194

The following is a list of 160 tags found in the Tagatune dataset.

no voice	singer	duet	hard rock
world	harpsichord	sitar	chorus
female opera	male vocal	vocals	clarinet
heavy	silence	beats	funky
no strings	chimes	foreign	no piano
horns	classical	female	spacey
jazz	guitar	quiet	no beat
banjo	electric	solo	violins
folk	female voice	wind	ambient
new age	synth	funk	no singing
middle eastern	trumpet	percussion	drum
airy	voice	repetitive	birds
strings	bass	harpsicord	medieval
male voice	girl	acoustic	loud
classic	string	drums	electronic
not classical	chanting	no violin	not rock
no guitar	organ	no vocal	talking
choral	weird	opera	fast
electric guitar	male singer	man singing	classical guitar
country	violin	electro	tribal
dark	male opera	no vocals	irish
electronica	horn	operatic	arabic
low	instrumental	trance	chant
strange	heavy metal	modern	bells
man	deep	fast beat	hard
harp	no flute	pop	lute
female vocal	oboe	mellow	orchestral
light	piano	celtic	male vocals
orchestra	eastern	old	flutes
punk	spanish	sad	sax
slow	male	blues	vocal
indian	india	woman	woman singing
rock	dance	piano solo	guitars
no drums	jazzy	singing	cello
calm	female vocals	voices	techno
clapping	house	flute	not opera
not english	oriental	beat	upbeat
soft	noise	choir	female singer
rap	metal	hip hop	water
baroque	women	fiddle	english

NOTE: An interesting effect of Tagatune is that we have collected many negative tags, which indicates the absence of an instrument (e.g. no piano, no guitar) or the genre that the song does not belong to (e.g. not classical, not rock). Participants of this evaluation might want to tailor their algorithms to take advantage of these negative tags that are not available on the MIREX 2008/2009 datasets.

MIREX Statistical Evaluation

Participating algorithms were evaluated over a single artist-filtered test/train split using both the full test set and only the 100 query subset used in Tagatune evaluation.

Binary (Classification) Evaluation

Algorithms are evaluated on their performance at tag classification using F-measure. Results are also reported for simple accuracy, however, as this statistic is dominated by the negative example accuracy it is not a reliable indicator of performance (as a system that returns no tags for any example will achieve a high score on this statistic). However, the accuracies are also reported for positive and negative examples separately as these can help elucidate the behaviour of an algorithm (for example demonstrating if the system is under of over predicting).

Affinity (Ranking) Evaluation

Algorithms are evaluated on their performance at tag ranking using the Area Under the Receiver Operating Characteristic Curve (AUC-ROC). The affinity scores for each tag to be applied to a track are sorted prior to the computation of the AUC-ROC statistic, which gives higher scores to ranked tag sets where the correct tags appear towards the top of the set.

General Legend

Team ID

Mandel = Michael Mandel
Manzagol = Pierre-Antoine Manzagol
Marsyas = George Tzanetakis
Zhi = Zhi-Sheng Chen
LabX = Anonymous

Results

The following sections provide detail the evaluation statistics computed. The results of the task are also detailed in the paper Evaluation of Algorithms Using Games: The Case of Music Tagging.

Overall Summary Results (Tagatune)

Measure	Human	LabX	Mandel	Manzagol	Marsyas	Zhi
Tagatune Metric	93.00%	26.80%	70.10%	67.50%	68.60%	60.90%

download these results as csv

Friedman's Test Results

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the Tagatune metric for each track in the test. The tags generated by the algorithms are pre-processed to remove redundant or contradictory tags, which is important to maintain a minimum quality for the algorithm bots. This pre-processing is not done on the data for which other metrics are computed.

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
"Human"	"Mandel"	0.563	1.265	1.967	TRUE
"Human"	"Manzagol"	0.513	1.215	1.917	TRUE
"Human"	"Marsyas"	0.773	1.475	2.177	TRUE
"Human"	"Zhi"	1.258	1.960	2.662	TRUE
"Human"	"LabX"	2.233	2.935	3.637	TRUE
"Mandel"	"Manzagol"	-0.752	-0.050	0.652	FALSE
"Mandel"	"Marsyas"	-0.492	0.210	0.912	FALSE
"Mandel"	"Zhi"	-0.007	0.695	1.397	FALSE
"Mandel"	"LabX"	0.968	1.670	2.372	TRUE
"Manzagol"	"Marsyas"	-0.442	0.260	0.962	FALSE
"Manzagol"	"Zhi"	0.043	0.745	1.447	TRUE
"Manzagol"	"LabX"	1.018	1.720	2.422	TRUE
"Marsyas"	"Zhi"	-0.217	0.485	1.187	FALSE
"Marsyas"	"LabX"	0.758	1.460	2.162	TRUE
"Zhi"	"LabX"	0.273	0.975	1.677	TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/tagatune_correctness.friedman.tukeyKramerHSD.png

Tagatune Correctness

Track	Human	LabX	Mandel	Manzagol	Marsyas	Zhi
4303	1	0	0.800	1	1	0
33934	1	0.200	1	0.800	0.800	1
48608	0.800	0.400	0.800	0.800	0.800	1
41810	1	0	0.600	0.200	1	1
42165	1	0.600	0.600	1	1	0.800
36668	0.800	0.200	0.800	0.800	0.600	0.400
54698	1	0.400	0.800	0.600	0.400	0.600
23910	1	0.600	0.400	0.600	0.600	0
55057	1	0.200	1	1	1	0.400
10943	1	0.800	1	1	0	0.800
32635	1	0.200	0.400	1	0.600	0.600
8802	1	0	0.800	1	0.600	1
48455	0.800	0.200	1	0.600	0.600	0.600
31267	0.600	0.600	1	0.800	0.600	0.400
25699	1	0.400	1	1	1	0
42361	1	0	1	0.800	0.400	0
21267	1	1	0.600	0	0	0.400
9956	1	0.200	0.400	0	0.400	0.400
44920	1	0	0.600	0.800	0.800	1
7313	1	0	0.400	0.400	0.400	0.400
28222	0.800	0	0.600	1	0.800	0.800
28224	1	0.200	0.600	0.800	0.600	0.800
19209	0.800	0.400	0.600	1	1	0.400
23773	1	0	0.600	1	1	1
43598	0.800	0	0.400	0.400	0	0.800
88	1	0	1	1	0.600	0.800
9494	1	0.200	0.200	0.400	0	0.600
16864	0.800	0.800	0.800	0.800	1	0.600
31905	1	0.200	0.800	1	0	0.200
15023	1	0	1	1	0.800	0.600
27304	1	0.200	0.400	0	0.800	0.600
16385	0.800	0.200	0.600	0.600	0.800	0.200
40029	0.800	1	1	1	0.800	0.200
43295	0.800	0	1	0	1	0.800
12795	0.600	1	0.200	0.200	1	0
44560	0.800	0	0.200	1	0	0.400
15325	1	0.200	1	1	0.600	0.400
33941	1	0.200	0.600	1	1	0.200
15134	0.800	0.400	0.600	0.600	1	0.400
4815	0.800	0.600	0.800	0.200	0.200	0.400
20022	1	0.400	0.400	1	1	0.600
26382	1	0.200	0.800	0.200	0.200	0.600
35687	1	0.200	1	0	0	0.200
45842	1	0	1	0.800	1	0.600
2456	0.800	0.800	0.600	0	0.200	0.600
15128	1	0	0.800	0.200	1	0.400
25228	0.800	0	0.600	0.800	0.600	0
46943	1	0	1	0.800	0	0.200
24215	0.800	0.400	0.400	0	0.200	0
20132	0.600	0.600	0.400	0.400	0.800	0.200
19370	1	0	0.800	0.800	0.800	0.200
2053	1	1	1	1	1	1
3217	1	0	0.800	0.400	0.200	1
49877	1	0	0.200	0.800	0.200	1
20030	1	0	0.600	1	1	0.600
55361	0.800	0.600	0.600	1	0.600	0.800
24920	1	0	0.800	1	0.800	0.600
25635	0.600	0	0.800	1	1	0.800
43638	0.800	0.800	0.800	0.600	0	0.200
13047	1	0.600	1	0.600	1	0.600
46941	0.600	0	0.400	0.200	0.400	0.400
34281	1	0	1	1	1	0.600
15093	0.600	0.400	0.800	0.400	0.200	0.400
36940	1	0	0.800	0.400	0.200	1
18122	0.600	0.800	0.200	0.200	0.400	0.200
3074	1	0	0.800	1	1	1
16429	1	0	0.600	0	1	1
44091	1	0.200	0.600	1	1	0.200
23230	1	0.200	0	0	0.400	0
29086	1	0.600	1	1	1	0.800
7561	1	0.800	1	0	0.600	1
42554	1	0.400	0.800	0.800	1	0.200
40638	0.800	0	0.400	1	0.600	0.200
19220	1	0.600	0.400	1	0.800	0.800
3227	1	0	0.800	1	0.600	0.600
14874	1	0.200	1	0.800	1	1
31684	1	0.800	1	0.400	0	0.800
48236	1	0.600	0.200	0.600	0.800	0.200
14241	1	0.200	0.200	1	1	0.800
48424	0.800	0	0.400	0.800	0.800	0.200
37579	0.800	0.200	1	1	1	1
27476	0.800	0.200	0.400	1	0.200	0.600
31220	1	0	0.400	0.800	0	0.800
36649	1	0.200	0.800	1	0	0
36168	1	0	0.800	0.600	1	1
36646	1	0.200	0.200	0.400	0.400	0
21872	1	0	1	0	0.600	0.400
13232	1	0.200	1	1	0.800	0.800
30803	0.800	0	1	0.400	0.600	1
45897	1	0.200	1	0.200	1	1
18287	0.800	1	0.600	0.800	0.800	0.800
26461	1	0.400	1	1	1	1
10949	1	0	0.800	1	0	0.400
51093	1	0	1	1	0.800	1
32514	1	0.200	0.800	0.600	0.800	0.600
53102	0.600	0.800	1	1	0.800	0
46769	1	0	1	1	1	0.200
4853	1	0.800	0.200	0.600	1	0.400
10944	1	1	0.800	1	0.600	0.800
36811	0.800	0.200	0.600	0.200	0.200	0.200

download these results as csv

Overall Summary Results (MIREX Statistical evaluation - Binary)

Full dataset

Measure	LabX	Mandel	Manzagol	Marsyas	Zhi
Average Tag F-measure	0.001	0.132	0.098	0.125	0.138
Average Tag Accuracy	0.972	0.789	0.967	0.948	0.914
Average Positive Tag Accuracy	0.004	0.698	0.120	0.223	0.413
Average Negative Tag Accuracy	0.994	0.790	0.983	0.954	0.922

download these results as csv

100 query subset used in Tagatune evaluation

Measure	LabX	Mandel	Manzagol	Marsyas	Zhi
Average Tag F-measure	0.002	0.304	0.110	0.212	0.250
Average Tag Accuracy	0.921	0.799	0.925	0.924	0.886
Average Positive Tag Accuracy	0.005	0.688	0.095	0.224	0.361
Average Negative Tag Accuracy	0.993	0.806	0.987	0.956	0.923

download these results as csv

Binary Relevance F-Measure

Full dataset

Tag	Positive Examples	Negative Examples	LabX	Mandel	Manzagol	Marsyas	Zhi
nostrings	13.000	6486.000	0.000	0.005	0.000	0.000	0.000
chimes	22.000	6477.000	0.022	0.016	0.000	0.035	0.046
sad	18.000	6481.000	0.006	0.014	0.062	0.000	0.026
nodrums	48.000	6451.000	0.000	0.019	0.000	0.000	0.017
femalevoice	105.000	6394.000	0.000	0.142	0.081	0.147	0.177
horn	7	6492.000	0.000	0.003	0.000	0.000	0.014
pop	196.000	6303.000	0.000	0.166	0.184	0.254	0.158
rock	601.000	5898.000	0.000	0.562	0.502	0.551	0.523
house	22.000	6477.000	0.000	0.025	0.028	0.000	0.029
birds	7	6492.000	0.000	0.012	0.000	0.034	0.020
harpsicord	59.000	6440.000	0.000	0.165	0.127	0.209	0.094
strange	22.000	6477.000	0.000	0.015	0.000	0.043	0.022
noflute	35.000	6464.000	0.000	0.008	0.000	0.000	0.015
novocal	263.000	6236.000	0.000	0.099	0.006	0.104	0.082
solo	217.000	6282.000	0.000	0.202	0.085	0.183	0.188
notenglish	11.000	6488.000	0.000	0.019	0.000	0.031	0.052
novoice	146.000	6353.000	0.000	0.058	0.011	0.058	0.048
newage	157.000	6342.000	0.000	0.139	0.000	0.174	0.116
synth	294.000	6205.000	0.000	0.190	0.083	0.233	0.192
upbeat	52.000	6447.000	0.000	0.040	0.029	0.070	0.057
slow	1043.000	5456.000	0.000	0.437	0.256	0.452	0.392
deep	12.000	6487.000	0.000	0.013	0.065	0.000	0.019
fiddle	14.000	6485.000	0.000	0.018	0.000	0.027	0.018
orchestral	12.000	6487.000	0.000	0.008	0.000	0.000	0.029
notclassical	14.000	6485.000	0.000	0.006	0.000	0.000	0.017
mansinging	46.000	6453.000	0.000	0.042	0.011	0.063	0.070
wind	22.000	6477.000	0.048	0.025	0.022	0.000	0.032
piano	630.000	5869.000	0.000	0.550	0.528	0.392	0.534
spanish	65.000	6434.000	0.000	0.050	0.011	0.060	0.068
femalesinger	30.000	6469.000	0.000	0.047	0.075	0.120	0.085
singing	242.000	6257.000	0.000	0.226	0.116	0.262	0.221
quiet	263.000	6236.000	0.000	0.219	0.054	0.342	0.212
oboe	12.000	6487.000	0.000	0.009	0.026	0.000	0.004
tribal	40.000	6459.000	0.000	0.022	0.036	0.090	0.082
noguitar	46.000	6453.000	0.000	0.018	0.000	0.011	0.049
femalevocal	126.000	6373.000	0.000	0.189	0.076	0.208	0.202
fastbeat	33.000	6466.000	0.000	0.029	0.000	0.000	0.057
hiphop	32.000	6467.000	0.000	0.058	0.222	0.000	0.121
instrumental	102.000	6397.000	0.000	0.045	0.026	0.053	0.048
chorus	50.000	6449.000	0.000	0.161	0.255	0.000	0.234
silence	12.000	6487.000	0.000	0.030	0.075	0.000	0.029
duet	18.000	6481.000	0.000	0.014	0.000	0.000	0.015
sax	20.000	6479.000	0.000	0.012	0.000	0.026	0.000
nobeat	14.000	6485.000	0.000	0.008	0.000	0.000	0.031
nopiano	90.000	6409.000	0.017	0.033	0.000	0.007	0.023
novocals	326.000	6173.000	0.000	0.119	0.006	0.117	0.100
pianosolo	13.000	6486.000	0.000	0.019	0.117	0.000	0.052
low	35.000	6464.000	0.000	0.039	0.097	0.031	0.030
weird	120.000	6379.000	0.000	0.075	0.036	0.145	0.103
dance	184.000	6315.000	0.000	0.216	0.214	0.179	0.265
harp	137.000	6362.000	0.000	0.137	0.085	0.148	0.128
horns	12.000	6487.000	0.000	0.009	0.035	0.000	0.012
funky	66.000	6433.000	0.000	0.073	0.082	0.000	0.105
hardrock	80.000	6419.000	0.000	0.182	0.115	0.000	0.171
bells	36.000	6463.000	0.000	0.021	0.028	0.046	0.042
punk	42.000	6457.000	0.000	0.122	0.159	0.000	0.120
electricguitar	51.000	6448.000	0.000	0.049	0.071	0.113	0.065
techno	827.000	5672.000	0.000	0.584	0.441	0.609	0.621
modern	73.000	6426.000	0.000	0.037	0.046	0.051	0.053
violins	258.000	6241.000	0.000	0.269	0.155	0.220	0.251
noviolin	18.000	6481.000	0.000	0.007	0.077	0.028	0.009
opera	325.000	6174.000	0.000	0.667	0.592	0.372	0.630
india	22.000	6477.000	0.000	0.025	0.074	0.033	0.167
cello	145.000	6354.000	0.000	0.376	0.268	0.208	0.266
sitar	250.000	6249.000	0.000	0.377	0.454	0.400	0.321
hard	25.000	6474.000	0.000	0.060	0.051	0.000	0.063
banjo	15.000	6484.000	0.000	0.015	0.051	0.013	0.026
blues	42.000	6457.000	0.000	0.095	0.103	0.121	0.053
man	128.000	6371.000	0.000	0.132	0.025	0.231	0.194
water	12.000	6487.000	0.000	0.027	0.000	0.000	0.026
femalevocals	90.000	6409.000	0.000	0.128	0.068	0.145	0.140
beat	534.000	5965.000	0.000	0.370	0.269	0.527	0.459
vocal	346.000	6153.000	0.000	0.277	0.102	0.295	0.228
jazz	88.000	6411.000	0.000	0.099	0.076	0.153	0.118
male	316.000	6183.000	0.000	0.310	0.193	0.327	0.293
maleopera	18.000	6481.000	0.000	0.125	0.147	0.000	0.128
drums	663.000	5836.000	0.000	0.374	0.235	0.417	0.367
electronic	578.000	5921.000	0.000	0.364	0.156	0.411	0.383
talking	27.000	6472.000	0.000	0.034	0.061	0.000	0.020
violin	908.000	5591.000	0.000	0.666	0.620	0.560	0.587
bass	73.000	6426.000	0.000	0.053	0.037	0.123	0.091
notrock	19.000	6480.000	0.000	0.004	0.000	0.034	0.035
string	91.000	6408.000	0.000	0.068	0.032	0.076	0.047
womansinging	32.000	6467.000	0.000	0.059	0.031	0.119	0.108
guitar	1166.000	5333.000	0.000	0.584	0.464	0.507	0.544
medieval	39.000	6460.000	0.000	0.044	0.072	0.012	0.038
clarinet	16.000	6483.000	0.000	0.028	0.000	0.000	0.036
world	14.000	6485.000	0.000	0.007	0.080	0.000	0.069
old	14.000	6485.000	0.000	0.012	0.000	0.040	0.012
middleeastern	17.000	6482.000	0.000	0.009	0.027	0.044	0.013
baroque	81.000	6418.000	0.019	0.120	0.015	0.155	0.093
oriental	50.000	6449.000	0.000	0.038	0.055	0.072	0.072
trumpet	17.000	6482.000	0.000	0.016	0.080	0.000	0.000
irish	49.000	6450.000	0.000	0.070	0.018	0.092	0.030
ambient	419.000	6080.000	0.000	0.432	0.028	0.397	0.308
funk	32.000	6467.000	0.000	0.062	0.086	0.000	0.057
metal	159.000	6340.000	0.006	0.333	0.187	0.000	0.295
woman	186.000	6313.000	0.000	0.292	0.106	0.274	0.335
dark	36.000	6463.000	0.000	0.045	0.035	0.000	0.027
acoustic	66.000	6433.000	0.012	0.081	0.097	0.106	0.124
light	16.000	6483.000	0.000	0.009	0.000	0.054	0.007
repetitive	24.000	6475.000	0.000	0.012	0.000	0.000	0.000
trance	51.000	6448.000	0.000	0.049	0.021	0.057	0.063
celtic	27.000	6472.000	0.000	0.024	0.000	0.067	0.017
electric	44.000	6455.000	0.000	0.022	0.000	0.013	0.039
malevocals	123.000	6376.000	0.066	0.122	0.122	0.178	0.101
heavy	59.000	6440.000	0.000	0.110	0.161	0.000	0.120
jazzy	68.000	6431.000	0.000	0.081	0.099	0.112	0.112
country	122.000	6377.000	0.000	0.179	0.141	0.190	0.128
beats	157.000	6342.000	0.009	0.131	0.123	0.240	0.223
loud	313.000	6186.000	0.000	0.307	0.148	0.448	0.318
classical	1544.000	4955.000	0.000	0.727	0.244	0.523	0.618
voices	39.000	6460.000	0.000	0.045	0.000	0.038	0.103
flutes	54.000	6445.000	0.000	0.124	0.298	0.000	0.227
choral	104.000	6395.000	0.000	0.360	0.282	0.256	0.475
harpsichord	263.000	6236.000	0.000	0.398	0.375	0.384	0.290
eastern	80.000	6419.000	0.000	0.068	0.100	0.185	0.110
foreign	51.000	6448.000	0.000	0.061	0.023	0.124	0.142
fast	616.000	5883.000	0.000	0.324	0.154	0.384	0.291
english	11.000	6488.000	0.000	0.005	0.000	0.000	0.014
spacey	27.000	6472.000	0.000	0.047	0.000	0.052	0.033
electro	87.000	6412.000	0.000	0.064	0.015	0.079	0.091
calm	33.000	6466.000	0.000	0.016	0.000	0.038	0.023
lute	15.000	6484.000	0.000	0.049	0.088	0.000	0.051
arabic	10.000	6489.000	0.000	0.003	0.019	0.000	0.000
voice	111.000	6388.000	0.000	0.086	0.057	0.115	0.086
vocals	256.000	6243.000	0.000	0.176	0.084	0.203	0.187
rap	41.000	6458.000	0.000	0.111	0.219	0.000	0.296
singer	25.000	6474.000	0.000	0.023	0.029	0.021	0.000
strings	997.000	5502.000	0.000	0.551	0.141	0.509	0.461
orchestra	98.000	6401.000	0.000	0.117	0.108	0.118	0.110
guitars	25.000	6474.000	0.000	0.011	0.000	0.000	0.036
chant	51.000	6448.000	0.000	0.190	0.182	0.350	0.262
heavymetal	43.000	6456.000	0.000	0.106	0.109	0.000	0.123
girl	10.000	6489.000	0.000	0.011	0.000	0.000	0.041
percussion	26.000	6473.000	0.004	0.020	0.000	0.032	0.059
flute	455.000	6044.000	0.000	0.609	0.589	0.479	0.475
drum	89.000	6410.000	0.000	0.066	0.060	0.110	0.101
classic	235.000	6264.000	0.000	0.210	0.106	0.180	0.185
nosinging	51.000	6448.000	0.000	0.019	0.000	0.038	0.012
chanting	32.000	6467.000	0.000	0.068	0.053	0.086	0.122
folk	48.000	6451.000	0.000	0.036	0.018	0.044	0.063
malesinger	39.000	6460.000	0.000	0.033	0.045	0.084	0.051
mellow	29.000	6470.000	0.000	0.012	0.000	0.000	0.015
indian	313.000	6186.000	0.000	0.284	0.185	0.319	0.269
electronica	39.000	6460.000	0.000	0.028	0.000	0.055	0.045
women	22.000	6477.000	0.000	0.047	0.057	0.111	0.129
notopera	19.000	6480.000	0.000	0.004	0.000	0.000	0.028
noise	16.000	6483.000	0.000	0.016	0.065	0.000	0.021
soft	248.000	6251.000	0.000	0.166	0.075	0.214	0.169
femaleopera	27.000	6472.000	0.000	0.107	0.123	0.000	0.126
malevoice	155.000	6344.000	0.000	0.141	0.062	0.192	0.153
organ	17.000	6482.000	0.000	0.007	0.006	0.011	0.009
female	320.000	6179.000	0.000	0.460	0.244	0.322	0.408
classicalguitar	38.000	6461.000	0.000	0.180	0.104	0.000	0.129
operatic	17.000	6482.000	0.000	0.036	0.000	0.000	0.070
airy	12.000	6487.000	0.026	0.013	0.000	0.038	0.023
malevocal	271.000	6228.000	0.000	0.273	0.200	0.304	0.218
clapping	12.000	6487.000	0.000	0.008	0.000	0.000	0.000
choir	161.000	6338.000	0.000	0.508	0.544	0.429	0.590

download these results as csv

100 query subset used in Tagatune evaluation

Tag	Positive Examples	Negative Examples	LabX	Mandel	Manzagol	Marsyas	Zhi
sad	5	95.000	0.125	0.167	0.333	0.000	0.214
nodrums	4	96.000	0.000	0.102	0.000	0.000	0.148
femalevoice	6	94.000	0.000	0.385	0.000	0.308	0.154
pop	10.000	90.000	0.000	0.462	0.522	0.700	0.500
rock	14.000	86.000	0.000	0.710	0.333	0.718	0.688
birds	1	99.000	0.000	0.200	0.000	0.000	0.000
harpsicord	3	97.000	0.000	0.400	0.000	0.000	0.200
strange	2	98.000	0.000	0.085	0.000	0.000	0.000
novocal	12.000	88.000	0.000	0.328	0.154	0.100	0.083
solo	11.000	89.000	0.000	0.276	0.154	0.235	0.211
notenglish	1	99.000	0.000	0.000	0.000	0.000	0.000
novoice	4	96.000	0.000	0.143	0.000	0.000	0.000
newage	12.000	88.000	0.000	0.421	0.000	0.400	0.333
synth	11.000	89.000	0.000	0.514	0.167	0.667	0.333
upbeat	4	96.000	0.000	0.214	0.000	0.000	0.308
slow	44.000	56.000	0.000	0.769	0.259	0.733	0.485
deep	2	98.000	0.000	0.091	0.000	0.000	0.000
fiddle	1	99.000	0.000	0.069	0.000	0.000	0.000
orchestral	2	98.000	0.000	0.062	0.000	0.000	0.121
mansinging	2	98.000	0.000	0.167	0.000	0.286	0.333
wind	1	99.000	0.000	0.095	0.000	0.000	0.105
piano	9	91.000	0.000	0.286	0.100	0.286	0.267
femalesinger	4	96.000	0.000	0.240	0.000	0.000	0.000
singing	13.000	87.000	0.000	0.667	0.235	0.480	0.500
quiet	16.000	84.000	0.000	0.500	0.118	0.611	0.471
tribal	1	99.000	0.000	0.167	0.000	0.000	1.000
noguitar	2	98.000	0.000	0.073	0.000	0.000	0.000
femalevocal	13.000	87.000	0.000	0.545	0.190	0.609	0.421
fastbeat	1	99.000	0.000	0.000	0.000	0.000	0.000
instrumental	4	96.000	0.000	0.131	0.000	0.000	0.125
chorus	3	97.000	0.000	0.444	0.000	0.000	0.500
silence	1	99.000	0.000	0.200	0.000	0.000	0.333
sax	2	98.000	0.000	0.000	0.000	0.000	0.000
nobeat	1	99.000	0.000	0.044	0.000	0.000	0.000
nopiano	3	97.000	0.000	0.095	0.000	0.333	0.000
novocals	15.000	85.000	0.000	0.353	0.000	0.222	0.087
low	5	95.000	0.000	0.207	0.000	0.000	0.000
weird	3	97.000	0.000	0.121	0.000	0.000	0.000
dance	4	96.000	0.000	0.615	0.250	0.444	0.500
harp	3	97.000	0.000	0.000	0.000	0.000	0.286
horns	3	97.000	0.000	0.000	0.000	0.000	0.000
funky	1	99.000	0.000	0.118	0.000	0.000	0.000
hardrock	4	96.000	0.000	0.533	0.000	0.000	0.421
bells	2	98.000	0.000	0.080	0.667	0.000	0.000
punk	4	96.000	0.000	0.615	0.857	0.000	0.500
techno	12.000	88.000	0.000	0.621	0.500	0.571	0.480
modern	5	95.000	0.000	0.158	0.333	0.250	0.364
violins	25.000	75.000	0.000	0.656	0.258	0.353	0.538
opera	6	94.000	0.000	0.571	0.182	0.462	0.571
cello	21.000	79.000	0.000	0.723	0.345	0.378	0.467
sitar	1	99.000	0.000	0.000	0.000	0.000	0.500
man	4	96.000	0.000	0.167	0.400	0.400	0.000
femalevocals	9	91.000	0.000	0.462	0.000	0.353	0.286
beat	13.000	87.000	0.000	0.457	0.000	0.545	0.500
vocal	22.000	78.000	0.000	0.708	0.000	0.718	0.345
jazz	3	97.000	0.000	0.333	0.000	0.000	0.333
male	10.000	90.000	0.000	0.438	0.133	0.545	0.000
drums	14.000	86.000	0.000	0.564	0.222	0.533	0.480
electronic	16.000	84.000	0.000	0.571	0.316	0.690	0.533
violin	44.000	56.000	0.000	0.886	0.800	0.835	0.685
bass	5	95.000	0.000	0.171	0.000	0.250	0.000
string	12.000	88.000	0.000	0.333	0.000	0.000	0.160
womansinging	3	97.000	0.000	0.240	0.000	0.000	0.000
guitar	15.000	85.000	0.000	0.500	0.500	0.393	0.381
medieval	5	95.000	0.000	0.267	0.000	0.000	0.000
old	1	99.000	0.000	0.050	0.000	0.000	0.061
middleeastern	1	99.000	0.000	0.000	0.000	0.000	0.000
baroque	7	93.000	0.000	0.276	0.000	0.000	0.238
oriental	1	99.000	0.000	0.000	0.000	0.000	0.000
trumpet	2	98.000	0.000	0.000	0.000	0.000	0.000
irish	1	99.000	0.000	0.091	0.000	0.000	0.222
ambient	14.000	86.000	0.000	0.625	0.000	0.667	0.333
funk	2	98.000	0.000	0.444	0.000	0.000	0.500
metal	5	95.000	0.160	0.714	0.333	0.000	0.500
woman	14.000	86.000	0.000	0.710	0.000	0.583	0.381
dark	1	99.000	0.000	0.000	0.000	0.000	0.000
acoustic	3	97.000	0.000	0.375	0.000	0.500	0.500
light	1	99.000	0.000	0.040	0.000	0.000	0.500
trance	4	96.000	0.000	0.316	0.000	0.000	0.462
celtic	1	99.000	0.000	0.057	0.000	0.000	0.000
electric	1	99.000	0.000	0.074	0.000	0.000	0.000
malevocals	6	94.000	0.000	0.154	0.250	0.267	0.300
heavy	4	96.000	0.000	0.250	0.400	0.000	0.353
jazzy	3	97.000	0.000	0.267	0.750	0.000	0.462
country	2	98.000	0.000	0.250	0.000	0.222	0.000
beats	4	96.000	0.000	0.087	0.000	0.000	0.000
loud	11.000	89.000	0.000	0.545	0.154	0.588	0.571
classical	48.000	52.000	0.000	0.839	0.038	0.821	0.771
voices	2	98.000	0.000	0.250	0.000	0.000	0.400
choral	7	93.000	0.000	0.500	0.000	0.000	0.333
harpsichord	12.000	88.000	0.000	0.621	0.435	0.741	0.458
eastern	1	99.000	0.000	0.091	0.000	0.000	0.667
foreign	2	98.000	0.000	0.111	0.000	0.000	0.000
fast	17.000	83.000	0.000	0.389	0.000	0.545	0.357
english	1	99.000	0.000	0.000	0.000	0.000	0.000
spacey	2	98.000	0.000	0.211	0.000	0.667	0.095
electro	4	96.000	0.000	0.231	0.000	0.000	0.308
calm	6	94.000	0.000	0.087	0.000	0.000	0.000
voice	10.000	90.000	0.000	0.545	0.000	0.333	0.143
vocals	16.000	84.000	0.000	0.512	0.273	0.387	0.200
singer	1	99.000	0.000	0.069	0.500	0.000	0.000
strings	47.000	53.000	0.000	0.905	0.151	0.872	0.780
orchestra	5	95.000	0.000	0.138	0.000	0.000	0.000
chant	4	96.000	0.000	0.444	0.000	0.400	0.667
heavymetal	2	98.000	0.000	0.364	0.000	0.000	0.308
girl	2	98.000	0.000	0.083	0.000	0.000	0.333
flute	8	92.000	0.000	0.444	0.500	0.316	0.143
drum	3	97.000	0.000	0.160	0.000	0.500	0.000
classic	24.000	76.000	0.000	0.687	0.148	0.368	0.462
nosinging	3	97.000	0.000	0.094	0.000	0.000	0.000
chanting	2	98.000	0.000	0.222	0.000	0.000	0.000
folk	2	98.000	0.000	0.105	0.000	0.000	0.000
malesinger	1	99.000	0.000	0.077	0.500	0.286	0.000
mellow	5	95.000	0.000	0.176	0.000	0.000	0.000
indian	3	97.000	0.000	0.154	0.000	0.000	0.400
electronica	2	98.000	0.000	0.125	0.000	0.000	0.200
women	3	97.000	0.000	0.235	0.000	0.500	0.333
soft	21.000	79.000	0.000	0.353	0.000	0.308	0.242
malevoice	9	91.000	0.000	0.452	0.000	0.471	0.143
organ	1	99.000	0.000	0.000	0.000	0.000	0.000
female	19.000	81.000	0.000	0.842	0.516	0.688	0.444
classicalguitar	1	99.000	0.000	0.500	0.000	0.000	0.500
airy	4	96.000	0.000	0.258	0.000	0.400	0.381
malevocal	11.000	89.000	0.000	0.545	0.333	0.571	0.286
clapping	2	98.000	0.000	0.000	0.000	0.000	0.000
choir	7	93.000	0.000	0.615	0.250	0.308	0.500

download these results as csv

Binary Accuracy

Full dataset

Tag	Positive Examples	Negative Examples	LabX	Mandel	Manzagol	Marsyas	Zhi
nostrings	13.000	6486.000	0.998	0.496	0.998	0.992	0.986
chimes	22.000	6477.000	0.986	0.778	0.994	0.992	0.981
sad	18.000	6481.000	0.894	0.705	0.995	0.989	0.862
nodrums	48.000	6451.000	0.993	0.569	0.991	0.977	0.824
femalevoice	105.000	6394.000	0.984	0.851	0.962	0.945	0.950
horn	7	6492.000	0.996	0.801	0.997	0.991	0.978
pop	196.000	6303.000	0.970	0.749	0.877	0.921	0.838
rock	601.000	5898.000	0.908	0.878	0.922	0.863	0.859
house	22.000	6477.000	0.997	0.793	0.989	0.991	0.939
birds	7	6492.000	0.998	0.922	0.997	0.991	0.923
harpsicord	59.000	6440.000	0.991	0.929	0.979	0.964	0.864
strange	22.000	6477.000	0.997	0.588	0.996	0.986	0.960
noflute	35.000	6464.000	0.987	0.559	0.994	0.982	0.939
novocal	263.000	6236.000	0.960	0.548	0.952	0.870	0.816
solo	217.000	6282.000	0.966	0.831	0.960	0.890	0.861
notenglish	11.000	6488.000	0.998	0.839	0.998	0.990	0.972
novoice	146.000	6353.000	0.978	0.569	0.972	0.935	0.865
newage	157.000	6342.000	0.976	0.759	0.974	0.936	0.783
synth	294.000	6205.000	0.955	0.696	0.949	0.874	0.789
upbeat	52.000	6447.000	0.992	0.693	0.979	0.980	0.883
slow	1043.000	5456.000	0.840	0.708	0.814	0.716	0.743
deep	12.000	6487.000	0.998	0.830	0.996	0.991	0.953
fiddle	14.000	6485.000	0.998	0.817	0.991	0.989	0.950
orchestral	12.000	6487.000	0.998	0.779	0.995	0.989	0.908
notclassical	14.000	6485.000	0.998	0.660	0.996	0.992	0.945
mansinging	46.000	6453.000	0.993	0.748	0.972	0.982	0.955
wind	22.000	6477.000	0.994	0.834	0.987	0.984	0.878
piano	630.000	5869.000	0.903	0.879	0.891	0.758	0.889
spanish	65.000	6434.000	0.990	0.826	0.973	0.976	0.954
femalesinger	30.000	6469.000	0.995	0.870	0.992	0.984	0.974
singing	242.000	6257.000	0.963	0.802	0.942	0.889	0.931
quiet	263.000	6236.000	0.960	0.792	0.957	0.890	0.813
oboe	12.000	6487.000	0.998	0.935	0.977	0.977	0.917
tribal	40.000	6459.000	0.994	0.770	0.992	0.975	0.973
noguitar	46.000	6453.000	0.993	0.560	0.992	0.972	0.916
femalevocal	126.000	6373.000	0.981	0.870	0.936	0.933	0.949
fastbeat	33.000	6466.000	0.995	0.784	0.991	0.989	0.914
hiphop	32.000	6467.000	0.995	0.906	0.992	0.991	0.978
instrumental	102.000	6397.000	0.976	0.598	0.965	0.950	0.819
chorus	50.000	6449.000	0.992	0.939	0.982	0.970	0.965
silence	12.000	6487.000	0.983	0.901	0.992	0.991	0.959
duet	18.000	6481.000	0.997	0.890	0.986	0.988	0.938
sax	20.000	6479.000	0.974	0.896	0.986	0.988	0.960
nobeat	14.000	6485.000	0.994	0.677	0.997	0.990	0.943
nopiano	90.000	6409.000	0.948	0.571	0.983	0.956	0.894
novocals	326.000	6173.000	0.950	0.546	0.948	0.856	0.814
pianosolo	13.000	6486.000	0.998	0.854	0.986	0.991	0.932
low	35.000	6464.000	0.995	0.816	0.986	0.981	0.872
weird	120.000	6379.000	0.982	0.641	0.975	0.946	0.914
dance	184.000	6315.000	0.972	0.823	0.941	0.939	0.888
harp	137.000	6362.000	0.979	0.885	0.974	0.932	0.882
horns	12.000	6487.000	0.989	0.935	0.992	0.988	0.975
funky	66.000	6433.000	0.990	0.774	0.972	0.979	0.916
hardrock	80.000	6419.000	0.988	0.894	0.981	0.968	0.887
bells	36.000	6463.000	0.994	0.731	0.989	0.981	0.972
punk	42.000	6457.000	0.964	0.927	0.959	0.986	0.924
electricguitar	51.000	6448.000	0.992	0.820	0.988	0.978	0.868
techno	827.000	5672.000	0.873	0.832	0.892	0.853	0.877
modern	73.000	6426.000	0.989	0.592	0.975	0.965	0.896
violins	258.000	6241.000	0.939	0.819	0.933	0.922	0.849
noviolin	18.000	6481.000	0.997	0.519	0.996	0.989	0.965
opera	325.000	6174.000	0.950	0.954	0.955	0.847	0.955
india	22.000	6477.000	0.996	0.770	0.996	0.991	0.989
cello	145.000	6354.000	0.978	0.947	0.950	0.913	0.955
sitar	250.000	6249.000	0.962	0.890	0.960	0.928	0.890
hard	25.000	6474.000	0.991	0.884	0.994	0.990	0.894
banjo	15.000	6484.000	0.998	0.959	0.994	0.977	0.942
blues	42.000	6457.000	0.994	0.921	0.987	0.982	0.989
man	128.000	6371.000	0.980	0.792	0.964	0.951	0.958
water	12.000	6487.000	0.998	0.911	0.994	0.991	0.896
femalevocals	90.000	6409.000	0.986	0.867	0.962	0.957	0.964
beat	534.000	5965.000	0.918	0.755	0.910	0.893	0.865
vocal	346.000	6153.000	0.947	0.784	0.930	0.847	0.916
jazz	88.000	6411.000	0.986	0.812	0.940	0.957	0.905
male	316.000	6183.000	0.951	0.822	0.934	0.895	0.922
maleopera	18.000	6481.000	0.997	0.966	0.991	0.988	0.966
drums	663.000	5836.000	0.898	0.714	0.863	0.803	0.819
electronic	578.000	5921.000	0.911	0.737	0.902	0.818	0.817
talking	27.000	6472.000	0.996	0.834	0.995	0.988	0.985
violin	908.000	5591.000	0.860	0.876	0.872	0.806	0.863
bass	73.000	6426.000	0.989	0.698	0.976	0.965	0.902
notrock	19.000	6480.000	0.997	0.493	0.996	0.991	0.983
string	91.000	6408.000	0.986	0.719	0.972	0.959	0.846
womansinging	32.000	6467.000	0.995	0.854	0.990	0.982	0.982
guitar	1166.000	5333.000	0.821	0.821	0.851	0.691	0.843
medieval	39.000	6460.000	0.994	0.791	0.988	0.976	0.874
clarinet	16.000	6483.000	0.998	0.893	0.992	0.989	0.950
world	14.000	6485.000	0.998	0.605	0.996	0.992	0.983
old	14.000	6485.000	0.998	0.713	0.995	0.993	0.821
middleeastern	17.000	6482.000	0.968	0.680	0.989	0.987	0.953
baroque	81.000	6418.000	0.854	0.843	0.980	0.955	0.796
oriental	50.000	6449.000	0.992	0.725	0.974	0.980	0.936
trumpet	17.000	6482.000	0.958	0.847	0.996	0.990	0.992
irish	49.000	6450.000	0.992	0.858	0.983	0.976	0.912
ambient	419.000	6080.000	0.936	0.866	0.936	0.858	0.814
funk	32.000	6467.000	0.995	0.870	0.987	0.990	0.944
metal	159.000	6340.000	0.842	0.912	0.973	0.939	0.887
woman	186.000	6313.000	0.971	0.889	0.956	0.914	0.948
dark	36.000	6463.000	0.994	0.798	0.992	0.979	0.856
acoustic	66.000	6433.000	0.974	0.842	0.966	0.979	0.941
light	16.000	6483.000	0.998	0.607	0.996	0.989	0.954
repetitive	24.000	6475.000	0.948	0.755	0.996	0.992	0.989
trance	51.000	6448.000	0.992	0.757	0.985	0.975	0.881
celtic	27.000	6472.000	0.996	0.751	0.992	0.987	0.963
electric	44.000	6455.000	0.993	0.610	0.971	0.976	0.932
malevocals	123.000	6376.000	0.930	0.776	0.958	0.952	0.872
heavy	59.000	6440.000	0.928	0.873	0.984	0.975	0.876
jazzy	68.000	6431.000	0.990	0.806	0.939	0.968	0.919
country	122.000	6377.000	0.981	0.880	0.925	0.945	0.952
beats	157.000	6342.000	0.967	0.721	0.952	0.947	0.881
loud	313.000	6186.000	0.952	0.827	0.947	0.923	0.842
classical	1544.000	4955.000	0.762	0.848	0.768	0.569	0.788
voices	39.000	6460.000	0.994	0.842	0.990	0.976	0.965
flutes	54.000	6445.000	0.992	0.904	0.982	0.976	0.957
choral	104.000	6395.000	0.984	0.952	0.984	0.959	0.971
harpsichord	263.000	6236.000	0.960	0.906	0.908	0.894	0.828
eastern	80.000	6419.000	0.988	0.736	0.978	0.959	0.920
foreign	51.000	6448.000	0.992	0.824	0.974	0.970	0.980
fast	616.000	5883.000	0.905	0.723	0.902	0.804	0.799
english	11.000	6488.000	0.998	0.768	0.991	0.991	0.978
spacey	27.000	6472.000	0.996	0.857	0.994	0.983	0.892
electro	87.000	6412.000	0.985	0.686	0.980	0.957	0.874
calm	33.000	6466.000	0.984	0.653	0.989	0.985	0.921
lute	15.000	6484.000	0.998	0.922	0.990	0.984	0.937
arabic	10.000	6489.000	0.998	0.732	0.984	0.988	0.971
voice	111.000	6388.000	0.983	0.708	0.964	0.938	0.938
vocals	256.000	6243.000	0.961	0.743	0.906	0.886	0.920
rap	41.000	6458.000	0.994	0.934	0.978	0.986	0.985
singer	25.000	6474.000	0.996	0.736	0.979	0.986	0.985
strings	997.000	5502.000	0.847	0.795	0.841	0.756	0.771
orchestra	98.000	6401.000	0.985	0.840	0.975	0.945	0.860
guitars	25.000	6474.000	0.996	0.730	0.992	0.990	0.919
chant	51.000	6448.000	0.992	0.950	0.989	0.982	0.971
heavymetal	43.000	6456.000	0.993	0.904	0.982	0.986	0.918
girl	10.000	6489.000	0.976	0.756	0.997	0.992	0.971
percussion	26.000	6473.000	0.862	0.729	0.993	0.981	0.961
flute	455.000	6044.000	0.930	0.927	0.944	0.889	0.902
drum	89.000	6410.000	0.986	0.689	0.976	0.960	0.931
classic	235.000	6264.000	0.964	0.760	0.938	0.888	0.807
nosinging	51.000	6448.000	0.992	0.529	0.987	0.977	0.926
chanting	32.000	6467.000	0.995	0.945	0.994	0.984	0.969
folk	48.000	6451.000	0.993	0.802	0.983	0.973	0.968
malesinger	39.000	6460.000	0.994	0.748	0.927	0.980	0.942
mellow	29.000	6470.000	0.992	0.649	0.992	0.987	0.958
indian	313.000	6186.000	0.952	0.813	0.942	0.878	0.927
electronica	39.000	6460.000	0.994	0.629	0.988	0.979	0.921
women	22.000	6477.000	0.997	0.886	0.995	0.990	0.981
notopera	19.000	6480.000	0.997	0.558	0.997	0.991	0.989
noise	16.000	6483.000	0.998	0.792	0.996	0.981	0.901
soft	248.000	6251.000	0.962	0.754	0.947	0.889	0.823
femaleopera	27.000	6472.000	0.996	0.936	0.985	0.980	0.949
malevoice	155.000	6344.000	0.976	0.766	0.949	0.940	0.933
organ	17.000	6482.000	0.997	0.786	0.951	0.973	0.867
female	320.000	6179.000	0.951	0.905	0.896	0.856	0.936
classicalguitar	38.000	6461.000	0.994	0.958	0.973	0.975	0.932
operatic	17.000	6482.000	0.997	0.867	0.996	0.989	0.943
airy	12.000	6487.000	0.988	0.762	0.997	0.992	0.881
malevocal	271.000	6228.000	0.958	0.810	0.908	0.911	0.889
clapping	12.000	6487.000	0.998	0.815	0.997	0.992	0.996
choir	161.000	6338.000	0.975	0.958	0.983	0.951	0.974

download these results as csv

100 query subset used in Tagatune evaluation

Tag	Positive Examples	Negative Examples	LabX	Mandel	Manzagol	Marsyas	Zhi
sad	5	95.000	0.860	0.600	0.960	0.950	0.780
nodrums	4	96.000	0.960	0.470	0.960	0.950	0.770
femalevoice	6	94.000	0.940	0.840	0.900	0.910	0.890
pop	10.000	90.000	0.900	0.790	0.890	0.940	0.860
rock	14.000	86.000	0.860	0.910	0.880	0.890	0.900
birds	1	99.000	0.990	0.920	0.990	0.990	0.830
harpsicord	3	97.000	0.970	0.910	0.950	0.970	0.760
strange	2	98.000	0.980	0.570	0.980	0.960	0.930
novocal	12.000	88.000	0.880	0.590	0.890	0.820	0.780
solo	11.000	89.000	0.890	0.790	0.890	0.870	0.850
notenglish	1	99.000	0.990	0.820	0.990	0.990	0.930
novoice	4	96.000	0.960	0.640	0.960	0.940	0.930
newage	12.000	88.000	0.880	0.780	0.880	0.910	0.760
synth	11.000	89.000	0.890	0.830	0.900	0.940	0.840
upbeat	4	96.000	0.960	0.780	0.950	0.960	0.910
slow	44.000	56.000	0.560	0.790	0.600	0.760	0.660
deep	2	98.000	0.980	0.800	0.980	0.970	0.910
fiddle	1	99.000	0.990	0.730	0.990	0.980	0.920
orchestral	2	98.000	0.980	0.700	0.980	0.980	0.710
mansinging	2	98.000	0.980	0.800	0.980	0.950	0.960
wind	1	99.000	0.990	0.810	0.980	0.980	0.830
piano	9	91.000	0.910	0.800	0.820	0.650	0.890
femalesinger	4	96.000	0.960	0.810	0.940	0.940	0.940
singing	13.000	87.000	0.870	0.890	0.870	0.870	0.900
quiet	16.000	84.000	0.840	0.780	0.850	0.860	0.820
tribal	1	99.000	0.990	0.900	0.990	0.980	1.000
noguitar	2	98.000	0.980	0.490	0.980	0.970	0.890
femalevocal	13.000	87.000	0.870	0.850	0.830	0.910	0.890
fastbeat	1	99.000	0.990	0.850	0.990	0.980	0.920
instrumental	4	96.000	0.960	0.470	0.950	0.930	0.860
chorus	3	97.000	0.970	0.950	0.970	0.940	0.960
silence	1	99.000	0.960	0.920	0.990	0.980	0.960
sax	2	98.000	0.970	0.890	0.970	0.980	0.930
nobeat	1	99.000	0.990	0.570	0.990	0.980	0.880
nopiano	3	97.000	0.910	0.430	0.970	0.960	0.840
novocals	15.000	85.000	0.850	0.560	0.850	0.790	0.790
low	5	95.000	0.950	0.770	0.950	0.940	0.830
weird	3	97.000	0.970	0.710	0.970	0.920	0.930
dance	4	96.000	0.960	0.950	0.940	0.950	0.940
harp	3	97.000	0.970	0.900	0.970	0.950	0.950
horns	3	97.000	0.970	0.850	0.970	0.950	0.950
funky	1	99.000	0.990	0.850	0.990	0.970	0.970
hardrock	4	96.000	0.960	0.930	0.960	0.950	0.890
bells	2	98.000	0.980	0.770	0.990	0.970	0.980
punk	4	96.000	0.900	0.950	0.990	0.960	0.920
techno	12.000	88.000	0.880	0.890	0.920	0.880	0.870
modern	5	95.000	0.950	0.680	0.960	0.940	0.930
violins	25.000	75.000	0.730	0.790	0.770	0.780	0.760
opera	6	94.000	0.940	0.940	0.910	0.860	0.940
cello	21.000	79.000	0.790	0.870	0.810	0.770	0.840
sitar	1	99.000	0.990	0.910	0.950	0.930	0.980
man	4	96.000	0.960	0.800	0.970	0.940	0.950
femalevocals	9	91.000	0.910	0.860	0.900	0.890	0.900
beat	13.000	87.000	0.870	0.810	0.870	0.900	0.880
vocal	22.000	78.000	0.780	0.860	0.760	0.890	0.810
jazz	3	97.000	0.970	0.920	0.960	0.930	0.920
male	10.000	90.000	0.900	0.820	0.870	0.900	0.880
drums	14.000	86.000	0.860	0.830	0.860	0.860	0.870
electronic	16.000	84.000	0.840	0.850	0.870	0.910	0.860
violin	44.000	56.000	0.560	0.900	0.840	0.850	0.770
bass	5	95.000	0.950	0.710	0.950	0.940	0.930
string	12.000	88.000	0.880	0.640	0.880	0.850	0.790
womansinging	3	97.000	0.970	0.810	0.940	0.950	0.960
guitar	15.000	85.000	0.850	0.840	0.900	0.630	0.870
medieval	5	95.000	0.950	0.780	0.950	0.950	0.830
old	1	99.000	0.990	0.620	0.990	0.990	0.690
middleeastern	1	99.000	0.950	0.590	0.990	0.990	0.950
baroque	7	93.000	0.830	0.790	0.930	0.900	0.680
oriental	1	99.000	0.990	0.680	0.970	0.990	0.990
trumpet	2	98.000	0.940	0.820	0.980	0.970	0.980
irish	1	99.000	0.990	0.800	0.980	0.980	0.930
ambient	14.000	86.000	0.860	0.880	0.860	0.890	0.720
funk	2	98.000	0.980	0.950	0.980	0.980	0.980
metal	5	95.000	0.790	0.960	0.960	0.900	0.900
woman	14.000	86.000	0.860	0.910	0.860	0.900	0.870
dark	1	99.000	0.990	0.760	0.990	0.990	0.790
acoustic	3	97.000	0.960	0.900	0.960	0.980	0.980
light	1	99.000	0.990	0.520	0.990	0.980	0.980
trance	4	96.000	0.960	0.870	0.960	0.950	0.930
celtic	1	99.000	0.990	0.670	0.990	0.980	0.920
electric	1	99.000	0.990	0.750	0.960	0.980	0.940
malevocals	6	94.000	0.890	0.780	0.940	0.890	0.860
heavy	4	96.000	0.930	0.880	0.970	0.940	0.890
jazzy	3	97.000	0.970	0.890	0.980	0.930	0.930
country	2	98.000	0.980	0.880	0.940	0.930	0.960
beats	4	96.000	0.950	0.790	0.930	0.960	0.880
loud	11.000	89.000	0.890	0.850	0.890	0.930	0.880
classical	48.000	52.000	0.520	0.850	0.500	0.790	0.810
voices	2	98.000	0.980	0.880	0.980	0.970	0.970
choral	7	93.000	0.930	0.940	0.920	0.900	0.920
harpsichord	12.000	88.000	0.880	0.890	0.870	0.930	0.740
eastern	1	99.000	0.990	0.800	0.990	0.970	0.990
foreign	2	98.000	0.980	0.840	0.960	0.960	0.960
fast	17.000	83.000	0.830	0.780	0.820	0.850	0.820
english	1	99.000	0.990	0.780	0.980	0.970	0.970
spacey	2	98.000	0.980	0.850	0.980	0.990	0.810
electro	4	96.000	0.960	0.800	0.950	0.960	0.910
calm	6	94.000	0.930	0.580	0.920	0.940	0.920
voice	10.000	90.000	0.900	0.850	0.880	0.880	0.880
vocals	16.000	84.000	0.840	0.790	0.840	0.810	0.840
singer	1	99.000	0.990	0.730	0.980	0.960	0.960
strings	47.000	53.000	0.530	0.910	0.550	0.880	0.820
orchestra	5	95.000	0.950	0.750	0.940	0.910	0.690
chant	4	96.000	0.960	0.950	0.960	0.970	0.980
heavymetal	2	98.000	0.980	0.930	0.970	0.980	0.910
girl	2	98.000	0.960	0.780	0.980	0.960	0.960
flute	8	92.000	0.920	0.900	0.940	0.870	0.880
drum	3	97.000	0.970	0.790	0.970	0.980	0.950
classic	24.000	76.000	0.760	0.790	0.770	0.760	0.720
nosinging	3	97.000	0.970	0.420	0.970	0.970	0.920
chanting	2	98.000	0.980	0.930	0.980	0.970	0.960
folk	2	98.000	0.980	0.830	0.980	0.950	0.970
malesinger	1	99.000	0.990	0.760	0.980	0.950	0.920
mellow	5	95.000	0.940	0.720	0.950	0.940	0.920
indian	3	97.000	0.970	0.890	0.960	0.850	0.970
electronica	2	98.000	0.980	0.720	0.980	0.960	0.920
women	3	97.000	0.970	0.870	0.970	0.980	0.960
soft	21.000	79.000	0.790	0.670	0.770	0.730	0.750
malevoice	9	91.000	0.910	0.830	0.890	0.910	0.880
organ	1	99.000	0.990	0.730	0.920	0.990	0.830
female	19.000	81.000	0.810	0.940	0.850	0.900	0.850
classicalguitar	1	99.000	0.990	0.980	0.990	0.980	0.980
airy	4	96.000	0.950	0.770	0.960	0.970	0.870
malevocal	11.000	89.000	0.890	0.850	0.880	0.910	0.850
clapping	2	98.000	0.980	0.870	0.980	0.970	0.980
choir	7	93.000	0.930	0.950	0.940	0.910	0.940

download these results as csv

Positive Example Accuracy

Full dataset

Tag	Positive Examples	Negative Examples	LabX	Mandel	Manzagol	Marsyas	Zhi
nostrings	13.000	6486.000	0.000	0.615	0.000	0.000	0.000
chimes	22.000	6477.000	0.045	0.545	0.000	0.045	0.136
sad	18.000	6481.000	0.111	0.778	0.056	0.000	0.667
nodrums	48.000	6451.000	0.000	0.562	0.000	0.000	0.208
femalevoice	105.000	6394.000	0.000	0.762	0.105	0.295	0.333
horn	7	6492.000	0.000	0.286	0.000	0.000	0.143
pop	196.000	6303.000	0.000	0.827	0.459	0.444	0.505
rock	601.000	5898.000	0.000	0.847	0.426	0.907	0.839
house	22.000	6477.000	0.000	0.773	0.045	0.000	0.273
birds	7	6492.000	0.000	0.429	0.000	0.143	0.714
harpsicord	59.000	6440.000	0.000	0.780	0.169	0.525	0.780
strange	22.000	6477.000	0.000	0.909	0.000	0.091	0.136
noflute	35.000	6464.000	0.000	0.314	0.000	0.000	0.086
novocal	263.000	6236.000	0.000	0.616	0.004	0.186	0.202
solo	217.000	6282.000	0.000	0.641	0.055	0.369	0.484
notenglish	11.000	6488.000	0.000	0.909	0.000	0.091	0.455
novoice	146.000	6353.000	0.000	0.589	0.007	0.089	0.151
newage	157.000	6342.000	0.000	0.803	0.000	0.280	0.592
synth	294.000	6205.000	0.000	0.786	0.051	0.422	0.554
upbeat	52.000	6447.000	0.000	0.808	0.038	0.096	0.442
slow	1043.000	5456.000	0.000	0.705	0.199	0.730	0.516
deep	12.000	6487.000	0.000	0.583	0.083	0.000	0.250
fiddle	14.000	6485.000	0.000	0.786	0.000	0.071	0.214
orchestral	12.000	6487.000	0.000	0.500	0.000	0.000	0.750
notclassical	14.000	6485.000	0.000	0.500	0.000	0.000	0.214
mansinging	46.000	6453.000	0.000	0.783	0.022	0.087	0.239
wind	22.000	6477.000	0.045	0.636	0.045	0.000	0.591
piano	630.000	5869.000	0.000	0.767	0.629	0.805	0.657
spanish	65.000	6434.000	0.000	0.462	0.015	0.077	0.169
femalesinger	30.000	6469.000	0.000	0.700	0.067	0.233	0.267
singing	242.000	6257.000	0.000	0.777	0.103	0.529	0.264
quiet	263.000	6236.000	0.000	0.719	0.030	0.707	0.620
oboe	12.000	6487.000	0.000	0.167	0.167	0.000	0.083
tribal	40.000	6459.000	0.000	0.425	0.025	0.200	0.200
noguitar	46.000	6453.000	0.000	0.565	0.000	0.022	0.304
femalevocal	126.000	6373.000	0.000	0.786	0.135	0.452	0.333
fastbeat	33.000	6466.000	0.000	0.636	0.000	0.000	0.515
hiphop	32.000	6467.000	0.000	0.594	0.219	0.000	0.312
instrumental	102.000	6397.000	0.000	0.598	0.029	0.088	0.294
chorus	50.000	6449.000	0.000	0.760	0.400	0.000	0.700
silence	12.000	6487.000	0.000	0.833	0.167	0.000	0.333
duet	18.000	6481.000	0.000	0.278	0.000	0.000	0.167
sax	20.000	6479.000	0.000	0.200	0.000	0.050	0.000
nobeat	14.000	6485.000	0.000	0.571	0.000	0.000	0.429
nopiano	90.000	6409.000	0.033	0.533	0.000	0.011	0.089
novocals	326.000	6173.000	0.000	0.610	0.003	0.190	0.206
pianosolo	13.000	6486.000	0.000	0.692	0.462	0.000	0.923
low	35.000	6464.000	0.000	0.686	0.143	0.057	0.371
weird	120.000	6379.000	0.000	0.783	0.025	0.250	0.267
dance	184.000	6315.000	0.000	0.864	0.283	0.234	0.712
harp	137.000	6362.000	0.000	0.431	0.058	0.277	0.409
horns	12.000	6487.000	0.000	0.167	0.083	0.000	0.083
funky	66.000	6433.000	0.000	0.879	0.121	0.000	0.485
hardrock	80.000	6419.000	0.000	0.963	0.100	0.000	0.950
bells	36.000	6463.000	0.000	0.528	0.028	0.083	0.111
punk	42.000	6457.000	0.000	0.786	0.595	0.000	0.810
electricguitar	51.000	6448.000	0.000	0.588	0.059	0.176	0.588
techno	827.000	5672.000	0.000	0.926	0.336	0.900	0.790
modern	73.000	6426.000	0.000	0.699	0.055	0.082	0.260
violins	258.000	6241.000	0.000	0.841	0.155	0.275	0.636
noviolin	18.000	6481.000	0.000	0.611	0.056	0.056	0.056
opera	325.000	6174.000	0.000	0.926	0.649	0.905	0.766
india	22.000	6477.000	0.000	0.864	0.045	0.045	0.318
cello	145.000	6354.000	0.000	0.717	0.414	0.510	0.366
sitar	250.000	6249.000	0.000	0.868	0.432	0.624	0.676
hard	25.000	6474.000	0.000	0.960	0.040	0.000	0.920
banjo	15.000	6484.000	0.000	0.133	0.067	0.067	0.333
blues	42.000	6457.000	0.000	0.643	0.119	0.190	0.048
man	128.000	6371.000	0.000	0.805	0.023	0.375	0.258
water	12.000	6487.000	0.000	0.667	0.000	0.000	0.750
femalevocals	90.000	6409.000	0.000	0.700	0.100	0.267	0.211
beat	534.000	5965.000	0.000	0.876	0.202	0.723	0.695
vocal	346.000	6153.000	0.000	0.777	0.075	0.601	0.234
jazz	88.000	6411.000	0.000	0.761	0.182	0.284	0.466
male	316.000	6183.000	0.000	0.826	0.161	0.525	0.332
maleopera	18.000	6481.000	0.000	0.889	0.278	0.000	0.889
drums	663.000	5836.000	0.000	0.839	0.207	0.691	0.514
electronic	578.000	5921.000	0.000	0.846	0.102	0.713	0.640
talking	27.000	6472.000	0.000	0.704	0.037	0.000	0.037
violin	908.000	5591.000	0.000	0.882	0.744	0.885	0.699
bass	73.000	6426.000	0.000	0.753	0.041	0.219	0.438
notrock	19.000	6480.000	0.000	0.368	0.000	0.053	0.105
string	91.000	6408.000	0.000	0.736	0.033	0.121	0.275
womansinging	32.000	6467.000	0.000	0.938	0.031	0.250	0.219
guitar	1166.000	5333.000	0.000	0.701	0.359	0.886	0.522
medieval	39.000	6460.000	0.000	0.795	0.077	0.026	0.410
clarinet	16.000	6483.000	0.000	0.625	0.000	0.000	0.375
world	14.000	6485.000	0.000	0.643	0.071	0.000	0.286
old	14.000	6485.000	0.000	0.786	0.000	0.071	0.500
middleeastern	17.000	6482.000	0.000	0.529	0.059	0.118	0.118
baroque	81.000	6418.000	0.111	0.864	0.012	0.333	0.840
oriental	50.000	6449.000	0.000	0.700	0.100	0.100	0.320
trumpet	17.000	6482.000	0.000	0.471	0.059	0.000	0.000
irish	49.000	6450.000	0.000	0.714	0.020	0.163	0.184
ambient	419.000	6080.000	0.000	0.788	0.014	0.726	0.644
funk	32.000	6467.000	0.000	0.875	0.125	0.000	0.344
metal	159.000	6340.000	0.019	0.899	0.126	0.000	0.969
woman	186.000	6313.000	0.000	0.801	0.091	0.565	0.457
dark	36.000	6463.000	0.000	0.861	0.028	0.000	0.361
acoustic	66.000	6433.000	0.015	0.682	0.182	0.121	0.409
light	16.000	6483.000	0.000	0.750	0.000	0.125	0.062
repetitive	24.000	6475.000	0.000	0.417	0.000	0.000	0.000
trance	51.000	6448.000	0.000	0.804	0.020	0.098	0.510
celtic	27.000	6472.000	0.000	0.741	0.000	0.111	0.074
electric	44.000	6455.000	0.000	0.659	0.000	0.023	0.205
malevocals	123.000	6376.000	0.130	0.821	0.154	0.276	0.382
heavy	59.000	6440.000	0.000	0.864	0.169	0.000	0.932
jazzy	68.000	6431.000	0.000	0.824	0.324	0.191	0.485
country	122.000	6377.000	0.000	0.697	0.328	0.344	0.189
beats	157.000	6342.000	0.006	0.866	0.140	0.344	0.707
loud	313.000	6186.000	0.000	0.799	0.096	0.645	0.764
classical	1544.000	4955.000	0.000	0.852	0.158	0.994	0.720
voices	39.000	6460.000	0.000	0.615	0.000	0.077	0.333
flutes	54.000	6445.000	0.000	0.815	0.463	0.000	0.759
choral	104.000	6395.000	0.000	0.846	0.202	0.442	0.817
harpsichord	263.000	6236.000	0.000	0.768	0.684	0.821	0.867
eastern	80.000	6419.000	0.000	0.787	0.100	0.375	0.400
foreign	51.000	6448.000	0.000	0.725	0.039	0.275	0.216
fast	616.000	5883.000	0.000	0.701	0.094	0.646	0.433
english	11.000	6488.000	0.000	0.364	0.000	0.000	0.091
spacey	27.000	6472.000	0.000	0.852	0.000	0.111	0.444
electro	87.000	6412.000	0.000	0.805	0.011	0.138	0.471
calm	33.000	6466.000	0.000	0.545	0.000	0.061	0.182
lute	15.000	6484.000	0.000	0.867	0.200	0.000	0.733
arabic	10.000	6489.000	0.000	0.300	0.100	0.000	0.000
voice	111.000	6388.000	0.000	0.802	0.063	0.234	0.171
vocals	256.000	6243.000	0.000	0.695	0.109	0.367	0.234
rap	41.000	6458.000	0.000	0.659	0.488	0.000	0.488
singer	25.000	6474.000	0.000	0.800	0.080	0.040	0.000
strings	997.000	5502.000	0.000	0.822	0.085	0.825	0.640
orchestra	98.000	6401.000	0.000	0.704	0.102	0.245	0.571
guitars	25.000	6474.000	0.000	0.400	0.000	0.000	0.400
chant	51.000	6448.000	0.000	0.745	0.157	0.627	0.647
heavymetal	43.000	6456.000	0.000	0.860	0.163	0.000	0.860
girl	10.000	6489.000	0.000	0.900	0.000	0.000	0.400
percussion	26.000	6473.000	0.077	0.692	0.000	0.077	0.308
flute	455.000	6044.000	0.000	0.807	0.569	0.732	0.631
drum	89.000	6410.000	0.000	0.798	0.056	0.180	0.281
classic	235.000	6264.000	0.000	0.881	0.102	0.340	0.604
nosinging	51.000	6448.000	0.000	0.569	0.000	0.059	0.059
chanting	32.000	6467.000	0.000	0.406	0.031	0.156	0.438
folk	48.000	6451.000	0.000	0.500	0.021	0.083	0.146
malesinger	39.000	6460.000	0.000	0.718	0.282	0.154	0.256
mellow	29.000	6470.000	0.000	0.483	0.000	0.000	0.069
indian	313.000	6186.000	0.000	0.770	0.137	0.594	0.278
electronica	39.000	6460.000	0.000	0.897	0.000	0.103	0.308
women	22.000	6477.000	0.000	0.818	0.045	0.182	0.409
notopera	19.000	6480.000	0.000	0.316	0.000	0.000	0.053
noise	16.000	6483.000	0.000	0.688	0.062	0.000	0.438
soft	248.000	6251.000	0.000	0.641	0.056	0.395	0.472
femaleopera	27.000	6472.000	0.000	0.926	0.259	0.000	0.889
malevoice	155.000	6344.000	0.000	0.806	0.071	0.297	0.252
organ	17.000	6482.000	0.000	0.294	0.059	0.059	0.235
female	320.000	6179.000	0.000	0.822	0.341	0.697	0.447
classicalguitar	38.000	6461.000	0.000	0.789	0.263	0.000	0.868
operatic	17.000	6482.000	0.000	0.941	0.000	0.000	0.824
airy	12.000	6487.000	0.083	0.833	0.000	0.083	0.750
malevocal	271.000	6228.000	0.000	0.856	0.277	0.465	0.373
clapping	12.000	6487.000	0.000	0.417	0.000	0.000	0.000
choir	161.000	6338.000	0.000	0.876	0.404	0.745	0.770

download these results as csv

100 query subset used in Tagatune evaluation

Tag	Positive Examples	Negative Examples	LabX	Mandel	Manzagol	Marsyas	Zhi
sad	5	95.000	0.200	0.800	0.200	0.000	0.600
nodrums	4	96.000	0.000	0.750	0.000	0.000	0.500
femalevoice	6	94.000	0.000	0.833	0.000	0.333	0.167
pop	10.000	90.000	0.000	0.900	0.600	0.700	0.700
rock	14.000	86.000	0.000	0.786	0.214	1.000	0.786
birds	1	99.000	0.000	1.000	0.000	0.000	0.000
harpsicord	3	97.000	0.000	1.000	0.000	0.000	1.000
strange	2	98.000	0.000	1.000	0.000	0.000	0.000
novocal	12.000	88.000	0.000	0.833	0.083	0.083	0.083
solo	11.000	89.000	0.000	0.364	0.091	0.182	0.182
notenglish	1	99.000	0.000	0.000	0.000	0.000	0.000
novoice	4	96.000	0.000	0.750	0.000	0.000	0.000
newage	12.000	88.000	0.000	0.667	0.000	0.250	0.500
synth	11.000	89.000	0.000	0.818	0.091	0.545	0.364
upbeat	4	96.000	0.000	0.750	0.000	0.000	0.500
slow	44.000	56.000	0.000	0.795	0.159	0.750	0.364
deep	2	98.000	0.000	0.500	0.000	0.000	0.000
fiddle	1	99.000	0.000	1.000	0.000	0.000	0.000
orchestral	2	98.000	0.000	0.500	0.000	0.000	1.000
mansinging	2	98.000	0.000	1.000	0.000	0.500	0.500
wind	1	99.000	0.000	1.000	0.000	0.000	1.000
piano	9	91.000	0.000	0.444	0.111	0.778	0.222
femalesinger	4	96.000	0.000	0.750	0.000	0.000	0.000
singing	13.000	87.000	0.000	0.846	0.154	0.462	0.385
quiet	16.000	84.000	0.000	0.688	0.062	0.688	0.500
tribal	1	99.000	0.000	1.000	0.000	0.000	1.000
noguitar	2	98.000	0.000	1.000	0.000	0.000	0.000
femalevocal	13.000	87.000	0.000	0.692	0.154	0.538	0.308
fastbeat	1	99.000	0.000	0.000	0.000	0.000	0.000
instrumental	4	96.000	0.000	1.000	0.000	0.000	0.250
chorus	3	97.000	0.000	0.667	0.000	0.000	0.667
silence	1	99.000	0.000	1.000	0.000	0.000	1.000
sax	2	98.000	0.000	0.000	0.000	0.000	0.000
nobeat	1	99.000	0.000	1.000	0.000	0.000	0.000
nopiano	3	97.000	0.000	1.000	0.000	0.333	0.000
novocals	15.000	85.000	0.000	0.800	0.000	0.200	0.067
low	5	95.000	0.000	0.600	0.000	0.000	0.000
weird	3	97.000	0.000	0.667	0.000	0.000	0.000
dance	4	96.000	0.000	1.000	0.250	0.500	0.750
harp	3	97.000	0.000	0.000	0.000	0.000	0.333
horns	3	97.000	0.000	0.000	0.000	0.000	0.000
funky	1	99.000	0.000	1.000	0.000	0.000	0.000
hardrock	4	96.000	0.000	1.000	0.000	0.000	1.000
bells	2	98.000	0.000	0.500	0.500	0.000	0.000
punk	4	96.000	0.000	1.000	0.750	0.000	1.000
techno	12.000	88.000	0.000	0.750	0.333	0.667	0.500
modern	5	95.000	0.000	0.600	0.200	0.200	0.400
violins	25.000	75.000	0.000	0.800	0.160	0.240	0.560
opera	6	94.000	0.000	0.667	0.167	1.000	0.667
cello	21.000	79.000	0.000	0.810	0.238	0.333	0.333
sitar	1	99.000	0.000	0.000	0.000	0.000	1.000
man	4	96.000	0.000	0.500	0.250	0.500	0.000
femalevocals	9	91.000	0.000	0.667	0.000	0.333	0.222
beat	13.000	87.000	0.000	0.615	0.000	0.462	0.462
vocal	22.000	78.000	0.000	0.773	0.000	0.636	0.227
jazz	3	97.000	0.000	0.667	0.000	0.000	0.667
male	10.000	90.000	0.000	0.700	0.100	0.600	0.000
drums	14.000	86.000	0.000	0.786	0.143	0.571	0.429
electronic	16.000	84.000	0.000	0.625	0.188	0.625	0.500
violin	44.000	56.000	0.000	0.886	0.727	0.864	0.568
bass	5	95.000	0.000	0.600	0.000	0.200	0.000
string	12.000	88.000	0.000	0.750	0.000	0.000	0.167
womansinging	3	97.000	0.000	1.000	0.000	0.000	0.000
guitar	15.000	85.000	0.000	0.533	0.333	0.800	0.267
medieval	5	95.000	0.000	0.800	0.000	0.000	0.000
old	1	99.000	0.000	1.000	0.000	0.000	1.000
middleeastern	1	99.000	0.000	0.000	0.000	0.000	0.000
baroque	7	93.000	0.000	0.571	0.000	0.000	0.714
oriental	1	99.000	0.000	0.000	0.000	0.000	0.000
trumpet	2	98.000	0.000	0.000	0.000	0.000	0.000
irish	1	99.000	0.000	1.000	0.000	0.000	1.000
ambient	14.000	86.000	0.000	0.714	0.000	0.786	0.500
funk	2	98.000	0.000	1.000	0.000	0.000	0.500
metal	5	95.000	0.400	1.000	0.200	0.000	1.000
woman	14.000	86.000	0.000	0.786	0.000	0.500	0.286
dark	1	99.000	0.000	0.000	0.000	0.000	0.000
acoustic	3	97.000	0.000	1.000	0.000	0.333	0.333
light	1	99.000	0.000	1.000	0.000	0.000	1.000
trance	4	96.000	0.000	0.750	0.000	0.000	0.750
celtic	1	99.000	0.000	1.000	0.000	0.000	0.000
electric	1	99.000	0.000	1.000	0.000	0.000	0.000
malevocals	6	94.000	0.000	0.333	0.167	0.333	0.500
heavy	4	96.000	0.000	0.500	0.250	0.000	0.750
jazzy	3	97.000	0.000	0.667	1.000	0.000	1.000
country	2	98.000	0.000	1.000	0.000	0.500	0.000
beats	4	96.000	0.000	0.250	0.000	0.000	0.000
loud	11.000	89.000	0.000	0.818	0.091	0.455	0.727
classical	48.000	52.000	0.000	0.812	0.021	1.000	0.667
voices	2	98.000	0.000	1.000	0.000	0.000	0.500
choral	7	93.000	0.000	0.429	0.000	0.000	0.286
harpsichord	12.000	88.000	0.000	0.750	0.417	0.833	0.917
eastern	1	99.000	0.000	1.000	0.000	0.000	1.000
foreign	2	98.000	0.000	0.500	0.000	0.000	0.000
fast	17.000	83.000	0.000	0.412	0.000	0.529	0.294
english	1	99.000	0.000	0.000	0.000	0.000	0.000
spacey	2	98.000	0.000	1.000	0.000	0.500	0.500
electro	4	96.000	0.000	0.750	0.000	0.000	0.500
calm	6	94.000	0.000	0.333	0.000	0.000	0.000
voice	10.000	90.000	0.000	0.900	0.000	0.300	0.100
vocals	16.000	84.000	0.000	0.688	0.188	0.375	0.125
singer	1	99.000	0.000	1.000	1.000	0.000	0.000
strings	47.000	53.000	0.000	0.915	0.085	0.872	0.681
orchestra	5	95.000	0.000	0.400	0.000	0.000	0.000
chant	4	96.000	0.000	0.500	0.000	0.250	0.500
heavymetal	2	98.000	0.000	1.000	0.000	0.000	1.000
girl	2	98.000	0.000	0.500	0.000	0.000	0.500
flute	8	92.000	0.000	0.500	0.375	0.375	0.125
drum	3	97.000	0.000	0.667	0.000	0.333	0.000
classic	24.000	76.000	0.000	0.958	0.083	0.292	0.500
nosinging	3	97.000	0.000	1.000	0.000	0.000	0.000
chanting	2	98.000	0.000	0.500	0.000	0.000	0.000
folk	2	98.000	0.000	0.500	0.000	0.000	0.000
malesinger	1	99.000	0.000	1.000	1.000	1.000	0.000
mellow	5	95.000	0.000	0.600	0.000	0.000	0.000
indian	3	97.000	0.000	0.333	0.000	0.000	0.333
electronica	2	98.000	0.000	1.000	0.000	0.000	0.500
women	3	97.000	0.000	0.667	0.000	0.333	0.333
soft	21.000	79.000	0.000	0.429	0.000	0.286	0.190
malevoice	9	91.000	0.000	0.778	0.000	0.444	0.111
organ	1	99.000	0.000	0.000	0.000	0.000	0.000
female	19.000	81.000	0.000	0.842	0.421	0.579	0.316
classicalguitar	1	99.000	0.000	1.000	0.000	0.000	1.000
airy	4	96.000	0.000	1.000	0.000	0.250	1.000
malevocal	11.000	89.000	0.000	0.818	0.273	0.545	0.273
clapping	2	98.000	0.000	0.000	0.000	0.000	0.000
choir	7	93.000	0.000	0.571	0.143	0.286	0.429

download these results as csv

Negative Example Accuracy

Full dataset

Tag	Positive Examples	Negative Examples	LabX	Mandel	Manzagol	Marsyas	Zhi
nostrings	13.000	6486.000	1.000	0.496	1.000	0.994	0.988
chimes	22.000	6477.000	0.989	0.779	0.998	0.995	0.984
sad	18.000	6481.000	0.896	0.705	0.998	0.992	0.863
nodrums	48.000	6451.000	1.000	0.569	0.998	0.984	0.828
femalevoice	105.000	6394.000	1.000	0.852	0.976	0.955	0.960
horn	7	6492.000	0.997	0.802	0.998	0.992	0.979
pop	196.000	6303.000	1.000	0.746	0.890	0.936	0.848
rock	601.000	5898.000	1.000	0.881	0.972	0.859	0.861
house	22.000	6477.000	1.000	0.793	0.992	0.995	0.941
birds	7	6492.000	1.000	0.922	0.998	0.992	0.924
harpsicord	59.000	6440.000	1.000	0.930	0.986	0.968	0.865
strange	22.000	6477.000	1.000	0.587	0.999	0.989	0.963
noflute	35.000	6464.000	0.993	0.560	0.999	0.987	0.943
novocal	263.000	6236.000	1.000	0.545	0.992	0.899	0.842
solo	217.000	6282.000	0.999	0.837	0.992	0.908	0.874
notenglish	11.000	6488.000	1.000	0.839	0.999	0.992	0.973
novoice	146.000	6353.000	1.000	0.568	0.994	0.954	0.882
newage	157.000	6342.000	1.000	0.758	0.998	0.952	0.787
synth	294.000	6205.000	1.000	0.692	0.991	0.896	0.800
upbeat	52.000	6447.000	1.000	0.692	0.987	0.987	0.887
slow	1043.000	5456.000	1.000	0.709	0.932	0.713	0.786
deep	12.000	6487.000	1.000	0.831	0.997	0.993	0.955
fiddle	14.000	6485.000	1.000	0.817	0.993	0.991	0.952
orchestral	12.000	6487.000	1.000	0.779	0.997	0.990	0.908
notclassical	14.000	6485.000	1.000	0.661	0.998	0.994	0.947
mansinging	46.000	6453.000	1.000	0.748	0.978	0.988	0.960
wind	22.000	6477.000	0.997	0.835	0.990	0.987	0.878
piano	630.000	5869.000	1.000	0.891	0.919	0.753	0.914
spanish	65.000	6434.000	1.000	0.829	0.983	0.985	0.962
femalesinger	30.000	6469.000	1.000	0.870	0.997	0.988	0.977
singing	242.000	6257.000	1.000	0.803	0.974	0.903	0.956
quiet	263.000	6236.000	1.000	0.795	0.996	0.898	0.821
oboe	12.000	6487.000	1.000	0.936	0.979	0.979	0.919
tribal	40.000	6459.000	1.000	0.772	0.998	0.980	0.977
noguitar	46.000	6453.000	1.000	0.560	0.999	0.979	0.920
femalevocal	126.000	6373.000	1.000	0.871	0.952	0.943	0.961
fastbeat	33.000	6466.000	1.000	0.785	0.996	0.994	0.916
hiphop	32.000	6467.000	1.000	0.907	0.996	0.996	0.981
instrumental	102.000	6397.000	0.992	0.598	0.980	0.964	0.827
chorus	50.000	6449.000	1.000	0.941	0.987	0.978	0.967
silence	12.000	6487.000	0.985	0.901	0.994	0.993	0.961
duet	18.000	6481.000	1.000	0.892	0.989	0.991	0.940
sax	20.000	6479.000	0.977	0.898	0.990	0.991	0.963
nobeat	14.000	6485.000	0.996	0.677	0.999	0.992	0.944
nopiano	90.000	6409.000	0.960	0.572	0.997	0.969	0.905
novocals	326.000	6173.000	1.000	0.543	0.998	0.891	0.846
pianosolo	13.000	6486.000	1.000	0.855	0.987	0.993	0.932
low	35.000	6464.000	1.000	0.817	0.990	0.986	0.875
weird	120.000	6379.000	1.000	0.638	0.993	0.959	0.926
dance	184.000	6315.000	1.000	0.822	0.960	0.960	0.893
harp	137.000	6362.000	1.000	0.895	0.993	0.947	0.893
horns	12.000	6487.000	0.991	0.937	0.993	0.990	0.977
funky	66.000	6433.000	1.000	0.773	0.981	0.989	0.921
hardrock	80.000	6419.000	1.000	0.893	0.992	0.980	0.886
bells	36.000	6463.000	1.000	0.732	0.995	0.986	0.977
punk	42.000	6457.000	0.971	0.928	0.962	0.993	0.924
electricguitar	51.000	6448.000	1.000	0.822	0.995	0.985	0.870
techno	827.000	5672.000	1.000	0.819	0.973	0.846	0.890
modern	73.000	6426.000	1.000	0.591	0.985	0.975	0.903
violins	258.000	6241.000	0.978	0.818	0.965	0.949	0.858
noviolin	18.000	6481.000	1.000	0.519	0.999	0.992	0.968
opera	325.000	6174.000	1.000	0.955	0.971	0.844	0.965
india	22.000	6477.000	1.000	0.770	0.999	0.994	0.992
cello	145.000	6354.000	1.000	0.952	0.962	0.922	0.968
sitar	250.000	6249.000	1.000	0.891	0.981	0.940	0.898
hard	25.000	6474.000	0.995	0.884	0.998	0.994	0.894
banjo	15.000	6484.000	1.000	0.961	0.996	0.979	0.943
blues	42.000	6457.000	1.000	0.923	0.992	0.987	0.995
man	128.000	6371.000	1.000	0.791	0.983	0.962	0.972
water	12.000	6487.000	1.000	0.911	0.996	0.992	0.897
femalevocals	90.000	6409.000	1.000	0.870	0.974	0.966	0.975
beat	534.000	5965.000	1.000	0.744	0.973	0.908	0.881
vocal	346.000	6153.000	1.000	0.785	0.978	0.861	0.954
jazz	88.000	6411.000	1.000	0.812	0.951	0.967	0.911
male	316.000	6183.000	1.000	0.821	0.974	0.914	0.952
maleopera	18.000	6481.000	1.000	0.966	0.993	0.991	0.967
drums	663.000	5836.000	1.000	0.700	0.937	0.816	0.853
electronic	578.000	5921.000	1.000	0.726	0.980	0.828	0.834
talking	27.000	6472.000	1.000	0.835	0.999	0.992	0.989
violin	908.000	5591.000	1.000	0.875	0.893	0.793	0.889
bass	73.000	6426.000	1.000	0.697	0.987	0.973	0.907
notrock	19.000	6480.000	1.000	0.494	0.999	0.994	0.985
string	91.000	6408.000	1.000	0.718	0.985	0.971	0.854
womansinging	32.000	6467.000	1.000	0.853	0.995	0.985	0.986
guitar	1166.000	5333.000	1.000	0.847	0.958	0.649	0.913
medieval	39.000	6460.000	1.000	0.791	0.994	0.981	0.877
clarinet	16.000	6483.000	1.000	0.894	0.995	0.991	0.952
world	14.000	6485.000	1.000	0.605	0.998	0.994	0.985
old	14.000	6485.000	1.000	0.713	0.997	0.995	0.821
middleeastern	17.000	6482.000	0.971	0.680	0.991	0.989	0.956
baroque	81.000	6418.000	0.863	0.842	0.992	0.962	0.795
oriental	50.000	6449.000	1.000	0.726	0.980	0.987	0.941
trumpet	17.000	6482.000	0.961	0.848	0.999	0.993	0.995
irish	49.000	6450.000	1.000	0.859	0.991	0.982	0.917
ambient	419.000	6080.000	1.000	0.872	0.999	0.867	0.825
funk	32.000	6467.000	1.000	0.870	0.991	0.995	0.947
metal	159.000	6340.000	0.862	0.912	0.994	0.963	0.885
woman	186.000	6313.000	1.000	0.892	0.981	0.925	0.962
dark	36.000	6463.000	1.000	0.798	0.997	0.984	0.859
acoustic	66.000	6433.000	0.984	0.844	0.974	0.988	0.947
light	16.000	6483.000	1.000	0.607	0.998	0.991	0.956
repetitive	24.000	6475.000	0.952	0.756	1.000	0.995	0.993
trance	51.000	6448.000	1.000	0.757	0.993	0.982	0.884
celtic	27.000	6472.000	1.000	0.751	0.996	0.991	0.967
electric	44.000	6455.000	1.000	0.610	0.978	0.982	0.937
malevocals	123.000	6376.000	0.945	0.775	0.973	0.965	0.881
heavy	59.000	6440.000	0.936	0.873	0.991	0.984	0.875
jazzy	68.000	6431.000	1.000	0.805	0.945	0.976	0.924
country	122.000	6377.000	1.000	0.883	0.936	0.956	0.966
beats	157.000	6342.000	0.991	0.718	0.972	0.962	0.885
loud	313.000	6186.000	1.000	0.828	0.990	0.937	0.846
classical	1544.000	4955.000	1.000	0.846	0.957	0.437	0.810
voices	39.000	6460.000	1.000	0.843	0.996	0.982	0.969
flutes	54.000	6445.000	1.000	0.905	0.986	0.984	0.959
choral	104.000	6395.000	1.000	0.954	0.996	0.967	0.974
harpsichord	263.000	6236.000	1.000	0.912	0.917	0.897	0.826
eastern	80.000	6419.000	1.000	0.735	0.989	0.967	0.927
foreign	51.000	6448.000	1.000	0.825	0.981	0.975	0.986
fast	616.000	5883.000	1.000	0.725	0.987	0.820	0.838
english	11.000	6488.000	1.000	0.768	0.993	0.992	0.979
spacey	27.000	6472.000	1.000	0.857	0.998	0.987	0.894
electro	87.000	6412.000	0.998	0.684	0.993	0.968	0.879
calm	33.000	6466.000	0.989	0.653	0.994	0.989	0.925
lute	15.000	6484.000	1.000	0.922	0.992	0.987	0.937
arabic	10.000	6489.000	1.000	0.733	0.985	0.990	0.973
voice	111.000	6388.000	1.000	0.707	0.980	0.951	0.951
vocals	256.000	6243.000	1.000	0.745	0.939	0.907	0.948
rap	41.000	6458.000	1.000	0.935	0.981	0.992	0.989
singer	25.000	6474.000	1.000	0.736	0.983	0.989	0.989
strings	997.000	5502.000	1.000	0.790	0.978	0.744	0.794
orchestra	98.000	6401.000	1.000	0.843	0.988	0.955	0.865
guitars	25.000	6474.000	1.000	0.731	0.996	0.994	0.921
chant	51.000	6448.000	1.000	0.952	0.996	0.984	0.974
heavymetal	43.000	6456.000	1.000	0.905	0.988	0.993	0.919
girl	10.000	6489.000	0.978	0.755	0.999	0.994	0.972
percussion	26.000	6473.000	0.865	0.729	0.997	0.985	0.964
flute	455.000	6044.000	1.000	0.936	0.973	0.901	0.923
drum	89.000	6410.000	1.000	0.688	0.989	0.971	0.940
classic	235.000	6264.000	1.000	0.756	0.969	0.908	0.815
nosinging	51.000	6448.000	1.000	0.529	0.995	0.984	0.933
chanting	32.000	6467.000	1.000	0.948	0.999	0.988	0.972
folk	48.000	6451.000	1.000	0.805	0.991	0.980	0.974
malesinger	39.000	6460.000	1.000	0.748	0.931	0.985	0.946
mellow	29.000	6470.000	0.996	0.650	0.997	0.992	0.962
indian	313.000	6186.000	1.000	0.815	0.983	0.892	0.960
electronica	39.000	6460.000	1.000	0.627	0.993	0.984	0.925
women	22.000	6477.000	1.000	0.887	0.998	0.993	0.983
notopera	19.000	6480.000	1.000	0.559	1.000	0.994	0.992
noise	16.000	6483.000	1.000	0.793	0.998	0.984	0.902
soft	248.000	6251.000	1.000	0.758	0.983	0.908	0.837
femaleopera	27.000	6472.000	1.000	0.936	0.988	0.984	0.949
malevoice	155.000	6344.000	1.000	0.765	0.970	0.956	0.950
organ	17.000	6482.000	1.000	0.787	0.954	0.975	0.868
female	320.000	6179.000	1.000	0.909	0.925	0.864	0.961
classicalguitar	38.000	6461.000	1.000	0.959	0.978	0.980	0.932
operatic	17.000	6482.000	1.000	0.866	0.999	0.992	0.943
airy	12.000	6487.000	0.990	0.762	0.998	0.994	0.881
malevocal	271.000	6228.000	1.000	0.808	0.935	0.931	0.911
clapping	12.000	6487.000	1.000	0.816	0.999	0.994	0.998
choir	161.000	6338.000	1.000	0.960	0.998	0.956	0.979

download these results as csv

100 query subset used in Tagatune evaluation

Tag	Positive Examples	Negative Examples	LabX	Mandel	Manzagol	Marsyas	Zhi
sad	5	95.000	0.895	0.589	1.000	1.000	0.789
nodrums	4	96.000	1.000	0.458	1.000	0.990	0.781
femalevoice	6	94.000	1.000	0.840	0.957	0.947	0.936
pop	10.000	90.000	1.000	0.778	0.922	0.967	0.878
rock	14.000	86.000	1.000	0.930	0.988	0.872	0.919
birds	1	99.000	1.000	0.919	1.000	1.000	0.838
harpsicord	3	97.000	1.000	0.907	0.979	1.000	0.753
strange	2	98.000	1.000	0.561	1.000	0.980	0.949
novocal	12.000	88.000	1.000	0.557	1.000	0.920	0.875
solo	11.000	89.000	1.000	0.843	0.989	0.955	0.933
notenglish	1	99.000	1.000	0.828	1.000	1.000	0.939
novoice	4	96.000	1.000	0.635	1.000	0.979	0.969
newage	12.000	88.000	1.000	0.795	1.000	1.000	0.795
synth	11.000	89.000	1.000	0.831	1.000	0.989	0.899
upbeat	4	96.000	1.000	0.781	0.990	1.000	0.927
slow	44.000	56.000	1.000	0.786	0.946	0.768	0.893
deep	2	98.000	1.000	0.806	1.000	0.990	0.929
fiddle	1	99.000	1.000	0.727	1.000	0.990	0.929
orchestral	2	98.000	1.000	0.704	1.000	1.000	0.704
mansinging	2	98.000	1.000	0.796	1.000	0.959	0.969
wind	1	99.000	1.000	0.808	0.990	0.990	0.828
piano	9	91.000	1.000	0.835	0.890	0.637	0.956
femalesinger	4	96.000	1.000	0.812	0.979	0.979	0.979
singing	13.000	87.000	1.000	0.897	0.977	0.931	0.977
quiet	16.000	84.000	1.000	0.798	1.000	0.893	0.881
tribal	1	99.000	1.000	0.899	1.000	0.990	1.000
noguitar	2	98.000	1.000	0.480	1.000	0.990	0.908
femalevocal	13.000	87.000	1.000	0.874	0.931	0.966	0.977
fastbeat	1	99.000	1.000	0.859	1.000	0.990	0.929
instrumental	4	96.000	1.000	0.448	0.990	0.969	0.885
chorus	3	97.000	1.000	0.959	1.000	0.969	0.969
silence	1	99.000	0.970	0.919	1.000	0.990	0.960
sax	2	98.000	0.990	0.908	0.990	1.000	0.949
nobeat	1	99.000	1.000	0.566	1.000	0.990	0.889
nopiano	3	97.000	0.938	0.412	1.000	0.979	0.866
novocals	15.000	85.000	1.000	0.518	1.000	0.894	0.918
low	5	95.000	1.000	0.779	1.000	0.989	0.874
weird	3	97.000	1.000	0.711	1.000	0.948	0.959
dance	4	96.000	1.000	0.948	0.969	0.969	0.948
harp	3	97.000	1.000	0.928	1.000	0.979	0.969
horns	3	97.000	1.000	0.876	1.000	0.979	0.979
funky	1	99.000	1.000	0.848	1.000	0.980	0.980
hardrock	4	96.000	1.000	0.927	1.000	0.990	0.885
bells	2	98.000	1.000	0.776	1.000	0.990	1.000
punk	4	96.000	0.938	0.948	1.000	1.000	0.917
techno	12.000	88.000	1.000	0.909	1.000	0.909	0.920
modern	5	95.000	1.000	0.684	1.000	0.979	0.958
violins	25.000	75.000	0.973	0.787	0.973	0.960	0.827
opera	6	94.000	1.000	0.957	0.957	0.851	0.957
cello	21.000	79.000	1.000	0.886	0.962	0.886	0.975
sitar	1	99.000	1.000	0.919	0.960	0.939	0.980
man	4	96.000	1.000	0.812	1.000	0.958	0.990
femalevocals	9	91.000	1.000	0.879	0.989	0.945	0.967
beat	13.000	87.000	1.000	0.839	1.000	0.966	0.943
vocal	22.000	78.000	1.000	0.885	0.974	0.962	0.974
jazz	3	97.000	1.000	0.928	0.990	0.959	0.928
male	10.000	90.000	1.000	0.833	0.956	0.933	0.978
drums	14.000	86.000	1.000	0.837	0.977	0.907	0.942
electronic	16.000	84.000	1.000	0.893	1.000	0.964	0.929
violin	44.000	56.000	1.000	0.911	0.929	0.839	0.929
bass	5	95.000	1.000	0.716	1.000	0.979	0.979
string	12.000	88.000	1.000	0.625	1.000	0.966	0.875
womansinging	3	97.000	1.000	0.804	0.969	0.979	0.990
guitar	15.000	85.000	1.000	0.894	1.000	0.600	0.976
medieval	5	95.000	1.000	0.779	1.000	1.000	0.874
old	1	99.000	1.000	0.616	1.000	1.000	0.687
middleeastern	1	99.000	0.960	0.596	1.000	1.000	0.960
baroque	7	93.000	0.892	0.806	1.000	0.968	0.677
oriental	1	99.000	1.000	0.687	0.980	1.000	1.000
trumpet	2	98.000	0.959	0.837	1.000	0.990	1.000
irish	1	99.000	1.000	0.798	0.990	0.990	0.929
ambient	14.000	86.000	1.000	0.907	1.000	0.907	0.756
funk	2	98.000	1.000	0.949	1.000	1.000	0.990
metal	5	95.000	0.811	0.958	1.000	0.947	0.895
woman	14.000	86.000	1.000	0.930	1.000	0.965	0.965
dark	1	99.000	1.000	0.768	1.000	1.000	0.798
acoustic	3	97.000	0.990	0.897	0.990	1.000	1.000
light	1	99.000	1.000	0.515	1.000	0.990	0.980
trance	4	96.000	1.000	0.875	1.000	0.990	0.938
celtic	1	99.000	1.000	0.667	1.000	0.990	0.929
electric	1	99.000	1.000	0.747	0.970	0.990	0.949
malevocals	6	94.000	0.947	0.809	0.989	0.926	0.883
heavy	4	96.000	0.969	0.896	1.000	0.979	0.896
jazzy	3	97.000	1.000	0.897	0.979	0.959	0.928
country	2	98.000	1.000	0.878	0.959	0.939	0.980
beats	4	96.000	0.990	0.812	0.969	1.000	0.917
loud	11.000	89.000	1.000	0.854	0.989	0.989	0.899
classical	48.000	52.000	1.000	0.885	0.942	0.596	0.942
voices	2	98.000	1.000	0.878	1.000	0.990	0.980
choral	7	93.000	1.000	0.978	0.989	0.968	0.968
harpsichord	12.000	88.000	1.000	0.909	0.932	0.943	0.716
eastern	1	99.000	1.000	0.798	1.000	0.980	0.990
foreign	2	98.000	1.000	0.847	0.980	0.980	0.980
fast	17.000	83.000	1.000	0.855	0.988	0.916	0.928
english	1	99.000	1.000	0.788	0.990	0.980	0.980
spacey	2	98.000	1.000	0.847	1.000	1.000	0.816
electro	4	96.000	1.000	0.802	0.990	1.000	0.927
calm	6	94.000	0.989	0.596	0.979	1.000	0.979
voice	10.000	90.000	1.000	0.844	0.978	0.944	0.967
vocals	16.000	84.000	1.000	0.810	0.964	0.893	0.976
singer	1	99.000	1.000	0.727	0.980	0.970	0.970
strings	47.000	53.000	1.000	0.906	0.962	0.887	0.943
orchestra	5	95.000	1.000	0.768	0.989	0.958	0.726
chant	4	96.000	1.000	0.969	1.000	1.000	1.000
heavymetal	2	98.000	1.000	0.929	0.990	1.000	0.908
girl	2	98.000	0.980	0.786	1.000	0.980	0.969
flute	8	92.000	1.000	0.935	0.989	0.913	0.946
drum	3	97.000	1.000	0.794	1.000	1.000	0.979
classic	24.000	76.000	1.000	0.737	0.987	0.908	0.789
nosinging	3	97.000	1.000	0.402	1.000	1.000	0.948
chanting	2	98.000	1.000	0.939	1.000	0.990	0.980
folk	2	98.000	1.000	0.837	1.000	0.969	0.990
malesinger	1	99.000	1.000	0.758	0.980	0.949	0.929
mellow	5	95.000	0.989	0.726	1.000	0.989	0.968
indian	3	97.000	1.000	0.907	0.990	0.876	0.990
electronica	2	98.000	1.000	0.714	1.000	0.980	0.929
women	3	97.000	1.000	0.876	1.000	1.000	0.979
soft	21.000	79.000	1.000	0.734	0.975	0.848	0.899
malevoice	9	91.000	1.000	0.835	0.978	0.956	0.956
organ	1	99.000	1.000	0.737	0.929	1.000	0.838
female	19.000	81.000	1.000	0.963	0.951	0.975	0.975
classicalguitar	1	99.000	1.000	0.980	1.000	0.990	0.980
airy	4	96.000	0.990	0.760	1.000	1.000	0.865
malevocal	11.000	89.000	1.000	0.854	0.955	0.955	0.921
clapping	2	98.000	1.000	0.888	1.000	0.990	1.000
choir	7	93.000	1.000	0.978	1.000	0.957	0.978

download these results as csv

Overall Summary Results (MIREX Statistical evaluation - Affinity)

Full dataset

Measure	Mandel	Manzagol	Marsyas	Zhi
Average AUC-ROC Tag	0.821	0.750	0.831	0.673
Average AUC-ROC Clip	0.886	0.810	0.933	0.748
Precision at 3	0.323	0.255	0.440	0.224
Precision at 6	0.245	0.194	0.314	0.192
Precision at 9	0.197	0.159	0.244	0.168
Precision at 12	0.167	0.136	0.201	0.146
Precision at 15	0.145	0.119	0.172	0.127

download these results as csv

100 query subset used in Tagatune evaluation

Measure	Mandel	Manzagol	Marsyas	Zhi
Average AUC-ROC Tag	0.646	0.566	0.636	0.499
Average AUC-ROC Clip	0.873	0.766	0.916	0.689
Precision at 3	0.613	0.477	0.743	0.363
Precision at 6	0.508	0.383	0.590	0.338
Precision at 9	0.434	0.322	0.489	0.310
Precision at 12	0.388	0.283	0.431	0.278
Precision at 15	0.339	0.252	0.383	0.248

download these results as csv

AUC-ROC Tag

Full dataset

Tag	Mandel	Manzagol	Marsyas	Zhi
nostrings	0.552	0.599	0.663	0.494
chimes	0.732	0.749	0.822	0.561
sad	0.842	0.662	0.816	0.785
nodrums	0.600	0.549	0.583	0.517
femalevoice	0.894	0.731	0.786	0.648
horn	0.435	0.683	0.760	0.560
pop	0.872	0.819	0.891	0.687
rock	0.947	0.926	0.957	0.879
house	0.870	0.651	0.949	0.607
birds	0.819	0.653	0.820	0.823
harpsicord	0.946	0.944	0.961	0.855
strange	0.801	0.736	0.744	0.550
noflute	0.454	0.542	0.532	0.516
novocal	0.617	0.538	0.626	0.519
solo	0.843	0.746	0.805	0.686
notenglish	0.861	0.587	0.822	0.717
novoice	0.623	0.538	0.578	0.517
newage	0.856	0.706	0.811	0.697
synth	0.816	0.728	0.778	0.690
upbeat	0.814	0.702	0.841	0.674
slow	0.778	0.733	0.797	0.662
deep	0.873	0.835	0.904	0.602
fiddle	0.895	0.825	0.844	0.585
orchestral	0.782	0.718	0.796	0.841
notclassical	0.605	0.498	0.752	0.578
mansinging	0.858	0.698	0.886	0.600
wind	0.855	0.779	0.866	0.752
piano	0.913	0.861	0.875	0.797
spanish	0.762	0.699	0.759	0.566
femalesinger	0.864	0.668	0.831	0.623
singing	0.866	0.720	0.847	0.611
quiet	0.854	0.777	0.906	0.736
oboe	0.806	0.869	0.863	0.504
tribal	0.732	0.737	0.761	0.588
noguitar	0.611	0.534	0.579	0.615
femalevocal	0.896	0.739	0.860	0.648
fastbeat	0.827	0.855	0.908	0.728
hiphop	0.901	0.856	0.912	0.647
instrumental	0.630	0.561	0.704	0.563
chorus	0.938	0.915	0.932	0.836
silence	0.927	0.936	0.988	0.648
duet	0.707	0.681	0.726	0.553
sax	0.683	0.666	0.663	0.482
nobeat	0.687	0.549	0.651	0.684
nopiano	0.559	0.480	0.542	0.496
novocals	0.613	0.538	0.640	0.526
pianosolo	0.906	0.970	0.959	0.937
low	0.830	0.814	0.887	0.629
weird	0.788	0.573	0.790	0.599
dance	0.910	0.850	0.925	0.813
harp	0.814	0.846	0.824	0.650
horns	0.637	0.659	0.524	0.530
funky	0.904	0.860	0.897	0.703
hardrock	0.976	0.940	0.969	0.948
bells	0.654	0.678	0.799	0.544
punk	0.939	0.958	0.958	0.879
electricguitar	0.834	0.790	0.836	0.734
techno	0.942	0.899	0.940	0.856
modern	0.725	0.642	0.772	0.582
violins	0.890	0.848	0.873	0.753
noviolin	0.521	0.637	0.697	0.512
opera	0.972	0.962	0.957	0.870
india	0.907	0.837	0.836	0.655
cello	0.930	0.882	0.900	0.669
sitar	0.940	0.902	0.896	0.793
hard	0.966	0.910	0.960	0.932
banjo	0.676	0.689	0.799	0.638
blues	0.909	0.894	0.894	0.521
man	0.886	0.739	0.883	0.616
water	0.896	0.634	0.956	0.851
femalevocals	0.891	0.780	0.777	0.593
beat	0.890	0.873	0.915	0.799
vocal	0.855	0.729	0.825	0.595
jazz	0.878	0.782	0.847	0.691
male	0.897	0.781	0.882	0.644
maleopera	0.982	0.923	0.992	0.935
drums	0.836	0.801	0.841	0.686
electronic	0.860	0.797	0.847	0.750
talking	0.875	0.718	0.933	0.513
violin	0.943	0.899	0.910	0.807
bass	0.804	0.779	0.827	0.677
notrock	0.510	0.568	0.603	0.546
string	0.795	0.682	0.768	0.567
womansinging	0.941	0.716	0.900	0.603
guitar	0.870	0.838	0.879	0.721
medieval	0.845	0.771	0.734	0.642
clarinet	0.886	0.753	0.920	0.666
world	0.600	0.565	0.786	0.636
old	0.744	0.644	0.720	0.679
middleeastern	0.668	0.604	0.693	0.536
baroque	0.925	0.847	0.877	0.840
oriental	0.792	0.798	0.826	0.628
trumpet	0.785	0.661	0.688	0.498
irish	0.861	0.792	0.783	0.551
ambient	0.916	0.761	0.873	0.761
funk	0.933	0.888	0.939	0.643
metal	0.971	0.954	0.974	0.962
woman	0.927	0.815	0.872	0.710
dark	0.897	0.794	0.874	0.613
acoustic	0.839	0.834	0.908	0.678
light	0.737	0.654	0.793	0.510
repetitive	0.655	0.497	0.865	0.496
trance	0.861	0.699	0.877	0.702
celtic	0.826	0.758	0.776	0.520
electric	0.709	0.527	0.753	0.572
malevocals	0.869	0.774	0.862	0.635
heavy	0.955	0.911	0.969	0.932
jazzy	0.881	0.801	0.849	0.709
country	0.876	0.771	0.883	0.577
beats	0.867	0.814	0.909	0.805
loud	0.893	0.817	0.927	0.831
classical	0.920	0.849	0.885	0.778
voices	0.803	0.691	0.806	0.651
flutes	0.943	0.906	0.939	0.865
choral	0.966	0.935	0.957	0.900
harpsichord	0.940	0.911	0.940	0.891
eastern	0.845	0.810	0.842	0.667
foreign	0.845	0.644	0.873	0.600
fast	0.802	0.758	0.815	0.642
english	0.664	0.519	0.811	0.536
spacey	0.907	0.759	0.886	0.680
electro	0.821	0.695	0.838	0.675
calm	0.668	0.601	0.820	0.554
lute	0.955	0.936	0.934	0.843
arabic	0.534	0.656	0.682	0.487
voice	0.819	0.620	0.780	0.562
vocals	0.812	0.676	0.791	0.591
rap	0.934	0.925	0.946	0.739
singer	0.811	0.642	0.838	0.494
strings	0.879	0.832	0.857	0.732
orchestra	0.883	0.797	0.867	0.725
guitars	0.659	0.690	0.817	0.664
chant	0.912	0.851	0.952	0.815
heavymetal	0.953	0.931	0.972	0.904
girl	0.919	0.701	0.806	0.685
percussion	0.778	0.850	0.850	0.638
flute	0.953	0.907	0.907	0.789
drum	0.814	0.735	0.821	0.611
classic	0.872	0.797	0.840	0.711
nosinging	0.566	0.526	0.657	0.495
chanting	0.891	0.657	0.878	0.706
folk	0.758	0.797	0.774	0.560
malesinger	0.835	0.742	0.883	0.603
mellow	0.647	0.632	0.787	0.515
indian	0.886	0.794	0.831	0.621
electronica	0.802	0.685	0.823	0.621
women	0.947	0.812	0.878	0.697
notopera	0.510	0.482	0.550	0.522
noise	0.817	0.737	0.759	0.676
soft	0.773	0.718	0.809	0.662
femaleopera	0.977	0.960	0.934	0.926
malevoice	0.870	0.660	0.869	0.602
organ	0.640	0.457	0.597	0.557
female	0.928	0.813	0.851	0.706
classicalguitar	0.963	0.949	0.926	0.909
operatic	0.948	0.862	0.913	0.892
airy	0.885	0.696	0.911	0.843
malevocal	0.895	0.761	0.892	0.645
clapping	0.798	0.575	0.628	0.499
choir	0.979	0.962	0.962	0.880

download these results as csv

100 query subset used in Tagatune evaluation

Tag	Mandel	Manzagol	Marsyas	Zhi
nostrings	0.000	0.000	0.000	0.000
chimes	0.000	0.000	0.000	0.000
sad	0.762	0.760	0.672	0.703
nodrums	0.609	0.320	0.622	0.535
femalevoice	0.885	0.704	0.910	0.556
horn	0.000	0.000	0.000	0.000
pop	0.931	0.892	0.969	0.752
rock	0.966	0.934	0.968	0.872
house	0.000	0.000	0.000	0.000
birds	0.939	0.980	0.606	0.424
harpsicord	0.935	0.887	0.914	0.880
strange	0.791	0.449	0.816	0.480
noflute	0.000	0.000	0.000	0.000
novocal	0.696	0.606	0.634	0.481
solo	0.796	0.612	0.776	0.568
notenglish	0.000	0.313	0.596	0.475
novoice	0.784	0.604	0.703	0.490
newage	0.813	0.665	0.863	0.633
synth	0.921	0.774	0.860	0.645
upbeat	0.888	0.789	0.828	0.724
slow	0.856	0.679	0.848	0.637
deep	0.755	0.959	0.852	0.469
fiddle	0.818	0.768	0.818	0.470
orchestral	0.709	0.796	0.806	0.908
notclassical	0.000	0.000	0.000	0.000
mansinging	0.980	0.816	0.969	0.745
wind	0.939	0.747	0.980	0.970
piano	0.819	0.568	0.683	0.531
spanish	0.000	0.000	0.000	0.000
femalesinger	0.794	0.581	0.839	0.495
singing	0.939	0.688	0.846	0.641
quiet	0.844	0.735	0.924	0.659
oboe	0.000	0.000	0.000	0.000
tribal	0.980	1.000	0.980	0.500
noguitar	0.699	0.480	0.189	0.459
femalevocal	0.836	0.753	0.925	0.648
fastbeat	0.737	0.485	0.838	0.470
hiphop	0.000	0.000	0.000	0.000
instrumental	0.846	0.622	0.643	0.578
chorus	0.900	0.852	0.969	0.820
silence	0.949	0.960	0.929	0.970
duet	0.000	0.000	0.000	0.000
sax	0.122	0.592	0.122	0.480
nobeat	0.778	0.141	0.828	0.449
nopiano	0.773	0.368	0.560	0.438
novocals	0.632	0.589	0.707	0.498
pianosolo	0.000	0.000	0.000	0.000
low	0.737	0.667	0.737	0.442
weird	0.746	0.333	0.773	0.485
dance	0.974	0.906	0.979	0.852
harp	0.663	0.746	0.928	0.653
horns	0.588	0.529	0.282	0.495
funky	1.000	0.980	0.939	0.495
hardrock	1.000	0.943	1.000	0.992
bells	0.536	0.857	0.393	0.500
punk	0.987	0.997	0.992	0.982
electricguitar	0.000	0.000	0.000	0.000
techno	0.922	0.852	0.891	0.713
modern	0.796	0.931	0.813	0.684
violins	0.890	0.864	0.890	0.676
noviolin	0.000	0.000	0.000	0.000
opera	0.846	0.911	0.943	0.821
india	0.000	0.000	0.000	0.000
cello	0.914	0.838	0.866	0.662
sitar	0.859	0.586	0.818	1.000
hard	0.000	0.000	0.000	0.000
banjo	0.000	0.000	0.000	0.000
blues	0.000	0.000	0.000	0.000
man	0.862	0.815	0.880	0.500
water	0.000	0.000	0.000	0.000
femalevocals	0.834	0.803	0.823	0.599
beat	0.877	0.887	0.947	0.660
vocal	0.904	0.722	0.859	0.578
jazz	0.907	0.790	0.883	0.799
male	0.909	0.723	0.926	0.494
maleopera	0.000	0.000	0.000	0.000
drums	0.894	0.868	0.866	0.686
electronic	0.890	0.794	0.846	0.689
talking	0.000	0.000	0.000	0.000
violin	0.958	0.903	0.932	0.740
bass	0.758	0.731	0.792	0.495
notrock	0.000	0.000	0.000	0.000
string	0.809	0.668	0.747	0.526
womansinging	0.911	0.897	0.914	0.500
guitar	0.864	0.800	0.807	0.587
medieval	0.745	0.752	0.501	0.442
clarinet	0.000	0.000	0.000	0.000
world	0.000	0.000	0.000	0.000
old	0.667	0.586	0.475	0.869
middleeastern	0.152	0.101	0.162	0.485
baroque	0.829	0.631	0.747	0.682
oriental	0.545	0.232	0.556	0.500
trumpet	0.643	0.648	0.372	0.500
irish	0.949	0.838	0.697	0.939
ambient	0.898	0.649	0.895	0.678
funk	0.995	0.939	0.974	0.495
metal	0.998	1.000	0.996	0.983
woman	0.857	0.841	0.856	0.588
dark	0.657	0.354	0.596	0.404
acoustic	0.948	0.983	0.997	0.500
light	0.929	0.677	0.697	1.000
repetitive	0.000	0.000	0.000	0.000
trance	0.909	0.867	0.990	0.845
celtic	1.000	0.919	0.909	0.470
electric	0.899	1.000	0.798	0.480
malevocals	0.814	0.652	0.881	0.695
heavy	0.943	0.932	0.943	0.855
jazzy	0.948	0.983	0.904	0.797
country	0.923	0.556	0.918	0.495
beats	0.792	0.698	0.776	0.464
loud	0.944	0.876	0.941	0.833
classical	0.945	0.802	0.898	0.787
voices	0.964	0.842	0.959	0.747
flutes	0.000	0.000	0.000	0.000
choral	0.862	0.647	0.846	0.631
harpsichord	0.926	0.747	0.929	0.897
eastern	0.909	1.000	0.687	1.000
foreign	0.449	0.347	0.796	0.495
fast	0.780	0.670	0.778	0.617
english	0.364	0.798	0.838	0.495
spacey	0.959	0.724	0.990	0.681
electro	0.844	0.651	0.927	0.590
calm	0.441	0.651	0.784	0.495
lute	0.000	0.000	0.000	0.000
arabic	0.000	0.000	0.000	0.000
voice	0.933	0.700	0.802	0.540
vocals	0.846	0.649	0.775	0.519
rap	0.000	0.000	0.000	0.000
singer	0.909	1.000	0.768	0.490
strings	0.972	0.902	0.933	0.814
orchestra	0.813	0.823	0.764	0.368
guitars	0.000	0.000	0.000	0.000
chant	0.815	0.721	0.974	0.625
heavymetal	0.995	0.867	0.990	0.995
girl	0.872	0.699	0.923	0.735
percussion	0.000	0.000	0.000	0.000
flute	0.807	0.802	0.837	0.542
drum	0.869	0.869	0.955	0.495
classic	0.880	0.776	0.815	0.635
nosinging	0.729	0.347	0.677	0.479
chanting	0.719	0.444	0.969	0.495
folk	0.673	0.867	0.536	0.500
malesinger	0.778	0.949	0.990	0.470
mellow	0.629	0.707	0.693	0.489
indian	0.801	0.419	0.481	0.667
electronica	0.852	0.526	0.913	0.735
women	0.928	0.739	0.845	0.660
notopera	0.000	0.000	0.000	0.000
noise	0.000	0.000	0.000	0.000
soft	0.661	0.667	0.761	0.548
femaleopera	0.000	0.000	0.000	0.000
malevoice	0.906	0.618	0.882	0.478
organ	0.707	0.101	0.525	0.424
female	0.910	0.712	0.827	0.620
classicalguitar	1.000	0.970	1.000	0.990
operatic	0.000	0.000	0.000	0.000
airy	0.995	0.693	0.982	0.992
malevocal	0.883	0.700	0.916	0.600
clapping	0.867	0.270	0.949	0.500
choir	0.892	0.688	0.848	0.710

download these results as csv

Select Friedman's Test Results

Tag F-measure (Binary) Friedman Test

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the F-measure for each tag in the test, averaged over all folds.

Full dataset

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
Zhi	Mandel	-0.196	0.275	0.746	FALSE
Zhi	Marsyas	-0.102	0.369	0.840	FALSE
Zhi	Manzagol	0.657	1.128	1.599	TRUE
Zhi	LabX	2.007	2.478	2.949	TRUE
Mandel	Marsyas	-0.377	0.094	0.565	FALSE
Mandel	Manzagol	0.382	0.853	1.324	TRUE
Mandel	LabX	1.732	2.203	2.674	TRUE
Marsyas	Manzagol	0.288	0.759	1.230	TRUE
Marsyas	LabX	1.639	2.109	2.580	TRUE
Manzagol	LabX	0.879	1.350	1.821	TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T/binary_FMeasure.friedman.tukeyKramerHSD.png

100 query subset used in Tagatune evaluation

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
Mandel	Zhi	0.383	0.853	1.323	TRUE
Mandel	Marsyas	0.609	1.079	1.550	TRUE
Mandel	Manzagol	1.284	1.754	2.224	TRUE
Mandel	LabX	1.895	2.365	2.835	TRUE
Zhi	Marsyas	-0.244	0.226	0.696	FALSE
Zhi	Manzagol	0.431	0.901	1.371	TRUE
Zhi	LabX	1.042	1.512	1.982	TRUE
Marsyas	Manzagol	0.204	0.675	1.145	TRUE
Marsyas	LabX	0.816	1.286	1.756	TRUE
Manzagol	LabX	0.141	0.611	1.081	TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T_subset/binary_FMeasure.friedman.tukeyKramerHSD.png

Per Track F-measure (Binary) Friedman Test

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the F-measure for each track in the test, averaged over all folds.

Full dataset

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
Marsyas	Manzagol	1.052	1.124	1.197	TRUE
Marsyas	Zhi	0.829	0.901	0.973	TRUE
Marsyas	Mandel	1.293	1.365	1.438	TRUE
Marsyas	LabX	2.795	2.867	2.939	TRUE
Manzagol	Zhi	-0.296	-0.223	-0.151	TRUE
Manzagol	Mandel	0.168	0.241	0.313	TRUE
Manzagol	LabX	1.670	1.743	1.815	TRUE
Zhi	Mandel	0.392	0.464	0.537	TRUE
Zhi	LabX	1.894	1.966	2.038	TRUE
Mandel	LabX	1.429	1.502	1.574	TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T/binary_FMeasure_per_track.friedman.tukeyKramerHSD.png

100 query subset used in Tagatune evaluation

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
Marsyas	Zhi	0.381	0.985	1.589	TRUE
Marsyas	Mandel	0.656	1.260	1.864	TRUE
Marsyas	Manzagol	1.261	1.865	2.469	TRUE
Marsyas	LabX	2.736	3.340	3.944	TRUE
Zhi	Mandel	-0.329	0.275	0.879	FALSE
Zhi	Manzagol	0.276	0.880	1.484	TRUE
Zhi	LabX	1.751	2.355	2.959	TRUE
Mandel	Manzagol	0.001	0.605	1.209	TRUE
Mandel	LabX	1.476	2.080	2.684	TRUE
Manzagol	LabX	0.871	1.475	2.079	TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T_subset/binary_FMeasure_per_track.friedman.tukeyKramerHSD.png

Tag AUC-ROC (Affinity) Friedman Test

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the Area Under the ROC curve (AUC-ROC) for each tag in the test, averaged over all folds.

Full dataset

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
Marsyas	Mandel	-0.283	0.087	0.458	FALSE
Marsyas	Manzagol	0.935	1.306	1.677	TRUE
Marsyas	Zhi	1.735	2.106	2.477	TRUE
Mandel	Manzagol	0.848	1.219	1.590	TRUE
Mandel	Zhi	1.648	2.019	2.390	TRUE
Manzagol	Zhi	0.429	0.800	1.171	TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T/affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.png

100 query subset used in Tagatune evaluation

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
Mandel	Marsyas	-0.206	0.122	0.450	FALSE
Mandel	Manzagol	0.522	0.850	1.178	TRUE
Mandel	Zhi	1.025	1.353	1.681	TRUE
Marsyas	Manzagol	0.400	0.728	1.056	TRUE
Marsyas	Zhi	0.903	1.231	1.559	TRUE
Manzagol	Zhi	0.175	0.503	0.831	TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T_subset/affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.png

Per Track AUC-ROC (Affinity) Friedman Test

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the Area Under the ROC curve (AUC-ROC) for each track/clip in the test, averaged over all folds.

Full dataset

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
Marsyas	Mandel	0.523	0.580	0.638	TRUE
Marsyas	Manzagol	1.184	1.242	1.299	TRUE
Marsyas	Zhi	1.611	1.668	1.726	TRUE
Mandel	Manzagol	0.604	0.661	0.719	TRUE
Mandel	Zhi	1.030	1.088	1.145	TRUE
Manzagol	Zhi	0.369	0.426	0.484	TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T/affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.png

100 query subset used in Tagatune evaluation

TeamID	TeamID	Lowerbound	Mean	Upperbound	Significance
Marsyas	Mandel	0.071	0.540	1.009	TRUE
Marsyas	Manzagol	1.191	1.660	2.129	TRUE
Marsyas	Zhi	1.731	2.200	2.669	TRUE
Mandel	Manzagol	0.651	1.120	1.589	TRUE
Mandel	Zhi	1.191	1.660	2.129	TRUE
Manzagol	Zhi	0.071	0.540	1.009	TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T_subset/affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.png

Assorted Results Files for Download

MIREX Statistical Evaluation Results

Full dataset

affinity_tag_fold_AUC_ROC.csv
affinity_clip_AUC_ROC.csv
binary_per_fold_Accuracy.csv
binary_per_fold_Fmeasure.csv
binary_per_fold_negative_example_Accuracy.csv
binary_per_fold_per_track_Accuracy.csv
binary_per_fold_per_track_Fmeasure.csv
binary_per_fold_per_track_negative_example_Accuracy.csv
binary_per_fold_per_track_positive_example_Accuracy.csv
binary_per_fold_positive_example_Accuracy.csv
affinity_clip_Precision_at_3.csv
affinity_clip_Precision_at_6.csv
affinity_clip_Precision_at_9.csv
affinity_clip_Precision_at_12.csv
affinity_clip_Precision_at_15.csv

100 query subset used in Tagatune evaluation

affinity_tag_fold_AUC_ROC.csv
affinity_clip_AUC_ROC.csv
binary_per_fold_Accuracy.csv
binary_per_fold_Fmeasure.csv
binary_per_fold_negative_example_Accuracy.csv
binary_per_fold_per_track_Accuracy.csv
binary_per_fold_per_track_Fmeasure.csv
binary_per_fold_per_track_negative_example_Accuracy.csv
binary_per_fold_per_track_positive_example_Accuracy.csv
binary_per_fold_positive_example_Accuracy.csv
affinity_clip_Precision_at_3.csv
affinity_clip_Precision_at_6.csv
affinity_clip_Precision_at_9.csv
affinity_clip_Precision_at_12.csv
affinity_clip_Precision_at_15.csv

Friedman's Tests Results

Full dataset

affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.csv
affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.png
affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.csv
affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.png
affinity.PrecisionAt3.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt3.friedman.tukeyKramerHSD.png
affinity.PrecisionAt6.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt6.friedman.tukeyKramerHSD.png
affinity.PrecisionAt9.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt9.friedman.tukeyKramerHSD.png
affinity.PrecisionAt12.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt12.friedman.tukeyKramerHSD.png
affinity.PrecisionAt15.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt15.friedman.tukeyKramerHSD.png
binary_Accuracy.friedman.tukeyKramerHSD.csv
binary_Accuracy.friedman.tukeyKramerHSD.png
binary_FMeasure.friedman.tukeyKramerHSD.csv
binary_FMeasure.friedman.tukeyKramerHSD.png
binary_FMeasure_per_track.friedman.tukeyKramerHSD.csv
binary_FMeasure_per_track.friedman.tukeyKramerHSD.png

100 query subset used in Tagatune evaluation

affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.csv
affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.png
affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.csv
affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.png
affinity.PrecisionAt3.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt3.friedman.tukeyKramerHSD.png
affinity.PrecisionAt6.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt6.friedman.tukeyKramerHSD.png
affinity.PrecisionAt9.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt9.friedman.tukeyKramerHSD.png
affinity.PrecisionAt12.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt12.friedman.tukeyKramerHSD.png
affinity.PrecisionAt15.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt15.friedman.tukeyKramerHSD.png
binary_Accuracy.friedman.tukeyKramerHSD.csv
binary_Accuracy.friedman.tukeyKramerHSD.png
binary_FMeasure.friedman.tukeyKramerHSD.csv
binary_FMeasure.friedman.tukeyKramerHSD.png
binary_FMeasure_per_track.friedman.tukeyKramerHSD.csv
binary_FMeasure_per_track.friedman.tukeyKramerHSD.png

Results By Algorithm

(.tgz format)

Full dataset

LabX = Anonymous
Mandel = Michael Mandel
Manzagol = Pierre-Antoine Manzagol
Marsyas = George Tzanetakis
Zhi = Zhi-Sheng Chen

100 query subset used in Tagatune evaluation

LabX = Anonymous
Mandel = Michael Mandel
Manzagol = Pierre-Antoine Manzagol
Marsyas = George Tzanetakis
Zhi = Zhi-Sheng Chen