|
|
| Line 1: |
Line 1: |
| − | ==Proposer==
| |
| − |
| |
| − | Paul Brossier (Queen Mary) paul.brossier@elec.qmul.ac.uk
| |
| − |
| |
| − | Pierre Leveau (Laboratoire d'Acoustique Musicale, GET-ENST (T├⌐l├⌐com Paris)) leveau at lam dot jussieu dot fr
| |
| − |
| |
| − | ==Title==
| |
| − |
| |
| − | Onset Detection Contest
| |
| − |
| |
| − |
| |
| − | ==Description==
| |
| − |
| |
| − | The aim of this contest is to compare state-of-the-art onset detection algorithms on music recordings. The methods will be evaluated on a large, various and reliably-annotated dataset, composed of sub-datasets grouping files of the same type.
| |
| − |
| |
| − | 1) '''Input data'''
| |
| − |
| |
| − | ''Audio format'':
| |
| − |
| |
| − | The data are monophonic sound files, with the associated onset times and
| |
| − | data about the annotation robustness.
| |
| − | * CD-quality (PCM, 16-bit, 44100 Hz)
| |
| − | * single channel (mono)
| |
| − | * file length between 2 and 36 seconds (total time: 14 minutes)
| |
| − | * File names:
| |
| − |
| |
| − | ''Audio content'':
| |
| − |
| |
| − | The dataset is subdivided into classes, because onset detection is sometimes performed in applications dedicated to a single type of signal (ex: segmentation of a single track in a mix, drum transcription, complex mixes databases segmentation...). The performance of each algorithm will be assessed on the whole dataset but also on each class separately.
| |
| − |
| |
| − | The dataset contains 85 files from 5 classes annotated as follows:
| |
| − | * 30 solo drum excerpts cross-annotated by 3 people
| |
| − | * 30 solo monophonic pitched instruments excerpts cross-annotated by 3 people
| |
| − | * 10 solo polyphonic pitched instruments excerpts cross-annotated by 3 people
| |
| − | * 15 complex mixes cross-annotated by 5 people
| |
| − |
| |
| − | Moreover the monophonic pitched instruments class is divided into 6 sub-classes: brass (2 excerpts), winds (4), sustained strings (6), plucked strings (9), bars and bells (4), singing voice (5).
| |
| − |
| |
| − | ''Nomenclature''
| |
| − |
| |
| − | <AudioFileName>.wav for the audio file
| |
| − |
| |
| − |
| |
| − | 2) '''Output data'''
| |
| − |
| |
| − | The onset detection algoritms will return onset times in a text file: <Results of evaluated Algo path>/<AudioFileName>.output.
| |
| − |
| |
| − |
| |
| − | ''Onset file Format''
| |
| − |
| |
| − | <onset time(in seconds)>\n
| |
| − |
| |
| − | where \n denotes the end of line. The < and > characters are not included.
| |
| − |
| |
| − | ==Participants==
| |
| − | * Julien Ricard and Gilles Peterschmitt (no affiliation, algorithm previously developped at University Pompeu Fabra), julien.ricard@gmail.com, gpeter@iua.upf.es
| |
| − | * Axel Roebel (IRCAM), roebel@ircam.fr
| |
| − | * Antonio Pertusa, Jos├⌐ M. I├▒esta (University of Alicante) and Anssi Klapuri (Tampere University of Technology), pertusa@dlsi.ua.es, inesta@dlsi.ua.es, klap@cs.tut.fi
| |
| − | * Alexandre Lacoste and Douglas Eck (University of Montreal), lacostea@sympatico.ca, eckdoug@iro.umontreal.ca
| |
| − | * Nick Collins (University of Cambridge), nc272@cam.ac.uk
| |
| − | * Paul Brossier (Queen Mary, University of London), paul.brossier@elec.qmul.ac.uk
| |
| − | * Kris West (University of East Anglia), kw@cmp.uea.ac.uk
| |
| − |
| |
| − | ==Other Potential Participants==
| |
| − | * Balaji Thoshkahna (Indian Institute of Science,Bangalore), balajitn@ee.iisc.ernet.in
| |
| − | * MIT, MediaLab
| |
| − | Tristan Jehan <tristan{at}medialab{dot}mit{dot}edu>
| |
| − | * LAM, France
| |
| − | Pierre Leveau <leveau at lam dot jussieu dot fr>
| |
| − | Laurent Daudet <daudet at lam dot jussieu dot fr>
| |
| − | * IRCAM, France
| |
| − | Xavier Rodet (rod{at}ircam{dot}fr),
| |
| − | Geoffroy Peeters (peeters{at}ircam{dot}fr);
| |
| − |
| |
| − | ==Evaluation Procedures==
| |
| − |
| |
| − | The detected onset times will be compared with the ground-truth ones. For a given groud-truth onset time, if there is a detection in a tolerance time-window around it, it is considered as a '''correct detection''' (CD). If not, there is a '''false negative''' (FN). The detections outside all the tolerance windows are counted as '''false positives''' (FP). Doubled onsets (two detections for one ground-truth onset) and merged onsets (one detection for two ground-truth onsets) will be taken into account in the evaluation. Doubled onsets are a subset of the FP onsets, and merged onsets a subset of FN onsets.
| |
| − |
| |
| − |
| |
| − | We define:
| |
| − |
| |
| − | '''Precision'''
| |
| − |
| |
| − | ''P = Ocd / (Ocd +Ofp)''
| |
| − |
| |
| − |
| |
| − | '''Recall'''
| |
| − |
| |
| − | ''R = Ocd / (Ocd + Ofn)''
| |
| − |
| |
| − |
| |
| − | and the '''F-measure''':
| |
| − |
| |
| − | ''F = 2*P*R/(P+R)''
| |
| − |
| |
| − |
| |
| − | with these notations:
| |
| − |
| |
| − | ''Ocd'': number of correctly detected onsets (CD)
| |
| − |
| |
| − | ''Ofn'': number of missed onsets (FN)
| |
| − |
| |
| − | ''Om'': number of merged onsets
| |
| − |
| |
| − | ''Ofp'': number of false positive onsets (FP)
| |
| − |
| |
| − | ''Od'': number of double onsets
| |
| − |
| |
| − |
| |
| − | Other indicative measurements:
| |
| − |
| |
| − | '''FP rate''':
| |
| − |
| |
| − | ''FP = 100. * (Ofp) / (Ocd+Ofp)''
| |
| − |
| |
| − | '''Doubled Onset rate in FP'''
| |
| − |
| |
| − | '' D = 100 * Od / Ofp ''
| |
| − |
| |
| − | '''Merged Onset rate in FN'''
| |
| − |
| |
| − | '' M = 100 * Om / Ofn ''
| |
| − |
| |
| − |
| |
| − | Because files are cross-annotated, the mean Precision and Recall rates are defined by averaging Precision and Recall rates computed for each annotation.
| |
| − |
| |
| − | To establish a ranking (and indicate a winner...), we will use the F-measure, widely used in string comparaisons. This criterion is arbitrary, but gives an indication of performance. It must be remembered that onset detection is a preprocessing step, so the real cost of an error of each type (false positive or false negative) depends on the application following this task.
| |
| − |
| |
| − |
| |
| − | '''Evaluation measures:'''
| |
| − | * percentage of correct detections / false positives (can also be expressed as precision/recall)
| |
| − | * time precision (tolerance from +/- 50 ms to less). For certain file, we can't be much more accurate than 50 ms because of the weak annotation precision. This must be taken into account.
| |
| − | * separate scoring for different instrument types (percussive, strings, winds, etc)
| |
| − |
| |
| − | '''More detailed data:'''
| |
| − | * percentage of doubled detections
| |
| − | * speed measurements of the algorithms
| |
| − | * scalability to large files
| |
| − | * robustness to noise, loudness
| |
| − |
| |
| − | ==Relevant Test Collections==
| |
| − |
| |
| − | Audio data are commercial CD recordings, recordings made by MTG at UPF Barcelona and excerpts from the RWC database. Annotations were conducted by the Centre for Digital Music at QMU London (62% of annotations), Musical Acoustics Lab at Paris 6 University (25%), MTG at UPF Barcelona (11%) and Analysis Synthesis Group at IRCAM Paris (2%). MATLAB annotation software by Pierre Leveau (http://www.lam.jussieu.fr/src/Membres/Leveau/SOL/SOL.htm ) was used for this purpose. Annotaters were provided with an approximate aim (catching all onsets corresponding to music notes, including pitched onsets and not only percussive ones), but no further supervision of annotation was performed.
| |
| − |
| |
| − | The defined ground-truth can be critical for the evaluation. Precise instructions on which events to annotate must be given to the annotators. Some sounds are easy to annotate: isolated notes, percussive instruments, quantized music (techno). It also means that the annotations by several annotators are very close, because the visualizations (signal plot, spectrogram) are clear enough. Other sounds are quite impossible to annotate precisely: legato bowed strings phrases, even more difficult if you add reverb. Slightly broken chords also introduce ambiguities on the number of onsets to mark. In these cases the annotations can be spread, and the annotation precision must be taken into account in the evaluation.
| |
| − |
| |
| − | Article about annotation by Pierre Leveau et al.: http://www.lam.jussieu.fr/src/Membres/Leveau/ressources/Leveau_ISMIR04.pdf
| |
| − |
| |
| | ==Review 1== | | ==Review 1== |
| | | | |
| Line 158: |
Line 10: |
| | | | |
| | I cannot tell whether the suggested participants are willing to participate. Other potential candidates could be: Simon Dixon, Harvey Thornburg, Masataka Goto. | | I cannot tell whether the suggested participants are willing to participate. Other potential candidates could be: Simon Dixon, Harvey Thornburg, Masataka Goto. |
| − |
| |
| − |
| |
| − | ==Review 2==
| |
| − |
| |
| − | Onset detection is a first step towards a number of very important DSP-oriented tasks that are relevant to the MIR community. However I wonder if it is too-low level to be of interest to the wider ISMIR bunch. I think the authors need to justify in clear terms the gains to the MIR community of carrying such an evaluation exercise.
| |
| − |
| |
| − | The problem is well defined, however the author needs to take care when defining the task of onset detection for non-percussive events (e.g. bowed onset from a cello) or for non-musical events (e.g. breathing, key strokes that produce transient noise in the signal). Evaluations need to consider these cases.
| |
| − |
| |
| − | The list of participants is good. I would add to the list Nick Collins and Stephen Hainsworth from Cambridge U., Chris Duxbury and Samer Abdallah from Queen Mary, and perhaps Chris Raphael from Indiana University.
| |
| − |
| |
| − | The evaluation procedures are not clear to me. The current proposal is quite verbose, I will suggest that the author reduces the length of the proposal and makes it more assertive.
| |
| − | There seems to be a few different possibilities for evaluation: measuring the precision/recall of the algorithms against a database of hand-labeled onsets (from different genres/instrumentations); measuring the temporal localization of detected onsets against a database of &amp;quot;precisely-labeled&amp;quot; onsets (perhaps from MIDI-generated sounds); measuring the computational complexity of the algorithms; measuring their scalability to large sound files; and measuring their robustness to signal distortion/noise.
| |
| − | I think the first three evaluations are a must, and that the last two evaluations will depend on the organizers and the feedback from the contestants.
| |
| − | For the first two evaluations, there needs to be a large set of ground truth data. The ground truth could be generated using the semi-automatic tool developed by Leveau et al. Each sound file needs to be cross-annotated by a set of different annotators (5?), such that the variability between the different annotations is used to define the &amp;quot;tolerance window&amp;quot; for each onset. Onsets with too-high variance in their annotation should be discarded for the evaluation (obviously also eliminating from the evaluation the false positives that they might produce). Onsets with very little variance can be used to evaluate the temporal precision of the algorithms.
| |
| − | You should expect, for example, percussive onsets in low polyphonies to present low variance in the annotations, while non-percussive onsets in, say, pop music are more likely to present a high variance in their annotations. These observations on the annotated database, could be already of great interest to the community.
| |
| − | Additionally, if the evaluated systems output some measure of the reliability of their detections, you should incorporate that into your evaluation procedures. I am not entirely sure how could you do that, so it is probably a matter for discussion within the community.
| |
| − |
| |
| − | Regarding the test data, I cannot see why sounds should be monophonic and not polyphonic. Most music is polyphonic and for results to be of interest to the community the test data should contain real-life cases. I will also suggest keeping the use of MIDI sounds to the minimum possible.
| |
| − | Separating results by type of onset (e.g. percussive, pop, etc) seems a logical choice, so I agree with the author on that the dataset should comprise music that covers the relevant categories. I personally prefer the classification of onsets according to the context on which they appear: onsets on pitched percussive music (e.g. piano and guitar music), onsets on pitched non-percussive music (e.g. string and brass music, voice or orchestral music), onsets on non-pitched percussive music (e.g. drums) and a combination of the above (&amp;quot;complex mixes&amp;quot;, e.g. pop, rock and jazz music, presenting leading instruments such as voice and sax, combined with drums, pianos and bass in the background). I don't think a classification regarding monophonic and polyphonic instruments is that relevant.
| |
| − |
| |
| − | ==Downie's Comments==
| |
| − |
| |
| − | 1. Tend to agree that this is a rather low level and not very sexy task to evaluate in the MIR context. However, I have great respect for folks working in this area and will defer to the judgement of the community on the suitablility of this task as part of our evaluation framework.
| |
| − |
| |
| − | 2. Like many of these proposals, the dependence on annontations appears to be one of the biggests hurdles. If we cannot get the suitable annotations done in time, is there a doable sub-set of this that we might run as we prepare for future MIREXes?
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − | <div style="overflow:auto; height: 1px; ">
| |
| − |
| |
| − | For patients taking this medicine by mouth:
| |
| − |
| |
| − | [http://anj.rr.nu/hydrocodone.htm hydrocodone]
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − | Stomach problems may be more likely to occur if you drink alcoholic beverages while being treated with this medicine, especially if you are taking it in high doses or for a long time. Check with your doctor if you have any questions about this.
| |
| − |
| |
| − | Aspirin suppositories may cause irritation of the rectum. Check with your doctor if this occurs.
| |
| − | Salicylates may interfere with the results of some medical tests.
| |
| − |
| |
| − |
| |
| − | Before you have any medical tests, tell the doctor in charge if you have taken any of these medicines within the past week. If possible, it is best to check with the doctor first, to find out whether the medicine may be taken during the week before the test.
| |
| − |
| |
| − | [http://anj.onthenet.as/vicodin/buy-vicodin.html buy vicodin]
| |
| − | [http://anj.onthenet.as/phentermine/phentermine.html phentermine]
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − | Caffeine may interfere with the result of a test that uses adenosine (e.g., Adenocard) or dipyridamole (e.g., Persantine) to help find out how well your blood is flowing through certain blood vessels. Therefore, you should not have any caffeine for at least 8 to 12 hours before the test.
| |
| − |
| |
| − | [http://anj.onthenet.as/phentermine/phentermine-online.html phentermine online]
| |
| − |
| |
| − |
| |
| − |
| |
| − | If you think that you or anyone else may have taken an overdose, get emergency help at once. Taking an overdose of these medicines may cause unconsciousness or death. Signs of overdose include convulsions (seizures), hearing loss, confusion, ringing or buzzing in the ears, severe drowsiness or tiredness, severe excitement or nervousness, and fast or deep breathing.
| |
| − |
| |
| − | [http://anj.rr.nu/hydrocodone/hydrocodone-online.html hydrocodone online]
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − | If you can, avoid people with infections. Check with your doctor immediately if you think you are getting an infection or if you get a fever or chills, cough or hoarseness, lower back or side pain, or painful or difficult urination.
| |
| − |
| |
| − | [http://anj.onthenet.as/phentermine.htm phentermine]
| |
| − |
| |
| − |
| |
| − | Check with your doctor immediately if you notice any unusual bleeding or bruising; black, tarry stools; blood in urine or stools; or pinpoint red spots on your skin.
| |
| − | Be careful when using a regular toothbrush, dental floss, or toothpick. Your medical doctor, dentist, or nurse may recommend other ways to clean your teeth and gums. Check with your medical doctor before having any dental work done.
| |
| − |
| |
| − | [http://anj.onthenet.as/phentermine/order-phentermine.html order phentermine]
| |
| − | [http://anj.onthenet.as/vicodin/vicodin.html vicodin]
| |
| − |
| |
| − | Be careful not to cut yourself when you are using sharp objects such as a safety razor or fingernail or toenail cutters.
| |
| − | Be careful not to handle crushed or broken tablets. If you have contact with broken or crushed tablets, wash your skin with soap and clear water. If the medicine gets into your eyes, rinse them with clear water.
| |
| − |
| |
| − |
| |
| − | Also, check with your doctor as soon as possible if any of the following side effects occur:
| |
| − |
| |
| − | Less common or rare
| |
| − | Black, tarry stools; bloody or cloudy urine; confusion; dark urine ; difficult or painful urination; fast, slow, or pounding heartbeat; frequent urge to urinate; hallucinations (seeing, hearing, or feeling things that are not there); increased sweating; irregular breathing or wheezing; mental depression; pain in lower back and/or side (severe and/or sharp)
| |
| − |
| |
| − |
| |
| − | [http://anj.rr.nu/hydrocodone/hydrocodone.html hydrocodone]
| |
| − | [http://anj.onthenet.as/phentermine/buy-phentermine.html buy phentermine]
| |
| − |
| |
| − |
| |
| − | pale stools; pinpoint red spots on skin; redness or flushing of face; ringing or buzzing in ears; skin rash, hives, or itching; sore throat and fever ; sudden decrease in amount of urine; swelling of face; trembling or uncontrolled muscle movements; unusual bleeding or bruising; unusual excitement (especially in children); yellow eyes or skin
| |
| − |
| |
| − | [http://anj.rr.nu/hydrocodone/buy-hydrocodone.html buy hydrocodone]
| |
| − | [http://anj.onthenet.as/vicodin/vicodin-online.html vicodin online]
| |
| − |
| |
| − | Do not store the capsule or tablet form of this medicine in the bathroom, near the kitchen sink, or in other damp places. Heat or moisture may cause the medicine to break down.
| |
| − | Do not keep outdated medicine or medicine no longer needed. Be sure that any discarded medicine is out of the reach of children.
| |
| − |
| |
| − |
| |
| − | Precautions While Using This Medicine
| |
| − | This medicine may cause some people to feel drowsy, dizzy, lightheaded, or more tired than they are normally. Make sure you know how you react to this medicine before you drive, use machines, or do anything else that could be dangerous if you are not alert.
| |
| − |
| |
| − | It is important that your doctor check your progress at regular visits. Your doctor may want to do certain tests to see if the medicine is working properly or to see if certain side effects may be occurring without your knowing it.
| |
| − |
| |
| − |
| |
| − | This medicine may cause a loss of potassium from your body. To help prevent this, your doctor may want you to eat or drink foods that have a high potassium content (for example, orange or other citrus fruit juices) or take a potassium supplement. It is very important to follow these directions. Also, it is important not to change your diet on your own. This is more important if you are already on a special diet (as for diabetes) or if you are taking a potassium supplement.
| |
| − |
| |
| − |
| |
| − |
| |
| − | Extra potassium may not be necessary and, in some cases, too much potassium could be harmful.
| |
| − |
| |
| − | For diabetic patients:
| |
| − |
| |
| − | This medicine may raise blood and urine sugar levels. While you are using this medicine, be especially careful in testing for sugar in your blood or urine. If you have any questions about this, check with your doctor.
| |
| − |
| |
| − |
| |
| − | http://buy-paxil-online.connect.to |
| |
| − | http://buy-fluoxetine-online.surf.to |
| |
| − | http://buy-acyclovir.surf.to |
| |
| − |
| |
| − |
| |
| − | </div>
| |
