2006:Best Coding Practices for MIREX

Best Coding Practices for MIREX

This document is meant to shed light on some good coding practices for all MIREX submissions, based on experiences from the previous iteration of MIREX. General best practice information is provided below.

IMPORTANT:

If something in this document contradicts an instruction given to you with regard to a task-specific submission requirement, the task-specific instructions take precedence.

Submitting

How and What to Submit: Overall, submissions went smoothly last year. To reiterate some points from last year, however, all submissions should be sent in a folder containing your algorithm, any pertanent libraries or additional files, and a README.txt document outlining your name and how your program is used and called from the command line. This folder, which clearly bears your name, should then be compressed into a zip, rar, or tar.

UNIX/Linux submissions: We would very, very strongly encourage that *NIX platform submissions this year be statically linked. Yes, it increases file and submission sizes quite dramatically, but the variance in all the flavors of LINUX and its libraries caused quite a few pains last year, as we often had to track down and download the necessary libraries, leaving us with 10 different versions of C libraries and the like.

Input/Output

Virtually all MIREX submissions operate on the principle that an input file is received, be it an audio or midi file, or a text list of paths and file names to such files. Submissions then process these files, and generate an output file of a format specified by the particular competition task.

Calling Conventions: This year, all submissions are required to take the output file path and name at the command line. Previously, this was not a requirement, but introduced a fair number of problems. For example, some submissions wrote their output files into the input data directory, which for obvious reasons were given read-only permissions. Also, output file naming by submissions sometimes did not correspond to a way that was useable by the evaluator programs. Therefore, all submissions this year should take the output file name and path at the command line, along with the input path and name. A simple example would be a program called in the following manner:

>./myMIREXProgram /data/inputFile.txt /outputDirectory/outputFile.txt

Doing so allows the IMIRSEL lab to explicitly name and place the output files how, and where, we would like them to be written. If parameters need to be set for the program, this is easily done, provided the manner in which the parameters is set are well documented by the submitter. So if, for example, your program needs a specified frame rate, set by a -fr flag, an example calling format could be of the form:

>./myMIREXProgram -fr 1024 /data/inputFile.txt /outputDirectory/outputFile.txt

where the manner by which, as well as the desired parameters to be used are specified upon submission and in the corresponding readme file bundled with the submission of the algorithm.

Concerning MATLAB: All MATLAB submissions should be written in the form of a function. This allows calling the script from the command line very easily. Therefore, a simple MATLAB submission can be called in the following manner:

>myMATLABSubmission(input, output)

where input and output correspond to two strings containing the full name and path of the input files and output files, respectively. Needed parameters can also be set within the function call, by adding input arguements to the function. So, if a frame rate of 1024 needs to be set, a simple example would be:

>myMATLABSubmission(input, output, 1024)

where again, the calling format for setting the parameters, as well as the parameter you wish to be used is specified upon submission and in the bundled read me file.

IMPORTANT: All submitted MATLAB functions should containt an exit statement after the output file is written. This signals to the evaluators that processing has completed, and the output file can now be passed to the evaluator algorithm.

Console Output: Although not trully required, console output does provide a sanity check to the IMIRSEL people when running code. It can be unnerving seeing a program run at 100% CPU for 4 hours, and not be certain if the program is functioning correctly, or has hung up. Therefore, some console output is, indeed, useful. This can be of the form of simply writing out a line corresponding to different stages of your algorithm, for example:

Preprocessing.......done
Extracting Features........done
Training Model........done

and so on. Writing console output is achieved through simple print statements in the code, or in MATLAB, functions such as disp( ). Moreover, if your submission operates on an input list file, it may be useful to write out the name of the file being processed, and its status. For example,

Processing /data/file1.wav .......... done
Processing /data/file2.wav........... 58% complete

This allows us to see if your algorithm is having a problem with a particular file or not, perhaps due to a corrupt wav file, so that we can then go through the data and look for abnormalities.

Intermediate or Temporary Files: If your submission writes some form of intermediate, temporary, or log file, it should be written into the same directory as where the submission executable or script exists, or a subdirectory of this parent directory if so desired. Directories containing any audio files used for training or testing of your algorithm will be write-protected. Your algorithm should not attempt to write into these directories or paths.

General Coding Practices

 Murphy's Law: If anything can go wrong, it will.

Perhaps the most pervasive of all problems run into last year were divide by zeros. Many algorithms rely on some manner of normalization, perhaps to normalize all frames in an audio analysis. Absolute silence in audio files is not at all uncommon, and the effects of this caused many algorithms to produce frame arrays of NaNs. In the case of the machine learning algorithms for the identification tasks, this would throw off the model learners, and so on. The effects were, however, not limitted to the classification tasks, and the effects of NaNs spread throughout tempo, drum detection, and so on. So, if you plan on doing division, check your denominator. Expanding to multiple dimensions, if you wish to perform matrix inversions, it is also suggested you check the conditioning of your matrices. Try to devise some clever way of handling such pathological cases. Remember, it's better to take an evaluation performance hit on a single file, than have it influence your entire submission.

A few submissions last year were sensitive to file lengths, either too short or too long. While we try to state as much as we can about the nature of the source data, bear in mind the effects. Therefore, if your algorithm operates by extracting only one minute of the source audio data to process, make sure it doesn't break the algorithm if the source file is less than a minute. Also, if you are expecting 30 second wav files, make sure your algorithm doesn't break if a longer one is encountered. Both situations happened last year.

If your submission belongs to one of the contests that does batch processing, i.e. operates on input list file such as the similarity competitions, try to see to it that failing to process a single file does not crash the entire program. Try and use some manner of try/catch and incorporate some default behavior for failures on a single file. Perhaps log where such errors occurred, so that we can track them down, but again, it's better to generate some result, than not any at all.