2014:Singing Voice Separation - Revision history

Tak-Shing Chan: /* Data */

2016-04-29T07:42:04Z

‎Data

Tak-Shing Chan: /* Data */

2016-04-29T07:37:04Z

‎Data

Tak-Shing Chan: /* Evaluation */

2014-10-01T07:41:29Z

‎Evaluation

Tak-Shing Chan: /* Evaluation */

2014-09-27T11:46:17Z

‎Evaluation

Tak-Shing Chan: /* Packaging submissions */

2014-09-25T16:10:56Z

‎Packaging submissions

Tak-Shing Chan: /* Submission format */

2014-09-12T04:34:44Z

‎Submission format

Tak-Shing Chan: /* Packaging submissions */

2014-09-12T04:08:21Z

‎Packaging submissions

Tak-Shing Chan: /* Submission format */

2014-09-12T04:06:44Z

‎Submission format

Tak-Shing Chan: /* Packaging submissions */

2014-09-12T04:04:10Z

‎Packaging submissions

Tak-Shing Chan: /* Submission format */

2014-09-12T03:55:54Z

‎Submission format

@@ Line 9: / Line 9: @@
 == Data ==
-A collection of 100 clips of recorded pop music (vocals plus music) are used to evaluate the singing voice separation algorithms.
+A collection of 100 clips of recorded pop music (vocals plus music) are used to evaluate the singing voice separation algorithms (these are the hidden parts of the [http://mac.citi.sinica.edu.tw/ikala/ iKala] dataset). If your algorithm is a supervised one, you are welcome to use the public part of the [http://mac.citi.sinica.edu.tw/ikala/ iKala] dataset for training.
 Collection statistics:
@@ Line 17: / Line 17: @@
 # Duration of each clip: 30 seconds
-This collection comprises of the hidden part of the [http://mac.citi.sinica.edu.tw/ikala/ iKala dataset]. See T.-S. Chan, T.-C. Yeh, Z.-C. Fan, H.-W. Chen, L. Su, Y.-H. Yang, and R. Jang,
+For more information about the [http://mac.citi.sinica.edu.tw/ikala/ iKala] dataset, see T.-S. Chan, T.-C. Yeh, Z.-C. Fan, H.-W. Chen, L. Su, Y.-H. Yang, and R. Jang, "Vocal activity informed singing voice separation with the iKala dataset," in Proc. IEEE Int. Conf. Acoust., Speech and Signal Process., 2015, pp. 718-722.
 == Evaluation ==

@@ Line 16: / Line 16: @@
 # Audio details: 16-bit, mono, 44.1kHz, WAV
 # Duration of each clip: 30 seconds
 == Evaluation ==

@@ Line 31: / Line 31: @@
   >> NSAR = SAR - NSAR;
-The final scores will be determined by the mean over all 100 clips:
+The final scores will be determined by the mean over all 100 clips (note that GSIR and GSAR are not normalized):
 <math>GNSDR=\frac{\sum_{i=1}^{100}NSDR_i}{100}</math>,
-<math>GNSIR=\frac{\sum_{i=1}^{100}NSIR_i}{100}</math>,
+<math>GSIR=\frac{\sum_{i=1}^{100}SIR_i}{100}</math>,
-<math>GNSAR=\frac{\sum_{i=1}^{100}NSAR_i}{100}</math>.
+<math>GSAR=\frac{\sum_{i=1}^{100}SAR_i}{100}</math>.
 In addition, sd, min, max and median will also be reported.

@@ Line 19: / Line 19: @@
 == Evaluation ==
-For evaluation we use [http://hal.inria.fr/inria-00630985/PDF/vincent_SigPro11.pdf Vincent ''et al.'''s (2012)] Source to Distortion Ratio (SDR), Source to Interferences Ratio (SIR), and Sources to Artifacts Ratio (SAR), as implemented by [http://bass-db.gforge.inria.fr/bss_eval/bss_eval_sources.m bss_eval_sources.m] in [http://bass-db.gforge.inria.fr/bss_eval/ BSS Eval Version 3.0]. More specifically, their function will be invoked as follows:
+For evaluation we use [http://hal.inria.fr/inria-00630985/PDF/vincent_SigPro11.pdf Vincent ''et al.'''s (2012)] Source to Distortion Ratio (SDR), Source to Interferences Ratio (SIR), and Sources to Artifacts Ratio (SAR), as implemented by [http://bass-db.gforge.inria.fr/bss_eval/bss_eval_sources.m bss_eval_sources.m] in [http://bass-db.gforge.inria.fr/bss_eval/ BSS Eval Version 3.0]. Everything will be normalized to enable a fairer evaluation. More specifically, their function will be invoked as follows:
   >> trueVoice = wavread('trueVoice.wav');
@@ Line 25: / Line 25: @@
   >> trueMixed = trueVoice + trueKaraoke;
   >> [estimatedVoice, estimatedKaraoke] = wrapper_function_calling_your_separation_algorithm(trueMixed);
-  >> [SDR, SIR, SAR] = bss_eval_sources([estimatedVoice estimatedKaraoke]', [trueVoice trueKaraoke]')
+  >> [SDR, SIR, SAR] = bss_eval_sources([estimatedVoice estimatedKaraoke]' / norm(estimatedVoice + estimatedKaraoke), [trueVoice trueKaraoke]' / norm(trueVoice + trueKaraoke));
+  >> [NSDR, NSIR, NSAR] = bss_eval_sources([trueMixed trueMixed]' / norm(trueMixed + trueMixed), [trueVoice trueKaraoke]' / norm(trueVoice + trueKaraoke));
-  SDR =
+ >> NSDR = SDR - NSDR;
-     -2.7443
+  >> NSIR = SIR - NSIR;
-.7514
+  >> NSAR = SAR - NSAR;
-The final scores will be determined by the average scores over all 100 clips:
+The final scores will be determined by the mean over all 100 clips:
-<math>GSDR=\frac{\sum_{i=1}^{100}SDR_i}{100}</math>,
+<math>GNSDR=\frac{\sum_{i=1}^{100}NSDR_i}{100}</math>,
-<math>GSIR=\frac{\sum_{i=1}^{100}SIR_i}{100}</math>,
+<math>GNSIR=\frac{\sum_{i=1}^{100}NSIR_i}{100}</math>,
-<math>GSAR=\frac{\sum_{i=1}^{100}SAR_i}{100}</math>.
+<math>GNSAR=\frac{\sum_{i=1}^{100}NSAR_i}{100}</math>.
 == Submission format ==

@@ Line 71: / Line 71: @@
 == Packaging submissions ==
 All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed).
-# Be sure to follow the [[2006:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]
+# Be sure to follow the [[2006:Best Coding Practices for MIREX | Best Coding Practices for MIREX]].
-# Be sure to follow the [[MIREX 2014 Submission Instructions]]
+# Be sure to follow the [[MIREX 2014 Submission Instructions]]. For example, under '''Very Important Things to Note''', Clause 6 states that if you plan to submit more than one algorithm or algorithm variant to a given task, EACH algorithm or variant needs its own complete submission to be made including the README and binary bundle upload. Each package will be given its own unique identifier. Tell us in the README the priority of a given algorithm in case we have to limit a task to only one or two algorithms/variants per submitter/team. [Note: our current limit is two entries per team.]
 All submissions should include a README file including the following the information:

@@ Line 62: / Line 62: @@
   wavwrite(voice, 44100, voiceoutfile);
   wavwrite(music, 44100, musicoutfile);
 Following the convention of other MIREX tasks, an extended abstract is also required (see MIREX 2014 Submission Instructions below).

@@ Line 66: / Line 66: @@
 == Packaging submissions ==
 All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guarenteed).
 All submissions should include a README file including the following the information:
 # Command line calling format for all executables and an example formatted set of commands
 # Number of threads/cores used or whether this should be specified on the command line
 # Expected memory footprint
 # Expected runtime
-# Any required environments (and versions), e.g. python, java, bash, matlab.
+# Approximately how much scratch disk space will the submission need to store any feature/cache files?
 == Time and hardware limits ==