2025:Live Leaderboards

Live Leaderboards

To foster a more competitive and/or educational environment, MIREX 2025 will offer experimental support for live leaderboards. This optional feature allows participants to receive immediate feedback by viewing their performance and ranking, enabling them to iteratively refine and resubmit their submissions. We will provide experimental support for (details will follow):

• An online repository to receive submissions and their revisions;
• An API for automatically fetching and downloading new submissions;
• A guide for automatically updating the corresponding MIREX Wiki page.

Please note: Task captains are still responsible for setting up and maintaining their own evaluation server, which must handle the following (either manually or automatically):

• Preparing the test set and evaluation metrics;
• Communicating with the MIREX server to retrieve submissions;
• Running evaluations and computing metrics;
• Uploading results to the MIREX Wiki leaderboard.

Best Practices

For tasks that wish to incorporate a live leaderboard, we recommend the following best practices:

• Use a validation or development set (rather than the final test set) during the leaderboard period to prevent overfitting to the test data;
• Limit the number of evaluation samples used during the live leaderboard period to reduce computational load;
• Re-evaluate all final submissions on the full test set after the live leaderboard closes to ensure fair and consistent benchmarking;
• Communicate with MIREX organizers if you encounter any issues or have suggestions for improving the process.