/mercury-challenge

Public Repository for the IARPA Mercury Challenge

Primary LanguageJupyter Notebook

mercury-challenge

Public Repository for the IARPA Mercury Challenge. Challenge participants should follow this repository to receive notices of code updates and other release information. The files in this directory are provided for IARPA’s Mercury Challenge. You can learn about the challenge and register to compete at https://iarpa.gov/challenges/mercury.html. The discussion forum for the Challenge can be found at http://apps.topcoder.com/forums/?module=ThreadList&forumID=630604 (login required.)

Software

Official Scoring

The official scoring package is hosted and the REST endpoints for warning submission and can be found by going to the challenge in Topcoder (see "Making a submission"): http://community.topcoder.com/longcontest/?module=ViewProblemStatement&rd=17231&compid=67293&popup=false

Detailed Submission process can be found here: https://solutions.topcoder.com/mercury-challenge/submission

Leaderboard can be seen here (updated as and when GSRs are completed): http://community.topcoder.com/longcontest/?module=ViewStandings&rd=17231

ExpressScore

The ExpressScorer package provides Challenge participants with a lightweight testing engine that does not require a Docker installation. The code is found at https://github.com/planetmercury/mercury-challenge/tree/master/src/ExpressScore and tutorial Jupyter notebooks are found at https://github.com/planetmercury/mercury-challenge/blob/master/src/ExpressScore/notebooks ExpressScore has been tested on MacOS Sierra, Windows 7, and Windows 10 and requires Python 3.4 or greater.

Installation

Before beginning, we recommend installing Anaconda (https://www.anaconda.com/download/)

  • Anaconda is a free and open source distribution of the Python and R programming languages for data science and machine learning related applications (large-scale data processing, predictive analytics, scientific computing), that aims to simplify package management and deployment.

Windows 7/8/10

Install Visual Studio Build Tools 2017

Linux

Installing CMake is required before continuing.

  • If using Ubuntu, run apt-get install cmake.
  • If using Centos/RHEL, run yum install cmake.
  • You can also build CMake from source by following the instructions here: https://cmake.org/install/

All Platforms

In the terminal of your choice, navigate to src/ExpressScore.

Run conda env create -f expressscore.yml -v to install the required dependencies for the ExpressScorer.

Once complete, activate the environment.

  • Run activate mc_minimal if on Windows
  • Run source activate mc_minimal if on Mac or Linux

To ensure successful installation, open the test directory inside src/ExpressScore and run test_ma_scoring.py and test_case_count_scoring.py. If all tests pass, congrats! You now have a working ExpressScorer environment.

Baserate Models

The Baserate models make predictions using only ground truth history. The scores for these predictions provide a minimum threshold that Challenge participants must exceed in order to be ranked.

Documentation

Challenge Handbook

The Mercury Challenge Handbook and Appendix provide complete documentation for the challenge, including definitions of the several event types and scoring rules. They can be found in https://github.com/planetmercury/mercury-challenge/blob/master/doc

Jupyter Notebooks for Code Tutorials