This tool allows participants to stand up their own evaluation environment that is the same as ours (except for the codalab queue delivery).
Docker
The worker.py takes 4 arguments, all required.
- --solutionfolder is the folder containing your custom starter kit solution
- --scenariofolder is the folder containing the test scenarios
- --airliftlibrary is the folder containing the version of the airlift environment you wish to use, this is separate from what might be installed on your own computer
- --hostmountfolder is the output folder you want the evaluator to put your results into
- Download the newest airlift challenge scenario:
- Download the newest starter kit if you don't already have it:
- Get sample scenarios:
worker.py --solutionfolder fullpathofmysolutionfolder --scenariofolder fullpathofmyscenariofolder --airliftlibrary fullpathofairliftenvfolder --hostmountfolder fullpathofoutputfolder
After making any changes or updates to the airlift library, you should delete the docker image created by using docker image rm eval, and stop and remove any associated running containers to ensure the image builds with the new code.
Notes: There's no need to install anything except docker, but having a local instance of the airlift environment may speed up development.
Distribution Statement A: Approved for Public Release; Distribution Unlimited: Case Number AFRL-2023-5705, CLEARED on 8 Nov 2023