AoC Bench
This application allows for benchmarking Advent of Code solutions.
There are 2 parts to this, the web frontend (under www
) and the benchmarker (bench.php
).
The benchmarker should be run on a cron and is responsible for generating a results.json
file that is then used by the web frontend to display the data.
There is additionally inputMatrix.php
which will attempt to run each participant against all the inputs from all participants for each day for comparison.
Installing
- Check out the code somewhere.
- Configure
config.local.php
(SeeConfiguration
section) - Cron
checkRun.sh
every minute (This will look for a.doRun
file to know when to run, and handle not clobbering existing runs) - Either cron
checkScheduledUpdates.php
(and enable appropriate configuration options) or periodically touch a.doRun
file to actually schedule the runs - Make
www
available via a web server.
Configuration
Most of the configuration should be done in a file called config.local.php
, all the supported config settings are defined in config.php
.
The important bit is configuring participants.
An example participant:
$participants[] = new class extends Participant {
public function getName() { return 'Dataforce'; }
public function getRepo() { return 'https://github.com/ShaneMcC/aoc-2018'; }
public function getDayFilename($day) { return $day; }
public function getInputFilename($day) { return $this->getDayFilename($day) . '/input.txt'; }
public function getRunCommand($day) { return './docker.sh --time ' . $day; }
};
The important bits are:
- The participant name (Spaces will be removed and used as the folder name for the repo on disk)
- The participant repo (This will be automatically checked out)
- The
getDayFilename($day)
function should return the path to the directory or file containing the day. - The
getInputFilename($day)
function should return the path to the input file for the day.- Shown here is the default implementation of this, which if sufficient, can be ignored.
- The input for each day needs to be a text file on disk (If all participants are to be benchmarked against the same input)
- It is assumed (Though not explicitly required) that each participant's code will run in Docker.
- In most cases, the Docker image is just a basic runtime environment for the language of choice, with the code mounted as a bind-mount from the on-disk repo.
- At the very least
getInputFilename($day)
should return a (relative to the repo root) path that will ultimately be mounted inside the container.
getInputAnswer($day, $part)
can also be defined, by default this will look for answers.txt withingetDayFilename($day)
with part 1 on line 1 and part 2 on line 2.
When benchmarking, the following happens per-participant:
- The
updateRepo($dir)
method of the Participant is called.- The default implementation should suffice for most people, it checks out the latest copy of the repo or calls
git update
- The default implementation should suffice for most people, it checks out the latest copy of the repo or calls
- The
prepare()
method of the Participant is called.- The default implementation will run docker.sh or run.sh if found.
- This should build the required docker container for the Participant for example.
hasDay($day)
will be checked for the participant.- The default implementation checks the git version of
getDayFilename($day)
returns non-null
- The default implementation checks the git version of
- If
$normaliseInput
is set, then the file specified bygetInputFilename($day)
will be overwritten with the input for the day.- This will either be taken from
./inputs/<day>.txt
or fallback to the file referenced bygetInputFilename($day)
on the first-defined participant. - The user should implement
getInputAnswer($day, $part)
as either a global functionconfig.local.php
or within the first-defined participant.- This should return a non-
NULL
string to look for in the output to allow for validation.
- This should return a non-
- This will either be taken from
run($day)
will be called multiple times to run the day the required number of times- By default this will call
getRunCommand($day)
and then run that and return the output. - If a custom
run($day)
implementation is required, this should return[$returnCode, $outputArray]
. A non-0$returnCode
is considered a fail and$outputArray
will be displayed for debugging.
- By default this will call
extractTime($outputArray)
will be called on the result fromrun($day)
to extract the time value.- The default implementation assumes that the 3rd-from-last line of the output will contain
real 0m0.000s
or so as per thetime
function inbash
. - If the output time is not in
real
format, it should be converted to0m0.000s
format for the frontend to understand.
- The default implementation assumes that the 3rd-from-last line of the output will contain
- After all the days are run,
cleanup()
will be called.- The default implementation runs
cleanup.sh
if it exists, and thengit reset --hard origin
- The default implementation runs
Repo Requirements
A repo that conforms to the following behaviour should just work "out of the box" with a Participant
that consists of just a getName()
and getRepo()
configuration. Most bits can be changed with the additional functions mentioned above.
- Days are stored in directories named
1
,2
, ...24
,25
- Input file for each day is stored as
input.txt
within the appropriate directory (eg1/input.txt
) - Expected answers for each day optionally within
answers.txt
within the appropriate directory (eg1/answers.txt
) - A script in the root repo called
run.sh
ordocker.sh
- This will be run with the checked out repo as
pwd
- This directory will be chmoded so that any user can write to it (eg if the container runs with
USER nobody
) - You should not assume any other directory is writable.
- This directory will be chmoded so that any user can write to it (eg if the container runs with
- When run without any arguments, this should build any required containers. It is expected that this container can be built infrequently and reused.
- When run with a day argument (eg
./run.sh 1
) this will compile (if required) and then run that day using bashtime
.- Any compilation output should be stored between runs of the same day for speed/efficiency between test runs.
- Compilation should happen within the docker container, and store the output within the mounted
pwd
directory so that it will be rediscovered by future runs of the container.
- Compilation should happen within the docker container, and store the output within the mounted
time
output should be the very last 3 lines in the result of the./run.sh 1
command.time
should not include any time spent compiling, just the time to run the script or compiled binary
- The code should always use
${pwd}/<day>/input.txt
as the input source.- The input file may be overwritten before running to ensure the same input is tested for each participant in case some yield a faster solve time.
- The input file should be mounted within the container (either on it's own or the whole
pwd
or so)
- The container should exit as soon as the single-run has finished.
- Any compilation output should be stored between runs of the same day for speed/efficiency between test runs.
- This will be run with the checked out repo as
- A
cleanup.sh
script can optionally exist in the repo to run any post-test cleanup required beyondgit reset --hard origin; git clean -fx
Updating
git pull
or equivalent.
Comments, Questions, Bugs, Feature Requests etc.
Bugs and Feature Requests should be raised on the issue tracker on github, and I'm happy to receive code pull requests via github (Though I do not guarantee that all will be merged.)
I can be found idling on various different IRC Networks, but the best way to get in touch would be to message "Dataforce" on Quakenet, or drop me a mail (email address is in my github profile). I can also probably be found on the unofficial AoC Discord (Dataforce#4726) and IRC Channels as Dataforce and on the subreddit.