Add continuous integration using our test suite
nocollier opened this issue · 1 comments
nocollier commented
ILAMB does have a test suite that until now I have manually managed. Before making non-trivial commits to master, I go into the test directory and run make. This will execute a reduced set of confrontations against a anonymous model and compare overall scores to the values in scores_test.csv.gold. This means that our current test coverage is based on aspects which would change overall scores (absolute tol = 1e-12).
There are a number of challenges involved in making this run automatically on PR's to master:
- ILAMB would need to be deploy-able automatically, which we can now do via conda. Currently this installs a fixed version, would have to set it up to install the PR.
- The observational and model data must be downloaded. I am not certain of the current size of the required test data, but it is large. CI is available to use for free if we do not need to run too long / very often. In order to make use of this we would need to trim down the model / obs data and perhaps even coarsen it. The tests currently take ~10 minutes to run on 4 processors.
- As ILAMB has grown, I have not continued to add tests to cover new areas. IOMB is unprotected, as is the diurnal work.