Automate running HMR "canary" tests before release
j-wags opened this issue · 2 comments
j-wags commented
Spec:
- GH action
- Runs on openforcefields PR that adds a new file
- Does HMR sims w/ 4fs timestep if a new offxml is being added
- Loads pre-charged molecule set
- Molecule set lives in a separate repo
- Should be seeded from s99F “coverage set”
- Also add propyne molecule
- Emphasize that tests are “canaries” – NOT meant to make rigorous benchmark data, just designed to prevent release of a bad FF.
- These tests will NOT make data artifacts
- Tests run
- 4fs, HMR, vacuum -- Example at #19
- Verbose output – Each test should print its SMILES and success/failure, so that it’s easy to reproduce locally if a molecule fails.
j-wags commented
A more fundamental source for the "coverage set" may be here: https://github.com/openforcefield/open-forcefield-data/tree/master/Utilize-All-Parameters/selected
mattwthompson commented
The first and simplest tests have been added. #22 adds HMR canary tests only for two molecules (a simple propyene and a larger molecule reported in the root issue). I think the next steps should be in two directions:
- Establish a broader set of molecules that we would expect to be stable in HMR simulations. Some of the molecules in a coverage set (https://github.com/openforcefield/openforcefields/blob/09da994f0745a62be2297709487f235c683542a1/canary/data/coverage.smi) cannot be typed.
- Consider, based on need, adding in other basic checks outside of HMR simulations. Keep in mind that rigorous benchmarking and physical validation is out of scope, but there may be other simple tests that can safeguard against the release of a bad force field.