Use BenchmarkCI to automatically detect major runtime regressions
mfherbst opened this issue · 3 comments
Since this runs in the CI we need to be a little conservative on what we time, some ideas:
Systems:
- Something with many k-points but low Ecut
- Something with many symmetries
- Something with few k-points and higher Ecut
Tasks:
- A Hamiltonian application
- A single SCF step
- Density computation and symmetrisation
- Basis construction
- A Sternheimer solve
- A very crude response solve / derivative computation using AD
Probably with timing an SCF step and a single application of ε^\dagger for response on a not too trivial system we already do the key 80%.
I've just started working on that framework before seeing this issue. I can propose something if you have not yet started.
It would use PkgBenchmark, and I am also thinking about flexibility to run big humongous test-cases from your testproblems.
PkgBenchmark seems to be able to do the job easily coupled to @testitems. However if you want to look at older regressions (before a PR using PkgBenchmark and even before TestItemRunner), it is a bit more delicate.
I have not yet worked on this, no. Feel free to propose something. I don't care so much about the backend. BenchmarkCI just looked like the least hassle and pretty neat to use in CI and locally.
I think most important is simplicity and flexibility, e.g. to just run basic tests like the ones I sketched above by dropping a julia runner script into a folder, but at the same time be able to only filter a subset for faster testing on github's CI.
I also bought a bigger machine recently, which we can use for larger periodic testing.
And of course regressions should be automatically annotated in PRs, so you don't miss them.