CondeNast/atjson

Add performance regression testing on pull requests

Closed this issue · 0 comments

We currently have some fairly rudimentary performance regression testing in renderer-commonmark. It turns out that it's really hard to catch these, because they don't cause any test failures, nor does it give any indication of whether the change in atjson caused any relative change in performance.

The proposal here is the following:

  • move these benchmarks / integration tests to a top level tests folder
  • run a github action that runs on each pull request that stores the previous performance runs in memory and provides an indication, with probability of whether this is actually meaningful.
  • improve our performance tests so we can track down performance regressions more accurately.