Suggestion: Use github actions to cache numba kernels
sjperkins opened this issue · 0 comments
Describe the problem that the feature should address
Currently CI is taking an increasing amount of time to run because all numba kernels have to be compiled from scratch.
In conversation with @JSKenyon, it seems most, if not all kernels are decorated with cache=True
. This implies that its not necessary to recompile kernels from scratch all the time.
Describe the solution you'd like
Numba respects a NUMBA_CACHE_DIR variable, while github actions has access to per-repository-caches.
A github action cache key can be constructed from shell/environment variables/hashing files and this is used to determine cache hits/misses. e.g. numba-cache-abcdef1234567890
. However, I suspect that we don't really want to manage these hits and misses ourselves -- we really just want some path that we can point NUMBA_CACHE_DIR
at and let numba handle individual kernel cache hits/misses according to it's internal logic.
If the above follows, then it should be sufficient to use a well structured key (i.e. prefixed with numba-cache-
) in conjunction with the actions/cache@v3
restore-keys
to restore the most recent numba kernel cache and construct a new one during the current CI run.
This should speed up CI runs to the degree that code changes propagate to all numba code -- a change fundamental to all numba kernels will recompile the entire code base, while a peripheral one might just change one kernel.