Cache /nix/store between jobs
Closed this issue · 14 comments
Hi!
Sorry if this is a stupid question, I am new to using GitHub Actions.
I would like to cache /nix/store
between builds to speed up the setup. Is it as easy as following this:
https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/managing-complex-workflows#caching-dependencies
or are there any caveats or pitfalls I should take into account?
Hmm, I'm giving that a try, but as far as I understand, that will use Cachix as a binary cache for stuff that is build from source. In my case, I just want to save time downloading binaries from Nixpkgs, keeping already a local copy using the Github Actions cache store. Will Cachix also do that? (Does not look like it from looking at it's code, but I'm not sure.)
That's not much of an improvement, either way all that content needs to be downloaded to the runner. Unless you self-host the runner.
Ok, thank you for your reponse! I am using Cachix now and it has still cut a couple of minutes off my actions. Thanks a lot for the great work supporting this great service and your overall contributions to the Nix ecosystem! ❤️
❤️
For posterity: FYI this is the setup I came up with that (currently) works to cache the plain /nix/store
. Note, the cache key is too simplistic, you should include the hash of your .nix
files maybe (with fallback key prefix specified).
But it overcomes: 1) the permission errors you get when trying to untar the stored cache on the second run, and 2) the storing part trying to persist lock files and else which will result in permission errors.
jobs:
tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2.4.0
- run: |
# Create with liberal rights, otherwise cache action will complain
# about permission errors.
sudo mkdir -p /nix/store
sudo chmod -R 777 /nix
- name: Cache nix env take N+1
uses: actions/cache@v2
with:
path: |
# See https://github.com/actions/cache/pull/726
/nix/store/**
# Missing something?
/nix/var/nix/*/*
/nix/var/nix/db/*
/nix/var/nix/db/*/**
!/nix/var/nix/daemon-socket/socket
!/nix/var/nix/userpool/*
!/nix/var/nix/gc.lock
!/nix/var/nix/db/big-lock
!/nix/var/nix/db/reserved
key: ${{ runner.os }}-nix-store
- uses: cachix/install-nix-action@v15
with:
nix_path: nixpkgs=channel:nixos-unstable
- name: Install custom nix env
run: nix-env -f my-build.nix -i '.*'
That's pretty cool, thanks @robinp !
For posterity: FYI this is the setup I came up with that (currently) works to cache the plain
/nix/store
. Note, the cache key is too simplistic, you should include the hash of your.nix
files maybe (with fallback key prefix specified).But it overcomes: 1) the permission errors you get when trying to untar the stored cache on the second run, and 2) the storing part trying to persist lock files and else which will result in permission errors.
jobs: tests: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2.4.0 - run: | # Create with liberal rights, otherwise cache action will complain # about permission errors. sudo mkdir -p /nix/store sudo chmod -R 777 /nix - name: Cache nix env take N+1 uses: actions/cache@v2 with: path: | # See https://github.com/actions/cache/pull/726 /nix/store/** # Missing something? /nix/var/nix/*/* /nix/var/nix/db/* /nix/var/nix/db/*/** !/nix/var/nix/daemon-socket/socket !/nix/var/nix/userpool/* !/nix/var/nix/gc.lock !/nix/var/nix/db/big-lock !/nix/var/nix/db/reserved key: ${{ runner.os }}-nix-store - uses: cachix/install-nix-action@v15 with: nix_path: nixpkgs=channel:nixos-unstable - name: Install custom nix env run: nix-env -f my-build.nix -i '.*'
I'd followed this but always got the following error when saving the nix packages to the cache
after having my project built successfully:
Warning: Maximum call stack size exceeded
@robinp I couldn't get your snippet to work, but had some luck with this (Edited, mentioned the wrong person):
- name: Cache nix env take N+1
uses: actions/cache@v2
id: nix-cache
with:
path: |
/tmp/nix-cache
key: ${{ runner.os }}-nix-cache
- name: cache hit
if: steps.nix-cache.outputs.cache-hit == 'true'
run: |
sudo chown -R root /tmp/nix-cache
sudo cp -fRT /tmp/nix-cache/ /nix/
# cp sometimes errors out on MacOS, but rsync works.
# sudo rsync -a /tmp/nix-cache/ /nix/
sudo pkill nix-daemon
- name: do stuff
run: nix-shell --command 'echo OK'
- run: |
sudo cp -r /nix /tmp/nix-cache
# cp sometimes errors out on MacOS, but rsync works as below.
# sudo rsync -a --exclude .Trashes --exclude /var/nix/daemon-socket /nix/ /tmp/nix-cache/
sudo rm -rf /tmp/nix-cache/var/nix/daemon-socket/socket
sudo rm -rf /tmp/nix-cache/var/nix/userpool
sudo rm -rf /tmp/nix-cache/var/nix/gc.lock
sudo rm -rf /tmp/nix-cache/var/nix/db/big-lock
sudo rm -rf /tmp/nix-cache/var/nix/db/reserved
Basically, I gave up on trying to cache /nix
directly as it's too special of a directory. Instead, I make an intermediary copy, then cache that. On cache hit/restore, I manually copy over stuff.
It worked great on Linux, but was really slow on MacOS. Especially creating the intermediary copy (which is absolutely necessary on MacOS, as /nix
is a special volume on MacOS).
Tragically, it's on MacOS where I need the speed up the most, and this didn't really help too much. On Linux, for my projects, it's almost not worth the complexity.
I wish nix
had a first-class way to get just the stuff we need to cache. I can't quite tell if this is helpful: https://fzakaria.com/2020/08/11/caching-your-nix-shell.html since I'm not actually compiling any binaries, really just downloading stuff.
Or this: https://docs.cachix.org/faq#is-there-a-way-to-cache-nix-shell
Both seem to help only when compiling stuff. I previously discussed this with @domenkozar on cachix/cachix-action#61 , but it was a while back. I wonder if things have changed since then.
If you want to cache the Nix store with the GitHub Actions cache, instead of trying to work around the challenges of caching the raw /nix/store
, there is a simpler way that also produces smaller caches: nix-store --{import,export}
.
- name: "Cache Nix store"
uses: actions/cache@v3.0.8
id: nix-cache
with:
path: /tmp/nixcache
key: "FIXME: Pick a cache key suitable for your use case"
- name: "Install Nix"
uses: cachix/install-nix-action@v17
with:
install_url: "https://releases.nixos.org/nix/nix-2.11.0/install"
- name: "Import Nix store cache"
if: "steps.nix-cache.outputs.cache-hit == 'true'"
run: "nix-store --import < /tmp/nixcache"
- name: "Enable Cachix"
uses: cachix/cachix-action@v10
with: "FIXME: Fill in your details"
- name: "Build application"
run: "nix build --print-build-logs"
- name: "Export Nix store cache"
if: "steps.nix-cache.outputs.cache-hit != 'true'"
run: "nix-store --export $(find /nix/store -maxdepth 1 -name '*-*') > /tmp/nixcache"
It works reasonably well in combination with Cachix; I’ve found that for some derivations that need to get hundreds of small-ish store paths from Cachix, it can take a few minutes to fetch them individually, while downloading from the GitHub Actions cache + importing into the Nix store together took less than a minute. The downside is that you need to watch out for what is in your /nix/store
when the cache entry gets produced. For example, if your build is a no-op when the GitHub Actions cache is filled, because the result was in Cachix, then the build dependencies will not be realized, and they will be absent from the GitHub Actions cache.
If you use flake, you could use experimental nix copy
command instead of nix-store --{import,export}
.
For example, if you want to cache your devShell
flake.nix
:
{
outputs = { nixpkgs }: with nixpkgs.legacyPackages.x86_64-linux; {
devShell.x86_64-linux = mkShell {
buildInputs = [
nodejs-16_x
];
};
};
}
You could do:
- name: "Cache Nix store"
uses: actions/cache@v3.0.8
id: nix-cache
with:
path: /tmp/nixcache
key: "FIXME: Pick a cache key suitable for your use case"
- name: "Install Nix"
uses: cachix/install-nix-action@v17
with:
install_url: "https://releases.nixos.org/nix/nix-2.11.0/install"
- name: "Import Nix store cache"
if: "steps.nix-cache.outputs.cache-hit == 'true'"
run: "nix copy --from /tmp/nixcache ./#devShell.x86_64-linux"
- name: "Enable Cachix"
uses: cachix/cachix-action@v10
with: "FIXME: Fill in your details"
- name: "Build application"
run: "nix build --print-build-logs"
- name: "Export Nix store cache"
if: "steps.nix-cache.outputs.cache-hit != 'true'"
run: "nix copy --to /tmp/nixcache ./#devShell.x86_64-linux"
Caching /nix/store directly worked for me in https://github.com/nix-community/cache-nix-action
Anybody has something equivalent for Gitlab CI ?