tazjin/nixery

nixery.dev registry is down

mkmik opened this issue · 9 comments

mkmik commented

FYI:

$ docker pull nixery.dev/shell
Using default tag: latest
Error response from daemon: received unexpected HTTP status: 500 Internal Server Error
tazjin commented

There seem to be some errors after the last channel bump. Nixery's logs are full of Nix evaluation errors. Taking a look.

tazjin commented 
<tazjin> [nix] while evaluating anonymous function at /nix/store/lz3s922vzs2fvb16p550w3wpga409kzm-source/third_party/overlays/emacs.nix:2:1, called from /nix/store/lz3s922vzs2fvb16p550w3wpga409kzm-source/nix/readTree/default.nix:67:10:
<tazjin> [nix] creating directory '//.cache': Read-only file system

This was the error previously causing the failures, after moving to the latest bump things seem to work fine. I suspect that a recent nixpkgs bump caused some incompatibility with this particular overlay.

Notably, monitoring detected this but didn't alert - going to take a look at that.

  • 👍1
mkmik commented

@tazjin

This was fixed but now it (or something similar) happened again:

  Warning  Failed     21s (x2 over 77s)  kubelet            Failed to pull image "nixery.dev/shell/findutils/gnugrep/gnutar/hexdump/less/mount/procps/curl/grpcurl/binutils/gdb/strace/rustc/linuxpackages.perf/perf-tools/gzip/zstd": rpc error: code = Unknown desc = Error response from daemon: received unexpected HTTP status: 500 Internal Server Error
tazjin commented

It's possible that you ran a command in the middle of a restart (which only take a few seconds); with the setup we have right now the number of restarts is unfortunately pretty high.

We have a tracking bug for this in the TVL infrastructure and I'll attempt to prioritise it soon.

mkmik commented

Thanks

tazjin commented

Quick update: As of cl/4064 the frequency of config changes for Nixery is going to be reduced significantly, which should make occurrences of this much less frequent.

Do note however that nixery.dev is provided without any SLO at the moment.

mkmik commented

Do note however that nixery.dev is provided without any SLO at the moment.

Thanks for running it!

(I have it in my backlog to self-host it somewhere, call me lazy but I'm just waiting until the self-hosting docs are more fleshed out :-) )

tazjin commented

Do you use NixOS? At TVL we have a module for it which you might be able to adapt pretty easily: https://cs.tvl.fyi/depot/-/blob/ops/modules/nixery.nix

mkmik commented

No I don't use nixos yet; always wanted to find a good excuse to play with it; I guess I just found one! :-)