Issues with repairing broken path in server
arximboldi opened this issue · 6 comments
Hi!
Something very strange has happened in one of my deployments. Even though I did not get any errors when using morph deploy switch, in the server, one of the path /nix/store/<asdasd>-some-package/... (a large one) is missing quite a few packages.
Running nix-store --verify --check-contents on the server shows that this in an issue.
However running nix-store --repair-path is not working, because the server I guess does not know how to redownload the package.
What is the recommended way to solve this situation? I am tempted to solve it manually but that's messy bussiness. I am looking for an option in morph itself to repair or force push a path but I can't find it...
Hmmm, worked around it by running:
sudo nix-store --delete --ignore-liveness /nix/store/...faulty-path...
That is a good way to do it. You should try and verify that the disk in question is healthy -- morph exclusively uses nix(-copy-closure) to transfer store paths, and nix should not break store paths en-route to registration.
I think it may have happened because of running deploy switch locally while CI was running deploy dry-activate simultaneously. Could that be the case?
It absolutely should not be the case. Everything goes through the nix-daemon (on a normal nixos system) and any number of clients should be able to realise paths at the same time. If you're able to reproduce corrupting store paths like this and your hardware is fine, that sounds like a quite serious bug (but one I have not witnessed.)
Hmmm, interesting. Really not sure what happened. The dry-activate on Github Actions seemed to have hanged, while a local deployed succeeded. Maybe the dry-activate never got to finish, maybe it hanged mid submission of that path. I'll keep my eyes open in case it happens again with a more clarity about how.
And thank you very much for the quick responses!