Text file busy: Issue with Bazel, Go & Tilt

[nikita-tkachov]

Hi Tilt team!

I really liked the configuration for 3-recommended in this repository, so I started to use it for my personal project.

However, I've ran into an issue with the live update portion for the binaries, and was wondering if I could get some guidance or advice on how to fix it.

The issue seems to be that sometimes, when updating any of the Go files, I get the below error:

sh: /app/example-go-image.binary.runfiles/__main__/example-go-image.binary_/example-go-image.binary: Text file busy

When this happens, the Tilt service seems to get "stuck", and it doesn't respond to subsequent updates. You have to manually kill the Kubernetes pod, and then it starts listening to changes again.

For the example repository, it seems to rarely happen, and the only way I was able to reproduce it was by spamming updates to the main.go file (adding and removing a string from the log functions). However, in my personal project, it happens every 1 - 2 updates, and I have to manually kill the pod in order to fix it. I suspect this is because in my personal project, I import a lot more libraries, which I believe means that my binary file will be bigger.

I think it could be awesome ( if there isn't a good solution to the problem) if Tilt was able to recover itself after a failure like this, since it seems to be related to Unix (https://stackoverflow.com/questions/16764946/what-generates-the-text-file-busy-message-in-unix)

Here's how the output looks like when the error occurs:

     Updating container: 44da823360
Will copy 1 file(s) to container: 44da823360
- '/home/nick/.cache/bazel/_bazel_nick/f71be73abc7d8b09a9274ac44e0c0a71/execroot/__main__/bazel-out/k8-fastbuild/bin/example-go_/example-go' --> '/app/example-go-image.binary.runfiles/__main__/example-go-image.binary_/example-go-image.binary'
sh: /app/example-go-image.binary.runfiles/__main__/example-go-image.binary_/example-go-image.binary: Text file busy
[K8s EVENT: Pod example-go-7b6d7bb7d-d72b9 (ns: default)] Container image "registry.local:5000/example-go-image:tilt-7ac3ff7ef2b67da5" already present on machine
Detected a container change for example-go. We could be running stale code. Rebuilding and deploying a new image.
2020/12/25 18:02:27 Serving on port 8000
2020/12/25 18:02:27 Deploy time: 1m40.4s

And here's a screenshot. If you look on the right-hand side, you can see that it says it "deployed", but the service is stuck in the "grey" state instead of the green state. Any subsequent updates to the files don't seem to have an effect on the service.

Thank you, and please let me know if there's anything I can do to help debug it.

[nikita-tkachov]

Edit: I noticed that I was on 0.17.6 and I updated to 0.18.1. That seems to have fixed the issue.

My bad!

[nicks]

Hi @nikita-tkachov ! I'm glad upgrading fixed it. But I'm not sure what change fixed it. 🤔 If you see this again, please re-open!