cmd/go: can't run go builds concurrently if they download modules
allenpetersen opened this issue · 20 comments
What version of Go are you using (go version)?
go1.11beta2
What operating system and processor architecture are you using (go env)?
amd64 docker container
Our current build workflow is to run glide install first and then start a number of go build jobs in parallel.
When I take a clean system and start the go build jobs in parallel they fail due to issues downloading modules, git shallow.lock files and race conditions creating and populating /go/src/mod directories.
If I first do something like go get ./foo ./bar ./baz to download the modules I can run the builds in parallel.
Is there an equivalent of glide install or dep ensure?
Is there a plan to make downloading of modules thread safe?
CC @bcmills
For now it should suffice to do 'go list ./foo ./bar ./baz' before your builds.
There is a plan to make downloading of modules by parallel go commands
safe but we haven't done that yet.
There is also a plan to provide 'go mod download' to prefetch all the
downloaded modules. That could also stand in for 'glide install' but
go list is enough and exists today.
Why are you running multiple go build commands in parallel instead of
a single go build command?
Why are you running multiple go build commands in parallel instead of
a single go build command?
We have many services in a monorepo written in a number of different languages. The top level CI builder doesn't know about what language they are written it. It just invokes a number of Makefiles most of which happen to be Go programs.
We have a different "multiple builds at once" use-case at hashicorp/terraform: we build the same application source (and thus the same dependencies) for a number of different GOOS/GOARCH pairs concurrently during our release process.
We are currently doing this with gox, which is a wrapper around go build that deals with templating the executable output paths to produce a systematic result. A number of other projects at HashiCorp, and I believe also elsewhere, are using this tool in the same way.
It sounds like it may be viable for gox to simply run an additional single command to completion before it starts its parallel work to ensure that all of the dependencies are available. Would that be go list ./..., or similar?
We have multiple builds at once at zikichombo.org as well. It is a set of libraries, each a module with very heterogenous versioning projected. We are counting on the module system to allow independent releases, but internally have a need to download and build the whole set of libraries as part of work flow.
Russ wrote this above comment #26794 (comment) back in August:
There is also a plan to provide 'go mod download' to prefetch all the
downloaded modules. That could also stand in for 'glide install' but
go list is enough and exists today.
As brief update, go mod download (documentation) now exists in 1.11, and hence that can now be used instead of go list ./foo ./bar ./baz before a build.
I did switch our build process to call go mod download as a pre-build step and that is working fine for us. This works around the race condition.
I haven't gone back to test this but the issue of having parallel builds fail due to races for the on disk module cache may still remain.
Change https://golang.org/cl/141218 mentions this issue: cmd/go: support background processes in TestScript
Change https://golang.org/cl/145177 mentions this issue: internal/syscall: add LockFileEx and UnlockFileEx for use in cmd/go on Windows
Change https://golang.org/cl/145178 mentions this issue: cmd/go/internal/lockedfile: add package and support library
Change https://golang.org/cl/146380 mentions this issue: cmd/go/internal/{modcmd,modload}: lock edits to go.mod
Change https://golang.org/cl/146382 mentions this issue: cmd/go/internal/modfetch: lock files and directories
Change https://golang.org/cl/146383 mentions this issue: cmd/go/internal/work: don't apply O_EXCL when probing for umask
Change https://golang.org/cl/146381 mentions this issue: cmd/go/internal/modfetch/codehost: add lockfiles for repos
Change https://golang.org/cl/146379 mentions this issue: cmd/go/internal/{clean,test}: lock testexpire.txt
Change https://golang.org/cl/146378 mentions this issue: cmd/go/internal/cache: lock shared mutable files
Change https://golang.org/cl/146377 mentions this issue: cmd/go/internal/renameio: add package
Change https://golang.org/cl/151341 mentions this issue: cmd/go/internal/modfetch: make Repo.Zip write to an io.Writer instead of a temporary file
The stack of changes to fix concurrent invocations has landed, and should be included in the Go 1.12 beta.
If you see any module cache corruption on a build of the go command from commit a2b4ac6 or later, please open a new issue with details.
Change https://golang.org/cl/151798 mentions this issue: cmd/go/internal/modfetch: make directories read-only after renaming, not before