cmd/go: should default to use mod=vendor flag if the vendor folder exists
JeremyLoy opened this issue · 89 comments
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go version go1.11 darwin/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
What did you do?
- Given a fresh install of
go, there are no files downloaded in$GOPATH/pkg/mod. - I cloned a repo that uses the new Go 1.11 modules with vendored dependencies
- I executed
go run main.goto run the application.
What did you expect to see?
- the application ran without downloading external dependencies.
What did you see instead?
- The go tool downloaded all dependencies to
$GOPATH/pkg/mod, despite the fact that the dependencies were vendored
I know this can easily be fixed by instead go run -mod=vendor main.go, but because the go tool is now being advertised as "context aware", I think this would be a good QoL change.
@JeremyLoy note that while the default is to ignore the vendor directory, you can relatively easily change that behavior with an environmental setting GOFLAGS=-mod=vendor. You could use that in CI, or could set it in a .bashrc or similar for day-to-day use if you want.
This is a new environmental variable in Go 1.11, where one of the justifications for introducing it was specifically to make it easier to opt in to vendoring given some people would want that regularly (e.g., see #26585 or https://go-review.googlesource.com/c/go/+/126656).
From the doc:
GOFLAGS
A space-separated list of -flag=value settings to apply
to go commands by default, when the given flag is known by
the current command. Flags listed on the command-line
are applied after this list and therefore override it.
One other quick note is that those downloads end up in a shared cached (so even if you don't opt-in to vendoring, it cuts down on how many times you would end up downloading dependencies).
I strongly agree with this proposal. If a project committed a vendor folder, this is a strong indicator that vendoring should be used by default. I can’t think of a workflow in which a developer would commit a vendor folder but prefer other developers to ignore it by default.
In other words, if a project decides to use vendoring, then the whole go tool should default to use vendor first, and download/cache only if not there.
By the way, this is part of what GOFLAGS was added for: https://go-review.googlesource.com/c/go/+/126656
GOFLAGS isn’t the right solution here. This is not a per-user / per-computer decision like “add debug flags”. This is a per-project decision that should affect all users building it. Currently, if a project uses vendoring, there is no way to let all builders use vendoring by default.
WhileGOFLAGS is a decent immediate workaround, I do want to stress the importance of the vendor folder as a part of this issue.
To put it bluntly, Github is not an artifact repository. At any given time, a repository owner can either
- delete their account
- delete a repository
- force push deletions to master
- remove a tag
- change a tag
To avoid any of the above damaging your project, vendoring is a necessity, and should be treated as a first class citizen in the go toolchain.
There seems to be a rush to remove vendor from the go toolchain. It is still referenced as an "experiment" and there was a bit of swirl in even getting modules to support vendoring in its first release. Until Project Athens or equivalent becomes mainstream, vendoring should be fully supported (but never required).
EDIT: coincidentally, I found out the official Heroku build pack for Go uses this very workaround. It seems like this really should be the default case.
Currently, if a project uses vendoring, there is no way to let all builders use vendoring by default.
But we're talking about working in module mode here, correct? Or by "currently" do you mean GOPATH-mode? Because vendoring as a "first class citizen" continues to work in GOPATH mode, it's just that in module mode it is not the default.
To avoid any of the above damaging your project, vendoring is a necessity, and should be treated as a first class citizen in the go toolchain.
There was a similar discussion over at https://twitter.com/_myitcv/status/1033824806930534400. Vendoring (as it was and still is in GOPATH mode) is one way of dealing with this problem. As I suggested, having a separate repo as a module download cache is another, which actually have a number of derivative benefits.
As you say, Athens (or indeed anything else that talks the published module protocol) are also solutions, solutions that in some cases try to solve even bigger problems.
There seems to be a rush to remove vendor from the go toolchain
I don't think this is quite accurate. In GOPATH mode (the default), vendoring works exactly as it has done up until now. In module mode (the opt in) the behaviour changes.
So for the record, in module mode I am against what is being proposed here, not least because of what I wrote above, but also because it confuses/complicates the rule that "vendor directories are totally ignored in module mode" which is particularly important when it to dependencies.
Because vendoring as a "first class citizen" continues to work in GOPATH mode, it's just that in module mode it is not the default.
Yes, I'm speaking about module mode. The point is that "not being the default" for projects that uses vendoring (that is, projects in which developers went through the process of calling go mod vendor and commit the vendor directory) is the wrong default. If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used. People that don't use vendoring don't need to have vendoring explicitly disabled; they simply won't have a vendor directory in their project. By switching the vendoring support off in module mode, go 1.11 is making using vendoring more difficult for no reasons I can understand, without making things simpler for those who don't need or want vendoring.
There was a similar discussion over at https://twitter.com/_myitcv/status/1033824806930534400. Vendoring (as it was and still is in GOPATH mode) is one way of dealing with this problem. As I suggested, having a separate repo as a module download cache is another, which actually have a number of derivative benefits.
Thanks, but that is largely suboptimal. It requires a custom installation step to be repeated for each user, and also handle and commit binary files into the SCM which are harder to handle (especially with git) and review.
As you say, Athens (or indeed anything else that talks the published module protocol) are also solutions, solutions that in some cases try to solve even bigger problems.
I fail to see how Athens solves the needs of people using vendoring. AFAICT Athens helps with the issue that people can remove/sell GitHub repositories to other parties, and this can break builds and/or even cause security problems. This is not what vendoring is after. Vendoring is required in workflows that need 100% of the source code for a project committed under a single SCM, so that it can be fully audited, versioned, and built completely offline, without any dependency on any third-party service or installation. I deliver code to enterprises that want the code to be complete, not depend on third party services, and fully versioned with their existing code tracking tools. Vendoring solves this problem; Athens doesn't seem to.
So for the record, in module mode I am against what is being proposed here, not least because of what I wrote above, but also because it confuses/complicates the rule that "vendor directories are totally ignored in module mode" which is particularly important when it to dependencies.
You seem against adding a feature for people that need it for the sake of orthogonality. If you don't want or need vendoring, there's nothing in vendoring support that creates problems to you. Just use modules, your download cache, Athens or whatever you prefer. You will never have a vendor directory in your repos and nothing will break or be suboptimal for you. But if I need vendoring support and commit a vendor directory, why should vendoring support be disabled by default for me, causing more difficult builds for all involved developers?
If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used.
That's not at all “obvious” to me. In fact, the current structure (in 1.11 module mode) is exactly the opposite: if a developer goes through the steps to commit vendored dependencies in module mode, it could just as easily be because they want to give their users the option to build using those vendored dependencies as an alternative to using modules.
Giving their users an option is not at all the same thing as providing an explicit recommendation.
Moreover, vendor directories can only reasonably work at the topmost level. If a developer makes edits within the vendor folder (intentionally or otherwise), they would effectively perform an implicit replace operation without surfacing that fact in the rest of the UI (such as in the output of go list).
If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used.
That's not at all “obvious” to me. In fact, the current structure (in 1.11 module mode) is exactly the opposite: if a developer goes through the steps to commit vendored dependencies in module mode, it could just as easily be because they want to give their users the option to build using those vendored dependencies as an alternative to using modules.
I'm not sure that we can deduce anything from what Go 1.11. It's pretty obvious that vendoring wasn't even plan of the original module plan, and has been retrofitted at some point later in the process given different pressures. What we ended up with is basically non working for many vendoring use cases, hence this bug (and others).
Rather than discussing hypothesis made by people that don't vendor, I'd rather somebody that does use vendoring in production shows up and say that they absolutely don't want vendored modules to be used by default unless explicitly opted in (like Go 1.11 does), and explain us why with a small experience report. I for sure know this doesn't work for my use case as already explained in mine, and others have already chimed in.
Moreover,
vendordirectories can only reasonably work at the topmost level. If a developer makes edits within thevendorfolder (intentionally or otherwise), they would effectively perform an implicitreplaceoperation without surfacing that fact in the rest of the UI (such as in the output ofgo list).
That's perfectly fine for me at least. In fact, in my use case, vendoring support could go as far as completely subsuming the module download cache and/or go.sum. That is, it would be great that once you opt-in to "vendoring mode", all new modules are automatically expanded into the vendor directory, used by default, removed when they're not used, etc.
I'd rather somebody that does use vendoring in production shows up and say that they absolutely don't want vendored modules to be used by default unless explicitly opted in (like Go 1.11 does), and explain us why with a small experience report.
That's kind of my point, though: you're requesting feedback from one particular subset (package maintainers that choose to use vendoring) and ignoring another (package consumers, who have no choice in the matter in GOPATH mode, but do now have a choice in module mode).
We need experience reports from everyone — including package maintainers and consumers, and including both open-source and enterprise codebases.
@JeremyLoy, could you give some more information about your use-case?
The go tool downloaded all dependencies to
$GOPATH/pkg/mod, despite the fact that the dependencies were vendored
- Did you run
go mod downloadafter cloning the repository? Why or why not? - What is the impact of downloading the dependencies?
- Do you intentionally want to bypass
go.sumverification, or do you have in mind that the vendored dependencies remain unmodified?
- Do you intentionally want to bypass
Until [module proxies become] mainstream, vendoring should be fully supported (but never required).
Vendoring is still supported in Go 1.11, both in GOPATH mode and with -mod=vendor.
I am not aware of any proposal to remove it in 1.12, and would be surprised to see one at this point.
It's pretty obvious that vendoring wasn't even plan of the original module plan, and has been retrofitted at some point later in the process given different pressures.
Please remember that among our gopher values is to “be charitable”.
The approach to vendoring in Go 1.11 does seem to have evolved with the implementation, but that doesn't mean it was “retrofitted”. A flag very similar to -mod=vendor was floated in #19234, and the ability to disable vendoring was discussed in #16562; as far as I am aware, both of those significantly predate the current module design.
@bcmills I'm currently using Go for enterprise application development, so I can give an experience report for that. You'll have to find someone else for the package maintainer report.
We currently vendor our dependencies using dep. We looked into converting to go modules, but found that the differences were too great to warrant a change right now, especially as modules are still in development.
We vendor for a variety of reasons, many of which were already stated here, but I will reiterate them for clarity.
- a single deliverable - the client is able to take the codebase and build the application from source without internet access.
- guaranteed stability - the leftpad problem simply cannot happen with vendoring. It is still an issue with both GOPATH and module development
- security audits - the client is able to audit not just our source code, but our dependencies in a reproducible manner.
we also vendor tools such as gomock, goimports, swagger etc.
Now to answer your questions:
Did you run go mod download after cloning the repository? Why or why not?
no. because with the vendor folder, I should not have to.
Do you intentionally want to bypass go.sum verification
no, quite the opposite in fact. I imagined that go.sum would be a drop-in replacement for dep's Gopkg.lock
do you have in mind that the vendored dependencies remain unmodified?
yes.
Having to set an environment variable or constantly add a flag is a bit of a nuisance, but not a game breaker. The biggest issue to me is that I expected go modules + vendoring to behave exactly like dep with vendoring (algorithmic changes not included)
It's pretty obvious that vendoring wasn't even plan of the original module plan, and has been retrofitted at some point later in the process given different pressures.
Please remember that among our gopher values is to “be charitable”.
The approach to vendoring in Go 1.11 does seem to have evolved with the implementation, but that doesn't mean it was “retrofitted”.
I'm sorry if it was borderline harsh, but anything related to "proposals" in Go often leads to a very frustrating experience, so much that I tend to steer away as much as possible from it and focus on where the community experience is vastly superior (eg: compiler development).
This said, I still think the quoted sentence is factually true. It was clearly stated in Russ' first blogpost:
This proposal keeps the best parts of go get, adds reproducible builds, adopts semantic versioning, eliminates vendoring, deprecates GOPATH in favor of a project-based workflow, and provides for a smooth migration from dep and its predecessors.
and has been repeated other times elsewhere. The vgo design didn't include vendoring; at some point, it was re-added to the picture, but there wasn't any discussion or attempt (at least that I could find) to rethink vendoring in the Go modules world. The fact that a default that doesn't work for people using vendoring was chosen, or that there are large overlaps between the vendor directory and the download cache is probably a consequence of not having fitted vendoring in the picture from day one.
@rasky I think you are right that the initial set of vgo blog posts initially proposed dropping support for vendoring.
However, there was then a fair amount of feedback from the community about the importance of vendoring. As far as I could see, that community feedback was clearly heard by the core Go team, which then resulted in the proposed plan being updated to actually retain support for vendoring with modules. The feedback was discussed in multiple venues, but this might have been the longest thread:
https://groups.google.com/d/msg/golang-dev/FTMScX1fsYk/uEUSjBAHAwAJ
...where within a week or so of the initial vgo blog post, the core Go team had switched to proposing retaining vendoring support with vgo/modules based on that community feedback.
In any event, sorry if my historical comments are not very interesting. A much more interesting question of course is what should the behavior be in 1.12+.
The vgo design didn't include vendoring; at some point, it was re-added to the picture, but there wasn't any discussion or attempt (at least that I could find) to rethink vendoring in the Go modules world.
Yeah, I can see that. Bear in mind that the module support in Go 1.11 is still preliminary: we know that it has some rough edges, and vendoring is one of them.
I think it's important that we figure out a path forward, but it's not clear to me whether that will be to improve vendoring and better integrate it into module mode, or to improve the available module operations and deprecate vendoring entirely. I expect we'll try something in at least one of those directions for 1.12, but it's important that we have enough time to try out the improvements and make sure they work for those use-cases before they become the default.
In particular, because vendoring support in module mode is pretty awkward in 1.11, I think it's important not to try to push it as a default: I don't think it's the right design point yet, and I don't want folks to start relying on a design that really needs more thought.
- a single deliverable - the client is able to take the codebase and build the application from source without internet access.
- guaranteed stability - the leftpad problem simply cannot happen with vendoring. It is still an issue with both GOPATH and module development
- security audits - the client is able to audit not just our source code, but our dependencies in a reproducible manner.
@JeremyLoy has made some great points here why vendoring is really needed in the software we build.
Without vendoring how are you planning to solve these points? Force everyone to host some caching/proxy thingy that would serve the dependencies transparently?
This would most likely compel developers to create various different solutions just like with vendor did before (Godeps, govendor..) and thus split the community once again while still not solve all of the points above.
What if some of the dependencies disappears and you don't have it in the proxy cache or anywhere else? This would force developers to fix/replace the dependency in their repository. However what if this is related to multiple versions with ongoing support? It would have to be fixed for all of them..
I'm really curious about your future solutions without the vendoring support that would satisfy the points above while at the same time not force us to manage extra things outside of the repository to successfully build our Go programs.
Without vendoring how are you planning to solve these points?
That is a matter for a separate issue (perhaps #27618, and perhaps others as well). If you have specific requirements, please contribute an experience report describing both your use-case and how you address it today. (As I've said, it's not clear to me whether the solution will be to improve vendoring or to replace it, but experience reports will help either way.)
The matter at hand for this issue is whether -mod=vendor should be the default, in module mode, when the module's root directory has a vendor subdirectory.
Is there a single experience report, email or comment on GitHub issue that describes a real-world workflow in which the current default is correct? I have not seen any.
@rasky I don't think we have any experience reports for vendoring with module mode at all. As I've said repeatedly, I think it needs work before we can recommend any particular workflow. For now, if the module support in 1.11 doesn't work for you, don't use it. GOPATH mode is still supported and shouldn't get in your way.
If you want to influence the future direction of module mode, you need to share your actual use-case instead of just arguing that what we have today doesn't work: your current criticism is not constructive. We already know that what we have today doesn't work for you: we need to know how to make that better.
@bcmills I added a link to my comments (and by proxy, this issue) to the experience report page you referenced.
If you want me to modify the experience report/submit it in a different way etc just let me know!
My experience with the current vendor behaviour has been as follows:
I've written a (partial) drop in replacement for git in Go, primarily so that I can use it on Plan 9 where there's no access to the real git client. A user requested that I vendor my dependencies so that it can be bootstrapped without @0intro's rc script hack, which seemed like a simple enough enough request so I decided to oblige.
I copied the files to the vendor directory as I did to vendor with previous versions of Go and it didn't work. I ran "go mod vendor" which appeared to generate some ancilliary files for Go 1.11 like vendor/modules.txt. It appeared to work (I was inside my $GOPATH), so I committed and pushed the results. The same user then filed a GitHub issue saying that it was still trying to pull the code that was supposed to be vendored code from bitbucket instead of using the vendored library when doing go build -x. After some investigation, I found the -mod vendor flag and it resolved his problems. (Most of this took place at driusan/dgit#142)
I support this proposal because the current Go 1.11 behaviour:
- Isn't how previous versions of Go worked.
- Isn't what package maintainers expect.
- Isn't what users expect.
- Makes it difficult to build offline, even when all dependencies are vendored.
I think mod=vendor is too coarse: ideally the vendor folder should function more like an implicit set of replace directives, and we should automatically use whatever modules are found there (and fetch the remaining modules as usual).
That's certainly not happening for 1.12, but it's still on my radar.
How about:
Make it possible to specify "vendor" as the module version number string in go.mod. In that case take the module from the "vendor" subdirectory. Otherwise fetch the module as usual.
"go mod init" could add the modules in the "vendor" subdirectory automatically to go.mod with the "vendor" version definition.
That way it would be easy to see from go.mod which module comes from where. No need to look in many places.
@snabb Then you lose all version information and we're back to square one.
I'd just like to comment that I'd be totally opposed to removing the vendor folder. For me, it is absolutely necessary to make sure I still have project dependencies that might be taken offline. The vendor support so far in the new module system is ok but I wish it would copy the entire dependency's repository. I've had to write a tool to do this instead. https://github.com/nomad-software/vend
I think people have different use cases for the "vendor" directory.
In my common use case the packages in the "vendor" directory have local fixes, improvements or other customizations which are not available in the original repository. Therefore the vendored module can not have any version number which would correspond with the versioning in the original repository. I just want anything that is there to be used, overriding any other sources and thus the version number becomes irrelevant.
(I do realize that I might be abusing the vendoring mechanism to implement local modifications to dependencies, but it works well for that even if it was not the original planned use case.)
Added to clarify: This use case is in a corporate environment where the source code and the local modifications to the public packages remain private and are stored in a VCS which does not work with Go tools. (Obviously a different work flow would be preferred with public code on Github.)
@nomad-software Oh, this is great. I did not realize replace can be used for that. The documentation go help go.mod fails to mention it can be used to refer to local file system paths.
I think
mod=vendoris too coarse: ideally thevendorfolder should function more like an implicit set ofreplacedirectives, and we should automatically use whatever modules are found there (and fetch the remaining modules as usual).That's certainly not happening for 1.12, but it's still on my radar.
@bcmills So the vendor/modules.txt should be commit to VCS?
Yes, if you're committing the vendor directory you should definitely include vendor/modules.txt.
go dep only create Gopkg.lock, and other language tool such as maven, composer, npm alse only create one file xxx.lock.
So why go mod create two and what's the difference between vendor/modules.txt and go.sum?
@yanyandenuonuo I am not an expert dep user, but my understanding is that dep uses a manifest file (Gopkg.toml) and creates a lock file (Gopkg.lock) and also populates vendor. E.g., from a list in the dep doc describing the "state" that dep uses:
- A manifest - a file describing the current project's dependency requirements. In dep, this is the Gopkg.toml file.
- A lock - a file containing a transitively-complete, reproducible description of the dependency graph. In dep, this is the Gopkg.lock file.
- The source code of the dependences themselves. In dep's current design, this is the vendor/ directory.
In other words, even with dep, there is a difference between a lock file and vendor.
With Go 1.11 modules, the go.mod files in a build include the dependency requirements as well as provide enough information for 100% reproducible builds (and hence there is not a separate go.lock file or similar in Go modules, and go.sum is not a lock file either).
Go 1.11 modules also provide you with the option of populating vendor. This is not a requirement, but it helps with use cases such as "GitHub is down", a repo disappearing, etc., and it also helps with allowing different projects within the ecosystem to transition to modules at different rates – older versions of Go such as 1.10 understand how to consume a vendor directory created by go mod vendor, so vendoring is one way for a module-based project to provide dependencies to older versions of Go that do not fully understand modules. vendor/modules.txt is an implementation detail of the vendor directory – you normally do not need to examine it and should not manually edit it, but as Bryan said above, you do want to check it in along with the rest of your vendor directory if you are checking in vendor.
These two modules FAQs go into a little more detail, and are probably worth a quick read:
@thepudds Yeah, i had read the FAQS, but there dosn't state more detail about go.sum, and no state about modules.txt.
The wiki say go.sum is not a lock file, but it's state just like a lock file. It's seem like a alias name or format of lock file.
I think go.sum and modules.txt is repeat in effect.
@yanyandenuonuo, the topic of this issue is whether the go command should use the vendor directory by default in module mode. See http://golang.org/wiki/Questions for appropriate venues for general questions about modules and vendoring.
@bcmills If vendor/modules.txt is relate with build, so it's ok that discuss the difference between modules.txt and go.sum. If not, i think both of the file shouldn't be commit to vcs.
FWIW I am in the same boat of maintaining Go projects in a proprietary environment that has limited or no external network access. So I have needed to structure my projects in a self contained way. Before the Go module support, I had to have our custom Go tool for our waf build system handle doing a dynamic GOPATH for the project. We would also recommend using Glide. This allowed for non Go devs to clone to any location and build with confidence from the project with its vendored deps. Since Go 1.11 this has become much more simple when using Go modules + vendoring. But we don't want the build to try and download. So each build has to make sure to set mod=vendor. I am totally onboard with the idea of respecting the project maintainers decision to commit a vendor directory and for that to be the default mode when detected.
but it's not clear to me whether that will be to improve vendoring and better integrate it into module mode, or to improve the available module operations and deprecate vendoring entirely.
I don't see why we would want to remove the option for vendoring when it is clearly a useful approach as suggested by the feedback from vendor users. If a project maintainer wants to use vendoring, then it should be propagated in the same way it is for GOPATH mode. What we have now is inconsistency between GOPATH and module modes.
In particular, because vendoring support in module mode is pretty awkward in 1.11, I think it's important not to try to push it as a default
It is only awkward because the behavior from GOPATH mode to module mode was not consistency retained. If the default mode could be detected as vendor then it is no longer about awkwardly needing to add a build flag. If a project maintainer doesn't want to have vendored deps used, they can remove the directory from the repo. And if a user of that project wants to ensure they always download, they can set the mod flag. Shouldn't we ensure the project owner has options? I see it similar to how a cgo project can choose to set default sane compiler and linker flags for the build, but a user still has the option to manually override those.
I have more feedback from real world experience.
At work we have a monorepo with a single vendor directory. Vendoring everything has been a very stable approach and we're likely to keep doing it for a while, at least until the tooling around modules reaches a similar point of stability.
The main problem we have with our GOPATH-based approach is that the tools we use for managing/updating the vendored dependencies seem to be persistently buggy/hard to use. I looked into replacing these with go mod vendor. My hope was that it would serve a dual purpose of both being more robust and also taking a small step toward in the module direction.
Unfortunately, we would need to use the GOFLAGS hack everywhere (local dev environments, build infrastructure, CI, etc) to ensure that all our go tool commands will use the vendored code. We never want to build the code without using the vendored dependencies.
I also have written/maintained some open-source projects. For library packages, whether or not they've been updated for modules, they never contain vendored code (as has always been best practice), so that's not relevant here.
For applications, either (a) they contain vendored code and expect a GOPATH workflow (no go.mod), or (b) I've updated them for modules and deleted the vendor directory. I have no problem requiring Go 1.11+ for building my applications.
If for some reason I updated an open-source application for modules and used go mod vendor to add a vendor directory, I would want my users to build with that vendored code (same as my work scenario). But I don't anticipate using go mod vendor for my open-source work at all.
I agree with what @JeremyLoy, @rasky, and others in this thread have said. It seems like using vendor for the main module, if it exists, is a much better default.
if a developer goes through the steps to commit vendored dependencies in module mode, it could just as easily be because they want to give their users the option to build using those vendored dependencies as an alternative to using modules.
I have a hard time seeing this practice coming into widespread use (as to justify the current default). It seems like it would be quite error-prone: wouldn't it be easy for the (Go 1.11+/module-using) developers to alter dependencies in the go.mod file (which various go commands will happily do) and then forget to update the vendor directory?
@cespare Please be sure to include your real world experience in experience reports.
@ondrej-fabry my feedback is specific this particular issue; it is not a report on my experience writing Go.
It seems that this issue is still marked as "NeedsDecision". The feedback in this ticket has been overwhelmingly in favor of making this change to be the default. The only dissent I've seen here has come from @bcmills @myitcv. I'm also not seeing anyone indicating agreement with their dissent (either via comment or by virtue of emoji).
@bcmills @rsc what else is needed to make a final decision for what will be better for the entire Go community? How can I help us work towards making a final decision on this issue?
The issue is already milestoned as 1.13; even if it was implemented today, it wouldn't ship to most users for another six months. Is there a particular urgency to getting this done?
@mvdan everyone adopting modules now who wants to keep vendoring is going to need to add this extra bit to their build scripts, after spending the time trying to figure out why they keep downloading the world, to then remove it later assuming the default is changed. Seems like it could become a pretty large distributed time investment, and would be a bit of churn.
Edit: I've seen it come up on Slack quite a few times, and this ticket has been open since August.
Sure. But hurrying to approve and implement this issue this week is not going to affect users any faster. I personally think that endorsing vendoring as it exists today isn't a great idea for modules, so I'd like to see what approaches arise during the 1.13 cycle. In my opinion, getting the correct solution is better than getting a solution fast.
this ticket has been open since August.
Note that there are likely hundreds of modules-related issues at any point nowadays; there's tons of stuff that has been pushed back to 1.13, and I'd argue this is not the most urgent issue of them all. This is precisely why GO111MODULE=on is not a default yet.
As far as I was able to follow, it didn’t seem to me that the core go team was against enabling vendoring by default, but rather there was a bit of a question as to exactly how it should behave when enabled by default, and then of course the time needed to implement.
See this recent comment from @bcmills, for example, which suggests what seems to me to be an improvement to behavior (compared to the current 1.11 mod=vendor behavior):
But of course, different people might have different perspectives as to whether not that is an improvement, which seems like a valid thing to discuss or capture in this issue.
@mvdan I'm not trying to approve and implement this issue this week. I'm looking for a final decision to be made so that the work can start for the Go 1.13 cycle. I'd personally prefer this gets in to 1.12, to short-cut some of the churn, but I don't see that as being a reasonable ask at this point.
I personally think that endorsing vendoring as it exists today isn't a great idea for modules
Can you elaborate further? Why are 100% reproducible builds a bad idea for modules?
I've been on the unlucky end of total loss of an artifact server, and a dependency having been killed in the process. That code was lost and the project could not be compiled. In that case Modules wouldn't have saved me, but vendor would have.
I don't think we should treat vendor as a bunch of replace statements, but instead a way for those who want absolutely reproducible builds to get them. It's my belief it should be a dependency cache that travels with the source repository.
As far as I can tell we're in agreement that we should use vendor more aggressively by default.
However, there are still some details to decide. For example:
- Should we substitute modules, individual packages, or the entire transitive source tree? (Probably modules, but then users will have to be careful to keep
modules.txtup-to-date. Thevendordirectory in the pre-modules world applies to individual packages; on the other hand,-mod=vendortoday applies to the entire source tree.) - Should we modify
go mod vendorto support more targeted vendoring? (Perhaps, because that may make it easier to maintain an accuratemodules.txt.)
FWIW, I'm planning to address this issue in Go 1.13.
@bcmills I think we need to first keep in mind what problem the vendor folder actually solves. In my mind, it supplies one hundred percent reproducible builds, which don't need an internet connection and protects you from online repositories being removed. That's probably as sophisticated as we need to get.
@nomad-software, the vendor folder is not necessary at all for 100% reproducible builds in module mode. You can save the contents of the module cache, and use the saved cache as a GOPROXY. (This was all covered in the preceding discussion.)
The vendor directory does not solve reproducibility; in fact, quite the opposite, since you can make arbitrary changes in the vendor contents that will not be reflected in builds initiated from outside your module. (See #29058 and #27348.)
What the vendor directory does provide is a more ergonomic presentation of the cached code: easier auditing without modifying the repo, simpler diffs for version control tools, etc. Some folks may want that for their entire transitive closure of dependencies, but others likely will not.
@leitzler points out that using vendor by default would change the source code in use for existing modules that have a vendor directory, which actually works against reproducibility for those modules.
That's true, and one of the details I'm contemplating is “when to activate vendor-by-default”: now that go mod tidy adds a go line to the go.mod file, we could trigger the behavior (or not) based on the declared version. For example, perhaps we should only use the vendor directory by default if the go.mod file specifies go 1.13 or later.
the vendor folder is not necessary at all for 100% reproducible builds in module mode. You can save the contents of the module cache, and use the saved cache as a GOPROXY. (This was all covered in the preceding discussion.)
@bcmills This just sounds like more manual vendoring, where you'll now need additional scripts to overlay the module cache and to resolve / save any new additions. Or you now need to have some sort of process to continually check in your module cache to source control. How do you then use that on different developer workstations and build systems? What tooling will go mod provide to help make that easier?
Thinking about it here it seems complex, with potential for pain in the future. It makes me even more convinced that we should keep the more simple vendor concept as the default.
The vendor directory does not solve reproducibility; in fact, quite the opposite, since you can make arbitrary changes in the vendor contents that will not be reflected in builds initiated from outside your module.
If you assume most people won't be just changing stuff in vendor willy-nilly, it doesn't have the opposite effect. It'd especially be true go mod toolkit were to control that folder and overwrite any changes that are made, and to fail of there were any modifications. I never would advocate for mutating vendor, and if you wanna change something there you need to fork it and have your dependency manager drop the fork in to vendor at the right path. I think replace would handle this nicely.
I also assume that vendor will not be considered for things consuming that module, and that generally the only thing using vendor are tests and the build for package main. We don't run tests from external packages, and we can't import their package main, so that seems fine. And with MVS, new consumers should get the same package versions as in their vendor folder, or the version semantically closest to it.
Now, if one of those vendored dependencies disappears, new consumers will fail to build. But the original project being imported will build on its own. And because it's vendored, it gives new consumers a clean way to copy the dependency in to their vendor folder in their repo and make use of it.
Or you now need to have some sort of process to continually check in your module cache to source control.
Yes, but you already need some sort of process to continually check in your dependencies to the vendor directory.
How do you then use that on different developer workstations and build systems?
You provide a GOPROXY that contains all of the modules you need (or, after #26334, one that provides only the modules you're worried about).
It makes me even more convinced that we should keep the more simple
vendorconcept as the default.
To be clear, all of the various vendoring approaches are power-user stuff: for the vast majority of users, version tags and go.sum already provide complete reproducibility, and module proxies can provide availability.
because it's vendored, it gives new consumers a clean way to copy the dependency in to their
vendorfolder in their repo and make use of it.
That is exactly what we don't want to happen. You should only need to store one copy of a given version of a module — not one copy per module you depend one, or one copy per module containing a package main. If a dependency needs to be durable, then it needs to be durable for everyone — not just for folks who happen to have stored it to their vendor directory ahead of time.
Yes, but you already need some sort of process to continually check in your dependencies to the vendor directory.
Ideally that would be go mod, and it's only happening within the scope of the repository on a developer's workstation. It's not needing to manage and merge the cache across different workstations, build/CI systems, etc.
You provide a GOPROXY that contains all of the modules you need (or, after #26334, one that provides only the modules you're worried about).
What if the GOPROXY is down or has suffered from complete data loss? It's more complexity / risk introduced with the build process, where vendor absolves the risk and simplifies it. No need for external network calls.
To be clear, all of the various vendoring approaches are power-user stuff: for the vast majority of users, version tags and go.sum already provide complete reproducibility, and module proxies can provide availability.
I am not aligned with this approaching power-user stuff, because there are non-power users who value what vendoring brings and appreciates the class of risk it mitigates. I'm also not agreeable to separating "reproducibility" in to "reproducible" and "available" If it's not available it's not reproducible.
That is exactly what we don't want to happen. You should only need to store one copy of a given version of a module — not one copy per module you depend one, or one copy per module containing a package main.
@bcmills I agree that you should only store one version of the module. I'm instead talking about the case where the module is removed from the Internet, and it can no longer be retrieved from the original location. The only place to get it from is via existing copies (i.e., forks or vendored versions). Ideally you would create a new repo out of the code "saved" from vendor or another fork, and then use a replace statement to use it, versus just injecting it straight in to vendor.
If a dependency needs to be durable, then it needs to be durable for everyone — not just for folks who happen to have stored it to their vendor directory ahead of time.
Unfortunately, that's just not possible unless the go mod tooling decides to go so far as to depend on and use code from its dependency's vendor directories. My initial gut feeling is 👎 for a few reasons, the largest being the perceived complexity it would introduce.
I feel like vendor provides the best effort at giving you the ability to have completely reproducible builds, and a way to resolve missing dependencies if you absolutely need to make use for something. We've had this happen to us a few times now, and those who used vendoring were saved. If you didn't use vendoring, and didn't have it in your module cache, you're going to need to find a friend with the right module cache.
you're going to need to find a friend with the right module cache.
That is one of several possible roles for the “module mirrors” described in https://blog.golang.org/modules2019, and mirrors seem much more ergonomic than expecting individuals to retrieve and install modules in vendor.
My experience report: I'm using a cloud hosted CI system (CircleCI, but could be Travis or Drone) and I want it to build my monorepo using exactly the same code I have on my workstation without it breaking when an upstream dependency is unavailable or has rewritten git history.
I also expect to be able git bisect my monorepo and reproduce exactly the build I had before at that point in the commit history.
Currently a vendor directory checked into my source control system satisfies this requirement exactly. Originally I managed the vendor directory with govendor and then moved to dep. I expect modules + vendoring to support this model.
mirrors seem much more ergonomic than expecting individuals to retrieve and install modules in vendor.
This I strongly disagree with. What's more ergonomic, this:
$ go mod vendor
or this:
$ docker pull gomods/athens:v0.2.0
$ docker run -p 3000:3000 gomods/athens:v0.2.0
# from another terminal window
$ git clone https://github.com/athens-artifacts/walkthrough.git
$ cd walkthrough
$ GO111MODULE=on GOPROXY=http://localhost:3000 go build
Not shown: the instructions to install and configure docker.
With the former the only thing I need to do to insure reproducible builds is commit the vendor directory, with the latter I also have to:
-
keep the athens proxy up to date
-
figure out where athens stores the actual cache and set up a backup strategy for it
-
hope athens doesn't change its storage format in the next 20 years
Using a mirror provided by someone else is, of course, remarkably easier but it doesn't give me the same level of build reproducibility because:
-
it doesn't work if I'm offline
-
it doesn't work if I'm in a country that happens to be cut off from the mirror for geopolitical reasons
-
it probably won't work in 10 years
I don't know for sure that Athens and the upcoming Google modules mirror won't last 10 years, but let's be realistic, your companies don't have a good track record for keeping services like this online on that timescale. Google Code, for example, lasted 8 years. This morning your company sent me an email reminding me that Google+ is not long for this world; and that, unlike the modules mirror, was a project that your company was fully behind just 5 years ago.
My personal opinion is that people entrusting mirrors with their project reproducibility are making a mistake. I appreciate that you, and others, don't see it this way, let's agree that only time will tell who's right. I hope I can convince you, with this, that there are reasons to prefer vendor directories over mirrors.
For the reasons above, I believe that there should be a simple way to insure that the vendor directory is sufficient to build the project. I don't care whether this solution or #29058 gets implemented, both guarantee fully reproducible builds (#29058 does it automatically, this one does it by making the build fail unless you remember to run go mod vendor) which is good enough for me.
I think points 1 and 2 are resolved by simply replacing git checkout $URL with git checkout $URL && go mod download. The local cache will be deduplicated across multiple projects (no more downloading Gorilla over and over again).
Point 3 is covered by making sure that there exist mirrors being run by multiple independent entities; for instance, I think JFrog recently announced one. And checking that these multiple entities are using some form of mutual replication. (This actually is probably improving the availability of your dependencies over that of your source code repository).
I think points 1 and 2 are resolved by simply replacing git checkout $URL with git checkout $URL && go mod download
Those three points were specifically a criticism of using mirrors for reproducible builds, using go mod download has different issues. Specifically, with go mod download I have to archive and backup the module cache alongside each project and since it's shared between al projects and there isn't a good way to prune it, if I actually start to use it for reproducibility it will have to grow indefinitely.
That is one of several possible roles for the “module mirrors” described in https://blog.golang.org/modules2019, and mirrors seem much more ergonomic than expecting individuals to retrieve and install modules in vendor.
What if you are not connected to the internet, the mirrors are down or the company hosting them just randomly decides to not make them available anymore? A vendor folder solves all these issues.
That is one of several possible roles for the “module mirrors” described in https://blog.golang.org/modules2019, and mirrors seem much more ergonomic than expecting individuals to retrieve and install modules in vendor
We should not forget about all the people that use Go from countries where access to certain websites are restricted.
Such users are going to have an easier time building code that is in their vendor directories than code distributed by mirrors that their governments deem 'inappropriate'
@aarzilli I was suggesting using mirrors and using go mod download, if you needed to solve those 3 given points.
And I don't see why you think growing indefinitely is a unique problem to go mod download, given that vendoring also grows your repository without bound (and likely your disk space too for all commonly used VCSs). Especially given vendor dirs are not shared across projects.
I may not have a complete breadth of knowledge (I've been wrestling with the tools a little), but this is my use & expectation of vendoring.
We have a small monolith repo of related Go projects, with a single vendor directory. We are using go.mod to manage versions. We are vendoring all our 3rd party packages and build tools (go-bindata and gomock at the moment). The goal is to have 100% consistent builds both locally and in our CI environment, as well as 100% reliability (durability against dependencies going away, working offline). It is also my hope that grabbing a shallow git clone will speed up the build process because all required resources will already be present.
The project that I am working on is to build tools to help our internal developers manage K8S / helm deployments. We are a team of 5 at the moment, supporting (guessing) 50 developers.
I'm shocked that -mod=vendor isn't the default behavior. Why would I create a vendor folder and then not want it used? Let's follow the principle of least surprise here, and if the vendor folder exists, assume it's intended to be used.
I'm running against this issue when deploying to Google AppEngine.
I have my vendor/ folder where a couple of modules reside on a private github repository.
This is configured by a github.com host in my .ssh/config.
A manual go build -mod=vendor works, but a go build that AppEngine launches does not use vendor/..
@hazcod, AppEngine supports module mode at this point, so it should not need a vendor directory. (See https://cloud.google.com/blog/products/application-development/go-1-11-is-now-available-on-app-engine.)
@webern, @ssoroka: see http://golang.org/wiki/NoPlusOne. Please keep comments productive.
@bcmills I’m not trying to be rude, but it seems like this entire conversation has fallen on deaf ears.
As noted above, there are valid use cases for having the project be a module, and still use the vendor folder for dependencies.
To completely dismiss @hazcod by saying “just use modules” completely misses the point of this entire thread.
@JeremyLoy, this issue is still open — it has not been dismissed or ignored, it just didn't happen in time for 1.13. However, for the particular use-case @hazcod describes, there is already a better alternative available: he shouldn't need to wait on this issue to be resolved in order to make progress.
That won't work for private dependencies - the AppEngine builder does not necessarily have access to private repositories.
@bcmills: For building locally I used go build -mod=vendor , but for appengine build (which uses go build hardcoded) I had to use a replace github.com/company/module with ./vendor/github.com/company/module in go.mod 🙁
I'm extremely disappointed this wasn't prioritized and will not be in Go 1.13. I was looking forward to unsetting $GOFLAGS and deleting -mod=vendor from CI configs, scripts, Makefiles, etc.
After forcing so much toil on the community with the modules switch, this would've been the tiniest concession to show you cared.
@y0ssar1an (and others), please remember that the Go team are human and there's only so much they can do in one release window. In particular, @bcmills already has too many issues assigned for the release that's just seven weeks away.
Also note that this issue isn't as trivial as it might seem from the outside. Switching a default flag value would be an easy change, but likely not the best. @bcmills has outlined a better plan in #30240, which has more details, and which is milestoned for 1.14.
If you would like to help, the best you can do right now is read that thread and give feedback on whether the plan would work for you, like @rasky did in #30240 (comment).
Modules straight-up should not have landed without this. Forcing everyone to put -mod=vendor on every go build, go install, and go test command, or to set $GOFLAGS everywhere, was never an acceptable solution. I'm sorry to say, but this just reinforces the perception that Go is not a community-driven language.
If the community had a choice, what would they have picked?
- A) Spend thousands of hours updating scripts, Makefiles, Dockerfiles, CI configs, shell configs, etc.
- B) Wait 6 months for modules to be fully baked
To summarize:
- You broke vendoring.
- You did not clearly communicate that you were breaking vendoring. If you were honest about what you were doing, you would've called it Go 1.11 with Modules:tm: Breaks Vendoring Edition:tm:
- Everybody in this thread had to learn the hard way that you broke vendoring.
- An uncountable number of broken builds later, we're told that @bcmills and company are too busy to fix this.
Modules straight-up should not have landed without this.
@y0ssar1an
but modules have not landed. AFAIK, it will land in Go1.14 which is some moons away.
Go modules is behind a feature flag that you can opt into, warts and all
Starting in Go 1.13, module mode will be the default for all development.
Forcing everyone
Of those people using modules (because nobody is forced to use modules; modules will only become the default in 1.14) it is demonstrably the case that not everyone has to use -mod=vendor.
As was stated before, for those people whose workflow has been broken/frustrated by a move to modules, yes, that is very unfortunate, but there are competing priorities here and limited resources.
A large part of the reason the default will only be switched in 1.14 is:
- this issue of "vendoring" has not been fully addressed
- documentation/examples/FAQs/etc are still lacking in places, including this one
Go is not a community-driven language
It is extremely unfortunate at best that that the term "community" continues to be used as a stick by (non-Google) people who feel like they haven't been listened to. Combined with the hyperbole of "forcing everyone", "spending thousands of hours", "wait 6 months for modules to be fully baked": these arguments/points simply become less rather than more convincing to my mind.
Contributing specific, constructive feedback/points is the way forward here. Please, let's all move forward on that basis.
I support the notion that having a vendor folder would look for dependencies there first, with a resulting error message to the developer that "hey, we checked your vendor folder, but your vendor folder is out of date/missing certain dependencies in go.mod file, press Y to update them to your system $GOPATH/src directory or C to update your vendor directory, or N to do nothing. I think this makes it more user friendly, especially to someone who is new to Go.
Also, there does not seem to be a way to share the vendor folder across several packages in the same repository. For example, instead of github.com/username/foo/vendorIBM and github.com/username/bar/vendorIBM containing the same vendor folders, having a vendor folder at the github.com/username/vendor level would provide dependencies for both foo and bar
Also, there does not seem to be a way to share the vendor folder across several packages in the same repository. For example, instead of
github.com/username/foo/vendorIBMandgithub.com/username/bar/vendorIBMcontaining the same vendor folders, having a vendor folder at thegithub.com/username/vendorlevel would provide dependencies for bothfooandbar
The way that we deal with that is to create a mono-repo for any projects that are expected to interoperate and share vendor.
@kliche the "shared vendor folder" you describe sounds just like a module cache, except for the fact that a module cache also know the exact version of you dependency. When you use "Go Modules" this is what you get today already. I can't see why you should add third party dependencies to your repo at all with the existing tooling around modules.
I work on a project that uses a custom source code re-writer to re-write some of the dependencies before using them in a build and vendoring was the easiest way to achieve this. I don't even commit the vendor directory into VCS. It is re-created with go mod vendor before every build. The project has a Makefile that makes it easy to add the -mod=vendor to all relevant commands so it is not as bad a solution but it would have been nicer if everyone could reliably just use go build without having to worry about where to read the dependencies from.
(To elaborate on the re-writer, I use a tool to replace all references to protobuf package with gogoproto allowing me to use gogoproto with dependencies that use protobuf internally)
Agree with the comments about this needing to be a project-wide setting, not a per-user configuration.
Is there a reason that -mod=vendor is not just the default? If people don't want to commit their vendor directory to git, that's their choice. But it seems that there would be no harm in always populating that directory by default (similar to node.js node_modules directory).
I stumbled upon this issue while googling and trying to understand why go build ignores the vendor directory even after adding it. I'm a huge fan of Go's explicitness and for this use case I tend to agree that having an explicit order of where Go looks for dependencies would be great. There is no need for -mod=vendor in my opinion, unless I'm missing something (didn't read the entire thread).
@amacneil @andreygrehov please see the very long thread above your comments; this is not a simple issue by any means. If you want to follow progress, see #27227. Asking the same basic questions asked in the original post again is just making the thread more unbearably long for everyone :)
@mvdan thank you, I re-read this thread and to be fair it still does come across as if the Go team doesn't really understand this use case. However, after some digging through links I came across #33848, where it appears there is a concrete proposal to support automatic vendoring, which is great news. Hopefully others coming across this issue can also follow this new proposal.