cmd/go: allow "go get" when outside a module in module mode
smyrman opened this issue Β· 45 comments
UPDATE: This is a feature request to be able to install a program/main package to a user's home directory (basically $GOBIN) outside of a module context.
Background
While dep, glide and other tools provide a way of fetching project dependencies, part of what they are missing which I think belongs in any official go get with versioning, is a way to install programs (tools, CLI, or a binary compiled from any main package) into a user's PATH (e.g. GOBIN, which may be relevant, even if GOPATH get's deprecated).
Please answer these questions before submitting your issue. Thanks!
What did you do?
$ vgo get github.com/golang/dep/cmd/dep@v0.4.1Example command only; but this would be the closest to how things currently work with go get.
What did you expect to see?
dep binary (in this case) installed to either $GOBIN, $GOPATH[0]/bin or $HOME/go/bin, in that order.
What did you see instead?
$ vgo get github.com/golang/dep/cmd/dep@v0.4.1
cannot determine module root; please create a go.mod file thereSystem details
go version go1.10 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/smyrman/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/smyrman/.local"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/sj/d5g1dfkd3g75ccz83k505v3r0000gn/T/go-build848907380=/tmp/go-build -gno-record-gcc-switches -fno-common"
VGOMODROOT=""
GOROOT/bin/go version: go version go1.10 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.10
uname -v: Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64
ProductName: Mac OS X
ProductVersion: 10.13.3
BuildVersion: 17D102
lldb --version: lldb-900.0.64
Swift-4.0
@smyrman I think this is a genuine concern, but it isn't just about pulling down a specific version of a program; it is using vgo outside the context of a module. You'll get the same error if you omit the versuib @v0.0.0 specifier as well. vgo thinks you are trying to pull in a dependency; you are trying to install a program into GOBIN.
Maybe:
x/vgo: no longer supports installing an executable from outside of a module
?
You'll get the same error if you omit the versuib @v0.0.0 specifier as well. vgo thinks you are trying to pull in a dependency; you are trying to install a program into GOBIN.
@kardianos, thanks for the response. I am actually not concerned about the error message being unclear. This is meant as a feature request to (somehow) still be able to install binaries to a user's PATH, no matter what your current working directory is, similar to what we can with go get today. Even better, the possibility to install a program at as specific version. I believe a language package manger should not be just about dependencies.
I am not to particular about the syntax, but maybe there could be a flag like -home, -user or even -gopath, that allowed installing stuff to either $GOBIN or $GOPATH/bin. This would be similar to how e.g. pip works, for Python, to mention one, where if you use e.g. venev or virtualenv for a per-project experience, you can still add an option to pip to install tools to a user's home:
- https://pip.pypa.io/en/stable/user_guide/#user-installs
- https://docs.python.org/3/install/index.html#alternate-installation-the-user-scheme
While OS package managers are good at doing more or less the same thing (not to a user's home of course, but system wide), there is often a delay for programs to be packaged, and language power users may prefer to use a language package manger for installing his/hers software, while also being able to pull down unreleased software, as we do today with go get. Users operating on a shared system may have a similar need to install things in their home directories.
Further, even though vgo deprecates GOPATH/GOBIN for project development, it's not necessarily right to deprecate GOPATH for other potentially valid use-cases, but that's probably another discussion, where the only thing I am interested in through this issue, is the GOBIN part.
PS! I am also not concerned about this issue being solved right away, but it would be really cool to solve it before vgo goes main-line, if that's what going to happen.
Updated the description.
$ vgo get github.com/golang/dep/cmd/dep@v0.4.1 cannot determine module root; please create a go.mod file there
This clearly must work eventually. The thing I'm not sure about is exactly what this does as far as the version is concerned: does it create a temporary module root and go.mod, do the install, and then throw it away? Probably. But I'm not completely sure, and for now I didn't want to confuse people by making vgo do things outside go.mod trees. Certainly the eventual go command integration has to support this.
Note that installs already happen to $GOBIN or else $GOPATH/bin (so default $HOME/go/bin), not the modue tree.
Will mark NeedsDecision because we still need to work out how to do this, but yes absolutely the command must work.
does it create a temporary module root and go.mod, do the install, and then throw it away? Probably.
This is a reasonable approach, and have some obvious benefits, such as the result of go get github.com/golang/dep/cmd/dep@v0.4.1 not being affected by other tools installed on the system. To make a decision easier, I will supply an alternative approach and highlight some problems with it:
I suppose an alternative would be to have a "default" module that is used when a flag is provided to the go get command.
E.g. one could picture a default module in $GOPATH[0]/src/u (first entry of GOPATH) that is automatically created and used when the -user flag is given to the go get command line. This would let go get -user github.com/golang/dep/cmd/dep@v0.4.1 be semantically equivalent to:
#!/bin/sh
set -e
_module_path=$(echo $GOPATH | awk -F':' '{ print $1}')/src/u
mkdir -p ${_module_path}
cd ${_module_path}
test -f go.mod || echo 'module "u"' > go.mod
go get github.com/golang/dep/cmd/dep@v0.4.1Some benefits
- One easy flag to access a user-specific module for user-installed cross-project binaries.
- The module path is preserved on disk, so power users could sync the folder via tools like Git or Dropbox to keep their tools with them at specific versions across machines.
Major drawback
- Tool installation (e.g. installation of dep at a specific version as used in the example above), may be affected by (minimal) version selection of overlapping dependencies with other tools, leading to potentially weird bugs.
Conclusion
For me, I belive the drawback outweighs any benefit, and a temporary module root / mod.go root is the semantically correct solution; perhaps implementation wise the temporary module root and go.mod file only exists in memory.
This isn't blocking Go 1.11. The main reason being that if GO111MODULE=auto (the default) then we can't reasonably do anything with modules when outside a go.mod tree. That's the opt-in. And inside a go.mod tree this already works.
I agree that this doesn't block Go 1.11, but I do think it should be a blocker for 1.12. There are a lot of repos that have READMEs saying "just run go get github.com/me/myrepo to install". That should continue to work once GO111MODULE changes to "on".
Agreed. I've marked it as a release-blocker for 1.12.
IMO go get with modules semantic is not the right place for the functionality.
go bin may be?
does it create a temporary module root and go.mod, do the install, and then throw it away?
This does not sound good unless the version is embedded in the executable somehow. Say an install is done using go get path/to/cmd/exe and the modules used to build that executable. Then a buggy behaviour is found. How does the user determine the version being run at a later stage? This problem already exists to an extent since sourc version are not currently embedded in artifacts, but currently at least the user can do forensics because the source is left lying around.
@kortschak, that's at least in part covered by #26404. It would be up to the programmer to expose that via a flag, command or similar.
I've been giving this some more thought, and by now I think the right behavior β if we're in module mode and not inside any other module β is to use the go.mod file from the module containing the requested package.
In that mode, we should emit an explicit error if the go.mod file has any replace directives that point to (relative or absolute) file paths outside of that module, or if the process of building the package required any additions to the go.mod file (i.e., as if the user had implicitly set -mod=readonly).
That has several nice properties:
- It applies the same versions, including replacements and exclusions, that the tool author (presumably) used when testing their binary and its dependencies.
- It uses an existing artifact β the
go.modfile β that doesn't disappear after the build completes. - The versions used are not specific to the individual user, so (modulo other
GOFLAGS) builds remain highly reproducible.
I can see one downside to this approach:
- Versions of packages that have dependencies beyond the standard library but lack
go.modfiles cannot be built outside a module in module mode.
I think that may actually be a good thing: if you want to build a tool that doesn't specify the versions for its dependencies, you need to specify those versions, and the logical place to do that is in a module definition.
@bcmills Just expanding slightly on the downside you listed, I think that means that go get tool-foo when run outside of a go.mod tree would not work in module mode for a tool that itself has not opted in to modules?
If so, that might be problematic as 1.12 behavior, or whenever module-mode becomes the default behavior of the go tooling. Ideally go get tool-foo would would continue to work for tools that have not updated their readmes, including for tools that have not yet opted in to modules (given it will take some time for the ecosystem to fully adopt modules).
Perhaps an alternative would be to do as you outlined for a module-based tool, but liberalize it such that a non-module based tool would also work via go get tool-foo in module-aware mode when run outside of a module (e.g., perhaps by following 1.10 behavior, or perhaps by effectively creating a temporary go.mod as outlined in #24250 (comment), perhaps by having a 'catch all' per-user go.mod file that picks up requirements when in module-mode outside of a go.mod tree, or ___).
@bcmills We are using replace directives in main modules pointing to submodules in the same repository, so that changes to submodules can immediately affect the main module and be tested without needing an intermediate step to publish a new module version. When you say
we should emit an explicit error if the
go.modfile has anyreplacedirectives that point to (relative or absolute) file paths outside of that module
do you mean the repo? What does a replace directive pointing "outside the module" mean?
do you mean the repo? What does a replace directive pointing "outside the module" mean?
I think I mean the module itself, since that's what module proxies will have available. βOutside the moduleβ means either above the directory containing the module's go.mod file, or at or below the directory of a submodule's go.mod file.
We are using replace directives in main modules pointing to submodules in the same repository, so that changes to submodules can immediately affect the main module and be tested without needing an intermediate step to publish a new module version.
But presumably this is just an intermediate, transition step? Because then the "released" version of a command/library should not have any directory-target replace directives, thereby indicating it was tested against specific versions of all its dependencies.
By the way, I can empathise with the pain of the multiple steps involved. I added an Experience Report on just this situation (actually with the added complication of a cyclic module dependency):
https://gist.github.com/myitcv/79c3f12372e13b0cbbdf0411c8c46fd5
I think that means that
go get tool-foowhen run outside of ago.modtree would not work in module mode for a tool that itself has not opted in to modules?
Yes.
Ideally
go get tool-foowould would continue to work for tools that have not updated their readmes, including for tools that have not yet opted in to modules (given it will take some time for the ecosystem to fully adopt modules).
That's true. Perhaps a fallback per-user go.mod would work for that, but I don't think we should use a fallback if we have the binary's actual go.mod file available.
@myitcv so far, we have been tagging releases with the replace directives, and fetching the repository or tarball has been a necessary step to build a release (I realize this doesn't play well with module proxies who don't operate on repos). It could be possible to remove those after we branch off to make a release and leave the replaces on our master branch.
For the case where you in module mode but outside a module and do go get example.com/tool-foo, and tool-foo lacks a go.mod file, the go tooling could automatically create something like:
$GOPATH/pkg/mod/....something.../example.com/tool-foo/go.mod
That would track dependencies and provide reproducible builds via that go.mod without cross contamination from other globally installed tools.
This is separate case from whatever is done for a tool that itself has a go.mod.
I've told a couple people that today with Go 1.11 they can manually create something like ~/tools/gorename/go.mod or ~/tools/goimports/go.mod and then do a go get or go install for the proper tool from within the corresponding directory, but approach outlined above would be automating that process and putting the go.mod into some canonical location (and given no human in the loop, avoid collisions by including the full import path in the directory structure).
If so, that might be problematic as 1.12 behavior, or whenever module-mode becomes the default behavior of the go tooling. Ideally
go get tool-foowould would continue to work for tools that have not updated their readmes, including for tools that have not yet opted in to modules (given it will take some time for the ecosystem to fully adopt modules).
I think that this is essential behavior for any future 'module only' world, whether it is in 1.12 or later. The reality of life is that there are many people who will not immediately add go.mod files to their programs for various reasons and some number of programs that will never add it because they are no longer being actively developed. A world in which go get <program> stops working unless the program has been updated to have a go.mod file is a world where a great many Go programs are no longer conveniently installable, probably for a significant amount of time. People are unlikely to be happy with this.
In my view, retaining GOPATH for at least this usage is the simplest answer, but people may not like it. Otherwise, the best solution I can think of is to create a temporary go.mod file for the package on the fly.
That would track dependencies and provide reproducible builds via that
go.modwithout cross contamination from other globally installed tools.
Maybe, although go get -u in non-module mode today is a nice forcing function to pick up at least some updates, and if there isn't any go.mod for the requested package, then maybe using other information gleaned from the community about what works isn't really a bad thing.
I could imagine, say, a $HOME/.go.mod file used as a fallback, where any go get or go install outside a module updates $HOME/.go.mod with indirect dependencies. My fear then is that it might be too convenient: we probably do want to retain some incentive for folks to add go.mod files to individual projects, and if we make it too easy to work without them we lose some of the broader benefits of reproducibility.
I've been giving this some more thought, and by now I think the right behavior β if we're in module mode and not inside any other module β is to use the go.mod file from the module containing the requested package.
I've also been thinking about it these days and came to the same conclusion. We have all we need in the go.mod of the module we're building.
I've got only one thing to think about.
If we are building an exteranal binary with go get inside some current module - the current module's dependencies could affect the build (implicitly set other versions than are set in the go.mod of the binary we're installing).
I don't think it's a good behaviour if we are driving for high-fidelity builds.
I would vote for a more predictable behaviour. Something like "if we run go get on a main-package having its own go.mod - we shouldn't use the dependencies from current go.mod (if having one)".
If we apply this rule, all binaries installed with go get will have a predictable build list, affected only by their own module requirements.
If we are building an [external] binary with
go getinside some current module - the current module's dependencies could affect the build [β¦].I don't think it's a good behaviour if we are driving for high-fidelity builds.
It may reduce fidelity somewhat, but seems like a desirable and perhaps even necessary behavior (see #25922). For example, suppose that one of the tools you're using calls out to a parser for some language (perhaps Go itself): if you run into a bug in the parser, you may want to rebuild the tool with an updated parser without waiting to push that update upstream.
You're absolutely right, it seems we just need a support for both cases.
What do you think about adding some build flag to switch the behaviour?
Maybe something like -mod=standalone, telling go build to use the original go.mod and nothing more.
Would it make sense to instead support something analogous to npx from the npm ecosystem? I.e. go mod run <modulespec> <package>? Like other module commands, it would transparently cache the relevant module, and grab dependencies as needed. It would only work for commands contained within modules.
If this seems valuable, I can write up a fuller proposal separate from this issue.
@punya the go tool already has go run pkg which does exactly that in a module context. In a global context it probably make sense to ensure that we retain this behaviour (although I don't think the multiple version support is necessary - major versions, yes, via the import path /vX, but not minor/patch versions).
However, the main issue with using go run is #25416. Although there is a clear way to sidestep that: #25416 (comment)
Thanks for the pointer. Russ's comment clearly goes against the substance of what I proposed so I'll hold off.
Just as a reminder when we come to look at this issue that we will need to bear in mind #26794
Would it make sense to re-title this issue? Some people read this current issue title:
"cmd/go: allow installing programs (at specific versions) to GOBIN"
...as meaning that this issue is just about making go get some/command@v1.2.3 work (in other words, that this issue is about allowing versions for go get with tools). However, the discussion in this issue is broader than that (including in terms of the other GitHub issues that are being closed as duplicates of this issue here).
I'm a go noobie (coming in at an excellent time with the introduction of go modules) and I wanted to throw in my 2Β’ of my use case.
First
I would expect either success (installed bin) or an error message when from my home directory I do this:
go install github.com/UnnoTed/fileb0xor this:
go get github.com/UnnoTed/fileb0x
Instead (go1.11 darwin/amd64) I get no error and, even worse, echo $? yields 0.
Second
I don't have a $GOBIN. I would expect that to be... dirname $(command -v go) if not otherwise specified?
Third
The purpose of fileb0x is to be used by go generate, which is why I was trying to install it following the readme.
For anyone else in a similar use case finding this via search, I did find that adding go run worked for me. However, it added everything to my go.mod. I'm not sure if that's desirable or undesirable. Personally I appreciate it.
//go:generate go run github.com/UnnoTed/fileb0x b0x.yaml
Thanks @davecheney. Since I'm using go1.11 I don't have a GOPATH either. I do see that the bins were put in $HOME/go/bin/. I'll set GOBIN=$(dirname $(command -v go)).
Interestingly, I actually don't have command -v go in my user PATH variables. It appears that go installed itself sneakily into /etc/paths.d/go when I wasn't looking (probably the mac .pkg installer)
P.S. I'd like to recommend that go install have an additional output of Installed xyz to /some/path/bin/thing. That would be useful also in cases where I happed to be in an environment that would install in places that I didn't expect due to my own forgetfulness (perhaps something like gvm or a folder with a go.mod that causes ambiguity), as well as the case above.
@coolaj86 you can use go list to find the target of go install:
# setup a fresh GOPATH for testing's sake
export GOPATH=$(mktemp -d)
# create a module for demo purposes
cd $(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello
# make this a main package
cat <<EOD > main.go
package main
EOD
# look at the "default" Target
go list -f "{{.Target}}" .
# now compare with GOBIN set
export GOBIN=$PWD/.bin
go list -f "{{.Target}}" .
For more details see go help list.
For anyone else in a similar use case finding this via search, I did find that adding go run worked for me. However, it added everything to my go.mod. I'm not sure if that's desirable or undesirable. Personally I appreciate it.
This falls under the umbrella of #25922 and #27653.
Since I'm using go1.11 I don't have a GOPATH either. I do see that the bins were put in
$HOME/go/bin/. I'll set GOBIN=$ (dirname $(command -v go)).
Perhaps GOROOT would be more helpful? For more details see go help environment. go env is your friend too.
I think that GOPATH GOBIN is "The Right Place" β’.
I will (except in rare circumstances) have write access to my GOBIN and GOPATH. I may or may not have write access to my GOBIN GOROOT (i.e. if I install via apt install go-1.11, heaven forbid).
However, I still am a firm believer that go install should print the install path.
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
...
Readability counts.
...
Although practicality beats purity.
...
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
...
If the implementation is hard to explain, it's a bad idea.
Printing the path is beautiful, explicit, obvious, simple, readable, practical, does not require guessing, not difficult to explain over the phone and remember later.
For example, I had to search my email to find the go list command you mentioned, and I had to try it three times to get it right.
- 1st time I used the '.', which was obviously wrong, but I just copy-pasted
- 2nd time I had the
"in the wrong place, including the package - 3rd time the quote was correct, as was my guess that
.should be replaced with the qualified name.
I'm assuming there are temporary variables that could be set in the environment (i.e. when using gvm or other tools) that could lead to the package installing into an alternate place that would become non-reproducible if I were to run the go list command at a later time or in another terminal.
I will (except in rare circumstances) have write access to my
GOPATH. I may or may not have write access to myGOBIN(i.e. if I install viaapt install go-1.11, heaven forbid).
I think you're confusing GOBIN with GOROOT?
You should always have write access to GOBIN, because that's where go get puts binaries normally.
Probably
s/GOBIN/GOROOT/g
s/GOPATH/GOBIN/g
As something of an experiment in this space, would appreciate feedback etc on https://github.com/myitcv/gobin
@bcmills is something going to happen here for the Go 1.12 freeze?
Not being able to use go get in the "fetch/update a tool globally" sense has been the biggest frustration for us with the module changes in Go 1.11. We need a command we can run from anywhere that updates tools, now that go get has been broken for that purpose.
I'm not sure I totally follow the entire lengthy discussion here, but it sounds like even with this change we would still need to do cd /somewhere/outside/of/a/module && go get blah.
I'm not sure I totally follow the entire lengthy discussion here, but it sounds like even with this change we would still need to do cd
/somewhere/outside/of/a/module && go get blah.
Well, I really hope the final solution will include a flag as suggested here:
Or that it uses a different command entirely... Or that go modules (as it's still an experiement), is in fact changed to use go mod add if you want to download something not a tool, but for the current module...
Change https://golang.org/cl/148517 mentions this issue: cmd/go: enable module mode without a main module when GO111MODULE=on
Regarding @cespare's comment
I'm not sure I totally follow the entire lengthy discussion here, but it sounds like even with this change we would still need to do
cd /somewhere/outside/of/a/module && go get blah.
This seems to be a major missing feature. Its nice things work concurrently now but sometimes you're in your project directory and want to download a tool. What should I do in this case? Just cd to a tmp dir?
Nvm.
To anyone wondering about my question in the future: https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md