cmd/go: drop $GOPATH/pkg
bradfitz opened this issue · 43 comments
(after discussion with Russ, trying to summarize here) It's nice to be able to tell people to set $GOPATH == $HOME, which means their source goes into $HOME/src and their binaries conveniently go into $HOME/bin. The only wart with $GOPATH == $HOME is the $HOME/pkg directory, which is pretty ugly, not useful for end users, and somewhat offensive to be littering in people's $HOME. If issue #4443 and issue #3895 get fixed, $GOPATH/pkg might get even uglier. Unlike $GOROOT/pkg, $GOPATH/pkg doesn't contain tool binaries. It's pretty much just a cache. Can't cmd/go's *.a cache go into /tmp/go-cache-$USER 0700 or ~/Library/Caches with appropriate occasional lazy cleaning? (or letting the system clean).
FYI, for binary-only package to work, you will need to create a dummy "package xxx" Go source file in the directory with an older timestamp than the .a file. This is issue #2775.
Every release (especially Go 1.5) has been getting us closer to eliminating "pkg" and instead having a build cache directory somewhere (some system-specific cache directory)
Having a cache instead of "pkg" would also eliminate the frequent problems people run into where "go build" does too much work when they could've used "go install" instead.
/cc @adg @davecheney
I would love see the pkg directory go away.
I've always viewed it as an artifact of our implementation which is based on historic ideas of how a program is compiled into object files which are then linked. (In the earliest days of Go I suggested that we make this compile/link process invisible - i.e., as a programmer one just "sees" source code. But we were not ready at that time for this).
I view 'pkg' as this cache, and I'm fine with it living inside GOPATH
because that makes it simple to explain where to find it if people are
determined to delete it.
The change I implemented in gb was to remove the distinction between go
build, go install, and the other odd -I variants; if gb can cache
something, even while building tests, it will and this has worked out
pretty well.
For my money removing the number of ways one can tell them go tool to cache
things and just make the caching more pervasive would be a welcome
improvement.
If that cache lives inside GOPATH, or some default location relative to
HOME feels less important, but I suspect that the windows users will have a
issues with a large number of files written to their home directories,
which may not be local to their machines.
Thanks
Dave
On Wed, 30 Sep 2015 03:17 Robert Griesemer notifications@github.com wrote:
I would love see the pkg directory go away.
I've always viewed it as an artifact of our implementation which is based
on historic ideas of how a program is compiled into object files which are
then linked. (In the earliest days of Go I suggested that we make this
compile/link process invisible - i.e., as a programmer one just "sees"
source code. But we were not ready at that time for this).—
Reply to this email directly or view it on GitHub
#4719 (comment).
makes it simple to explain where to find it if people are
determined to delete it
If the cache is 100% correct (which isn't unimaginable; bazel does it, and Go 1.5 is basically there already), then deleting it is unnecessary. If we really wanted, we could have a go XXX subcommand to free up disk space, if that's the concern.
Great to hear that gb does caching for both build and install!
If we get to the point where the current "pkg" is just a cache and also cached go build, that's great, but then my next annoyance is that I have to look at it. I like to set my $GOPATH to $HOME, and that means I have $HOME/pkg being a useless turd right in my home directory. So I'd rather put things in the right spots:
- For Linux etc we have stuff like http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html for the user-specific cache directory.
- OS X has $HOME/Library/Caches
- Windows has %USERPROFILE%\AppData\Local\Temp (aka $TMP aka $TEMP) (from http://technet.microsoft.com/en-us/library/cc749104(v=ws.10).aspx
I cannot explain why people want to delete GOPATH/pkg, I believe it is
unnecessary, and in every case where I have asked for more details not have
been provided, yet the lore continues to propagate itself.
I'm fine with using the LSB mandates tmp directories, that feels like the
smaller argument vs making the go tool more aggressive with its caching.
On Wed, 30 Sep 2015 07:50 Brad Fitzpatrick notifications@github.com wrote:
makes it simple to explain where to find it if people are
determined to delete itIf the cache is 100% correct (which isn't unimaginable; bazel does it, and
Go 1.5 is basically there already), then deleting it is unnecessary. If we
really wanted, we could have a go XXX subcommand to free up disk space,
if that's the concern.Great to hear that gb does caching for both build and install!
If we get to the point where the current "pkg" is just a cache and also
cached go build, that's great, but then my next annoyance is that I have
to look at it. I like to set my $GOPATH to $HOME, and that means I have
$HOME/pkg being a useless turd right in my home directory. So I'd rather
put things in the right spots:
- For Linux etc we have stuff like
http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
for the user-specific cache directory.- OS X has $HOME/Library/Caches
- Windows has %USERPROFILE%\AppData\Local\Temp (aka $TMP aka $TEMP)
(from http://technet.microsoft.com/en-us/library/cc749104(v=ws.10).aspx—
Reply to this email directly or view it on GitHub
#4719 (comment).
If pkg is to become a cache, and possibly more hidden, it would be good to fix the ways in which GOOS + GOARCH is not quite a valid cache key: GOARM is probably the worst offender here, but also tags and things like buildmodes and the race detector (although the go tool handles the last two via installstuffix currently).
Forgive me if I have missed subtleties that have already been discussed.
I have not seen any discussion of how a system global pkg cache would work for people who switch between multiple GOPATHs, each of which might contain different versions of the same package(s). I understand that recent versions of the go tool will hash all the file names used to build a .a file to help keep the cache accurate. But different versions of a package in different GOPATHs could easily have the same file list but different code. It seems to me either separate caches are required per GOPATH (as the pkg directory does today) or the cache key would need to include something to differentiate between the multiple GOPATHs used.
Yes, it would have to be a good cache. :-)
(That is, it would have to take all these things into consideration. It's not a trivial problem, which is why it hasn't happened.)
+1 to this, particularly making it toolchain-sensitive too. I often find myself switching between different compiler versions so I can check behaviour differences, and switching/removing $GOPATH/pkg is a pain.
It would be nice if binary-only packages (#2775) still live in $GOPATH/pkg, even if the remaining files where to move to OS specific cache directories.
Change https://golang.org/cl/75473 mentions this issue: cmd/go: cache built packages
Change https://golang.org/cl/75850 mentions this issue: cmd/go: do not install dependencies during "go install"
I've been mentally translating this bug to "remove pkg from both $GOROOT and $GOPATH", which is difficult, but @crawshaw pointed out to me earlier today that it's only about $GOPATH. $GOROOT/pkg is difficult to kill because we want to keep allowing the building of programs with cgo-enabled package net from systems without a C compiler. We will probably have $GOROOT/pkg or an equivalent for quite some time. But $GOPATH/pkg doesn't have that concern.
Assuming build artifact caching goes well in Go 1.10, the only thing left in the way of removing $GOPATH/pkg would be updating source code analysis tools that want access to the .a files to know where to find them. We're working out how to do that, because we can see that proper package management is not going to fit into the usual $GOPATH/pkg structure either. I expect the answer will be in Go 1.11 and that we can use that release as a grace period in which old tools still work but people should convert to new tools that also work. If that happens, then we could possibly consider dropping $GOPATH/pkg in Go 1.12.
Tentatively putting in Go 1.12. Note that this means disallowing GOCACHE=off.
Note that this means disallowing GOCACHE=off
Just to double-check, and my use-case might not be the norm, but does this mean having no way whatsoever to disable caching, outside of deleting the cache itself?
Reason I ask is that I sometimes am working with randomly-failing integration tests that I want to run in a loop to have it trigger the error, since it doesn't trigger every time.
Right now, when working with such projects, I always export GOCACHE=off before I start testing, because I know one successful run won't mean the test works the second time, and so I want to keep triggering those random failures so that I can debug and squash them.
Again, this might be an edge-case, and I can change those loops to remove the cache before running, but I just wanted to put this use-case out here for consideration.
I came here from #17271 (comment) and I am interested if dropping $GOPATH/pkg will allow me to have my dependency in vendor/dependency checkout without touching $GOPATH at all?
@abitrolly, the bug you were at was the right place. This bug is more of a cleanup that's not very user-visible.
Now that the previous bug is closed, does that mean that the feature with vendor imports without $GOPATH works already in 1.12?
@abitrolly minimal vendor support without GOPATH is being added as part of new modules feature to Go.11. Note Go.1.11 is not released yet. You can try the beta version.
Change https://golang.org/cl/126755 mentions this issue: cmd/go: move module cache from $GOPATH/src/mod to $GOPATH/pkg/mod
Change https://golang.org/cl/129679 mentions this issue: doc/code: drop mentions of GOPATH/pkg directory
shouldn't the module cache be in os.UserCacheDir() instead of $GOPATH/pkg/mod?
@perillo, we discussed that (somewhere) earlier and decided to just use $GOPATH/pkg for now while we had it. But yes, it'd have to move later when this bug is resolved.
@perillo, or we might leave the module source cache in $GOPATH/pkg.
I think there are good reasons not to put cached module downloads in $HOME/.cache. Quoting myself on https://dave.cheney.net/2018/07/14/taking-go-modules-for-a-spin and https://groups.google.com/forum/#!msg/golang-dev/RjSj4bGSmsw/KMHhU8fmAwAJ, back when the module cache was $GOPATH/src/mod not $GOPATH/pkg/mod:
The build cache ($GOCACHE, defaulting to $HOME/.cache/go-build) is for storing recent compilation results, so that if you need to do that exact compilation again, you can just reuse the file. The build cache holds entries that are like “if you run this exact compiler on these exact inputs. this is the output you’d get.” If the answer is not in the cache, your build uses a little more CPU to run the compiler instead of reusing the output. But you are guaranteed to be able to run the compiler instead, since you have the exact inputs and the compiler binary (or else you couldn’t even look up the answer in the cache).
The module cache ($GOPATH/src/mod, defaulting to $HOME/go/src/mod) is for storing downloaded source code, so that every build does not redownload the same code and does not require the network or the original code to be available. The module cache holds entries that are like “if you need to download mymodule@v1.2.3, here are the files you’d get.” If the answer is not in the cache, you have to go out to the network. Maybe you don’t have a network right now. Maybe the code has been deleted. It’s not anywhere near guaranteed that you can redownload the sources and also get the same result. Hopefully you can, but it’s not an absolute certainty like for the build cache. (The go.sum file will detect if you get a different answer on re-download, but knowing you got the wrong bits doesn’t help you make progress on actually building your code. Also these paths end up in file-line information in binaries, so they show up in stack traces, and the like and feed into tools like text editors or debuggers that don’t necessarily know how to trigger the right cache refresh.)
I think that's still basically right. If the module cache is the only thing left in $GOPATH/pkg, we could decide to move it elsewhere, or we could decide to leave it there. Moving it to $HOME/.cache seems wrong though.
Really I'm just restating what I said in b8f42d7 on Jul 30, 2018 above:
If we do eliminate GOPATH/pkg for holding .a files (#4719)
then we could still keep it around for pkg/mod.
(Or we could move the module cache again then.)
@rsc I think I understand and agree with all those points individually, but I'm missing the link between them and being opposed to using $XDG_CACHE_HOME (default $HOME/.cache) or its equivalent on other platforms.
My motivation for wanting to use a platform's standard cache location is that these caches tend to grow and never shrink, and eventually it becomes necessary to reclaim some disk space. Having a single location where all "well behaved" software stores caches that are safe to delete makes it more convenient to do that kind of maintenance.
Without standards like the XDG base directory spec, most software tends to think it deserves its own entry in your ls -a ~ output. But the more software you use, the more of a mess that becomes.
I suspect we have a different level of concern around this point:
It’s not anywhere near guaranteed that you can redownload the sources and also get the same result.
I don't think ~/.cache is meant to be limited to things that could be recomputed from locally available non-cache data. Looking inside mine, I see multiple browser caches, yarn module cache, typescript typing cache, a system package manager cache (yay), etc. All of those are caching data from the internet.
Looking at the XDG data dir spec, if a distinction must be made between the durability of the two go caches, maybe $XDG_DATA_HOME ($HOME/.local/share) should be considered.
I do get rsc's point, some of Go cache is "cheap to discard", some is "expensive to discard and potentially unavailable if lost". Note that XDG says nothing about how $XDG_CACHE_HOME should or can be cleaned, or how often etc, all it says is the directory is for "non-essential (cached) data". So I guess you could argue that any automatic tool that removes ~/.cache/go/mod or whatever in way that annoys the user is just too eager, and should learn better rules about prioritization and retainment. Maybe.
For what it's worth, rust seems to plan migrating from ~/.cargo to ~/.cache/cargo for what seem to me to be similar uses: rust-lang/cargo#148 (don't trust Github claims of "Merged").
The other thing a single cache directory location is for is a simple place to exclude things from backups.
Why will we want a long cache when there will be global modules proxies ?
I think if we want to be really safe we need a cache per project (instead of vendor ?). I'd like this cache hardlinked on linux.
This issue was originally about the .a files in $GOPATH/pkg.
Those are no longer generated when using the build cache,
and as of Go 1.12, use of the build cache is required.
So it has been quite a long time since we wrote .a files to $GOPATH/pkg.
(The build cache also fixed issue #4443 and #3895 that Brad mentioned in the issue text.)
Since the issue was written, we added $GOPATH/pkg/mod and $GOPATH/pkg/sumdb.
I am comfortable with those staying as they are today.
They don't have any of the problems that the .a files had.
Now that GOPATH defaults to $HOME/go,
$HOME/go/pkg/mod and $HOME/go/pkg/sumdb
are completely reasonable paths.
(I still prefer to set GOPATH=$HOME, but that's a choice not a default.)
Closing this issue as fixed by the now-required build cache.