cmd/go: go get $module@$commit should resolve it to the highest appropriate pseudo-version

Question

cmd/go: go get $module@$commit should resolve it to the highest appropriate pseudo-version

rogpeppe opened this issue 7 years ago · 35 comments

go version go1.11rc2 linux/amd64

With my existing $GOPATH, I can run go get to get a particular commit, and it doesn't use the correct version as I'd expect from https://go-review.googlesource.com/c/go/+/124515, but downgrades instead. With a fresh $GOPATH, it works OK, so I suspect that somehow the cached info isn't being updated appropriately.

Here's what I did:

% cd /tmp
% mkdir modtest
% cd modtest
% go mod init example.com/foo/bar
go: creating new go.mod: module example.com/foo/bar
% cat > main.go
package main
import _ "github.com/frankban/quicktest"

func main() {
}
% go mod tidy
% go list -m all | grep cmp
github.com/google/go-cmp v0.2.0
% go get -x github.com/google/go-cmp@5411ab924f9ffa6566244a9e504bc347edacffd3
# /home/rog/src/go/pkg/mod/cache/vcs/bf8dbe084a71342417894c38c6144465bdc383bd4eb51954ad4212efcf8a7445 for git2 https://github.com/frankban/quicktest
cd /home/rog/src/go/pkg/mod/cache/vcs/bf8dbe084a71342417894c38c6144465bdc383bd4eb51954ad4212efcf8a7445; git ls-remote -q https://github.com/frankban/quicktest
1.303s # cd /home/rog/src/go/pkg/mod/cache/vcs/bf8dbe084a71342417894c38c6144465bdc383bd4eb51954ad4212efcf8a7445; git ls-remote -q https://github.com/frankban/quicktest
# /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e for git2 https://github.com/google/go-cmp
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git ls-remote -q https://github.com/google/go-cmp
1.243s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git ls-remote -q https://github.com/google/go-cmp
% cat go.mod
module example.com/foo/bar

require (
	github.com/frankban/quicktest v0.8.0
	github.com/google/go-cmp v0.0.0-20180328201512-5411ab924f9f // indirect
	github.com/kr/pretty v0.1.0 // indirect
)
%

Note that despite getting the latest master commit of go-cmp, it has downgraded the quicktest package inappropriately, and at no point did it run the git describe command which would have told it which version to use. If I use @master, it works correctly:

% GOPATH=$h/src/go
% echo module example.com/foo/bar > go.mod
% go mod tidy
% cat go.mod
module example.com/foo/bar

require github.com/frankban/quicktest v1.1.0
% go get -x github.com/google/go-cmp@master
# /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e for git2 https://github.com/google/go-cmp
go: finding github.com/google/go-cmp master
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git tag -l
0.004s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git tag -l
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git ls-remote -q https://github.com/google/go-cmp
1.300s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git ls-remote -q https://github.com/google/go-cmp
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
0.004s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
0.005s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
0.007s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git describe --first-parent --always --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' --tags 5411ab924f9ffa6566244a9e504bc347edacffd3
0.007s # cd /home/rog/src/go/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git describe --first-parent --always --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' --tags 5411ab924f9ffa6566244a9e504bc347edacffd3
% cat go.mod
module example.com/foo/bar

require (
	github.com/frankban/quicktest v1.1.0
	github.com/google/go-cmp v0.2.1-0.20180328201512-5411ab924f9f // indirect
)

If I use a fresh $GOPATH, it also works OK:

% echo module example.com/foo/bar > go.mod
% go mod tidy
go: finding github.com/frankban/quicktest v1.1.0
go: downloading github.com/frankban/quicktest v1.1.0
go: finding github.com/google/go-cmp v0.2.0
go: finding github.com/kr/pretty v0.1.0
go: finding github.com/kr/text v0.1.0
go: finding github.com/kr/pty v1.1.1
go: downloading github.com/google/go-cmp v0.2.0
go: downloading github.com/kr/pretty v0.1.0
go: downloading github.com/kr/text v0.1.0
% cat go.mod
module example.com/foo/bar

require github.com/frankban/quicktest v1.1.0
% go list -m all | grep cmp
github.com/google/go-cmp v0.2.0
% go get -x github.com/google/go-cmp@5411ab924f9ffa6566244a9e504bc347edacffd3
# /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e for git2 https://github.com/google/go-cmp
go: finding github.com/google/go-cmp 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
0.003s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git tag -l
0.002s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git tag -l
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git ls-remote -q https://github.com/google/go-cmp
1.328s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git ls-remote -q https://github.com/google/go-cmp
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git fetch -f --depth=1 https://github.com/google/go-cmp 5411ab924f9ffa6566244a9e504bc347edacffd3:refs/dummy
1.477s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git fetch -f --depth=1 https://github.com/google/go-cmp 5411ab924f9ffa6566244a9e504bc347edacffd3:refs/dummy
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
0.002s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
0.002s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git describe --first-parent --always --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' --tags 5411ab924f9ffa6566244a9e504bc347edacffd3
0.002s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git describe --first-parent --always --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' --tags 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c protocol.version=0 fetch --unshallow -f https://github.com/google/go-cmp 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
1.729s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c protocol.version=0 fetch --unshallow -f https://github.com/google/go-cmp 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git describe --first-parent --always --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' --tags 5411ab924f9ffa6566244a9e504bc347edacffd3
0.014s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git describe --first-parent --always --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' --tags 5411ab924f9ffa6566244a9e504bc347edacffd3
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9f
0.006s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 5411ab924f9f
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
0.004s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
0.003s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
go: downloading github.com/google/go-cmp v0.2.1-0.20180328201512-5411ab924f9f
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
0.002s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git cat-file blob 5411ab924f9ffa6566244a9e504bc347edacffd3:go.mod
cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 5411ab924f9ffa6566244a9e504bc347edacffd3
0.018s # cd /tmp/gomodtestpath/pkg/mod/cache/vcs/e5e4370a1db1bf6e0ffae53084bf5ba02ef21c66da499923d373134469cb366e; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 5411ab924f9ffa6566244a9e504bc347edacffd3
% 
% cat go.mod
module example.com/foo/bar

require (
	github.com/frankban/quicktest v1.1.0
	github.com/google/go-cmp v0.2.1-0.20180328201512-5411ab924f9f // indirect
)

Answer 1 · 2018-11-16T22:42:04.000Z

github.com/google/go-cmp v0.0.0-20180328201512-5411ab924f9f // indirect

The v0.0.0 pseudo-version is the problem: go get sees that and decides that it needs to downgrade quicktest, because the previous quicktest version required v0.2.0.

I don't know how you got an entry mapping that commit to v0.0.0 in your module cache, given that it's after the v0.2.0 tag, but I'm not surprised: we are a bit sloppy about the pseudo-versions we allow (#27173). Perhaps you resolved it as v0.0.0 via an older go.mod file at some point?

At any rate, if you go get the go-cmp repo with a higher-numbered pseudoversion, you'll find that it sticks:

go: creating new go.mod: module golang.org/issue/27171

$ go get -m github.com/google/go-cmp@v0.0.0-20180328201512-5411ab924f9f
go: finding github.com/google/go-cmp v0.0.0-20180328201512-5411ab924f9f

$ go get -m github.com/frankban/quicktest@v1.1.0
go: finding github.com/frankban/quicktest v1.1.0
go: finding github.com/google/go-cmp v0.2.0
go: finding github.com/kr/pretty v0.1.0
go: finding github.com/kr/text v0.1.0
go: finding github.com/kr/pty v1.1.1

$ go list -m all
golang.org/issue/27171
github.com/frankban/quicktest v1.1.0
github.com/google/go-cmp v0.2.0
github.com/kr/pretty v0.1.0
github.com/kr/pty v1.1.1
github.com/kr/text v0.1.0

$ go get -m github.com/google/go-cmp@5411ab924f9f
go: finding github.com/frankban/quicktest v1.0.0
go: finding github.com/frankban/quicktest v0.9.1
go: finding github.com/frankban/quicktest v0.9.0
go: finding github.com/frankban/quicktest v0.8.0

$ go list -m all
golang.org/issue/27171
github.com/frankban/quicktest v0.8.0
github.com/google/go-cmp v0.0.0-20180328201512-5411ab924f9f
github.com/kr/pretty v0.1.0
github.com/kr/pty v1.1.1
github.com/kr/text v0.1.0

$ go get -m github.com/google/go-cmp@v0.2.1-0.20180328201512-5411ab924f9f
go: finding github.com/google/go-cmp v0.2.1-0.20180328201512-5411ab924f9f

$ go get -m github.com/frankban/quicktest@v1.1.0

$ go get -m github.com/google/go-cmp@5411ab924f9f

$ go list -m all
golang.org/issue/27171
github.com/frankban/quicktest v1.1.0
github.com/google/go-cmp v0.2.1-0.20180328201512-5411ab924f9f
github.com/kr/pretty v0.1.0
github.com/kr/pty v1.1.1
github.com/kr/text v0.1.0

Answer 2 · 2019-01-16T02:51:28.000Z

Another observation.
Seems like the pseudo-version generation is sensitive to what's in the module cache.

$ export GOPATH=$(mktemp -d)
$ go list -m -json golang.org/x/text@master
$ cat go.mod
module foo

go 1.12

$ go list -m -json golang.org/x/text@master
go: finding golang.org/x/text master
{
	"Path": "golang.org/x/text",
	"Version": "v0.0.0-20181227161524-e6919f6577db",
	"Time": "2018-12-27T16:15:24Z"
}

$ go list -m -json golang.org/x/text@master
go: finding golang.org/x/text master
{
	"Path": "golang.org/x/text",
	"Version": "v0.3.1-0.20181227161524-e6919f6577db",
	"Time": "2018-12-27T16:15:24Z"
}

Note that two subsequent go list commands (back to back) resulted in two different pseudo versions based on the same hash. It seems the pseudo-version generation is sensitive to what is in the module cache.

$ go version
go version devel +5fae09b738 Tue Jan 15 23:30:58 2019 +0000 darwin/amd64

Answer 3 · 2019-01-17T18:24:37.000Z

Working with docker with Go modules is currently a bit convoluted for multiple reasons (e.g., sirupsen/logrus#799 (comment)). There are solutions (e.g., see #28489 (comment), #28489 (comment), and others), but this issue here adds to the complexity.

For example, you can have something like the following version (from early 2018) in one go.mod:

$ go1.12beta1 get github.com/docker/docker@0ede01237c9a
$ grep github.com/docker/docker go.mod

require github.com/docker/docker v0.0.0-20180221164450-0ede01237c9a // indirect

but then because that pseudo-version seems to incorrectly use v0.0.0, it is treated as happening prior to the latest syntactically correct semver version tag for docker of v1.13.1 (from early 2017), and you get thrown to the older v1.13.1 as the selected version in your build rather than the newer commit you specified, which is not the correct end-to-end result. (The semver comparison itself is correct, but the pseudo-version fed into the comparison is incorrect, which I think is this issue here, if following).

On top of that, v1.13.1 shows up for docker probably more than it should, either due to the "expected" reasons related to having dependencies that do not themselves have a go.mod, or in some cases I suspect perhaps due to other module issues.

CC @vito-c

Answer 4 · 2019-03-06T14:47:59.000Z

This write-up here is based on a older quick look, so please take with grain of salt.

Depending on how you count, there might be 2-3 different things going on here:

Docker and other repos that have tags that match a glob pattern of v[0-9]*.[0-9]*.[0-9]* can end up with v0.0.0 pseudo-versions instead of finding a proper semver tag to use as input to building a pseudo-version (when then causes problems when the v0.0.0 pseudo-version is fed into MVS). There is a RecentTag method that is defined to be "best effort", but not 100% clear if the caller handles the cases when the result is incorrect. describe here in RecentTag returns success here even if the result will be effectively rejected by the caller of RecentTag when the caller then calls tagToVersion. Perhaps that is deliberate, or perhaps that was just the first cut implementation.

	// RecentTag returns the most recent tag at or before the given rev
	// with the given prefix. It should make a best-effort attempt to
	// find a tag that is a valid semantic version (following the prefix),
	// or else the result is not useful to the caller, but it need not
	// incur great expense in doing so.

This issue crops up in multiple repos, including notably docker, but a simplified constructed example showing a bad result vs. a good result:

# incorrectly records v0.0.0-20181229213331-5f2c3dc5628b, seemingly due to tag v18.06.16-ce
$ go get github.com/thepudds/nomodlib@5f2c3dc5628b

# correctly records v1.0.1-0.20181229213050-3e4103ddf9e0
$ go get github.com/thepudds/nomodlib@3e4103ddf9e0

In #29262, rsc.io/quote at master has two different major versions following the Major Subdirectory approach. It has a v1 module in the root of the repo, and a v3 module in a /v3 subdirectory. If you ask for go get rsc.io/quote@master, that is asking for v1 of rsc.io/quote, which is a valid request, but the most recent tag on master matching that glob pattern ends up being v3.0.0, which I think is what is returned by RecentTag. The v3 is not what is desired, and the caller gives up and falls back to a v0.0.0- pseudo version (probably here). This could be viewed as a variation of 1., or could be viewed as a separate problem.
golang/xtext in #27171 (comment) seems to get different results for the tag returned by git describe on later runs of go get. Shot in the dark -- perhaps more git history is fetched as part of one of the earlier go get invocations, but that happens too late in that earlier go get to find the v0.3.x tag, but that additional history in the cache then makes a later go get find the v0.3.x tag?

Separately, I have not looked at the original report in #27171 (comment). Perhaps that falls into one of these three buckets, or perhaps it was just an older go.mod from vgo, or perhaps that is a distinct issue.

Answer 5 · 2019-04-24T07:40:39.000Z

I'm trying to wrap my head around pseudo-versions and how they are intended to work, in particular the vn.n.n-0.date-hash form. The current documentation in pseudo.go states that:

If the most recent tagged version before the target commit is vX.Y.Z or vX.Y.Z+incompatible, then the pseudo-version uses form (2) or (3), [...]

What is the exact definition of "most recent" here? Is it the most recent in the current branch, the most recent that is merged into the current branch or the most recent that exists on the current branch? (or something else).

As an example, if I have this tree:

*   5aaa858 (HEAD -> master) Merge branch 'branch2'
|\  
| * 56bdc7d (tag: v0.2.2, branch2) ...
| * 1ce5c87 ...
* | 8d46cd4 (tag: v0.2.1) .
|/  
* 987d3d9 (tag: v0.2.0) .
*   ba5332a Merge branch 'branch1'
|\  
| * 620c636 (branch1) ..
| * 64deab7 (tag: v0.1.2) ..
| * 12d424d (tag: v0.1.1) ..
| * 070f504 .
* | 6c864f2 .
* | 3e081ae .
* | 98b862a (tag: v0.1.3) .
|/  
* b8cb65c (tag: v0.1.0) .
* 8b8037c .
* c16c020 (origin/master) empty readme

.. and runs go get <repo>@master should I expect a pseudo version based on:

v0.2.2 - the most recent in creation date, that is merged into master
v0.2.1 - the most recent, that is tagged on master
(or something else?)

Answer 6 · 2019-04-24T16:46:32.000Z

What is the exact definition of "most recent" here?

It is intended to be the semantically-latest tag that appears on any commit that is a (transitive) parent of the commit with the given hash, regardless of branches. (The pseudo-version is supposed to sort after every version — tagged or otherwise — that came before it, but before the next tag that a human might plausibly want to apply to the branch.)

In your example, you should expect a pseudo-version that sorts just after v0.2.2, since it is semantically higher than the other tags and is a parent of the current master (5aaa858).

Of course, bugs are possible (and, I suspect in this case, likely). 🙂

Answer 7 · 2019-04-24T18:29:47.000Z

Nice, and two more questions for the same context:

Is it just annotated tags or should non-annotated tags be considered as well?
semantically-latest as in limited to major versions v0 and v1 right? Like the following example should give v1.0.1

* 91ba41 (HEAD -> master) ..
* b319ca (tag: v2.1.0) ..
* c49efa (tag: v1.0.1) ..
* 4baa0c (tag: v0.9.5) ..

Answer 8 · 2019-04-24T19:54:49.000Z

Is it just annotated tags or should non-annotated tags be considered as well?

We don't go out of our way to distinguish. I suspect that we should consider both.
(If you have evidence that we are or are not already — especially cases where we should arguably doing something other than what we are — please do let us know.)

Answer 9 · 2019-04-24T19:59:21.000Z

semantically-latest as in limited to major versions v0 and v1 right?

The major-version handling probably needs work.

If there is no go.mod file, then for tagged versions we normally allow a major-version mismatch but add a +incompatible suffix. It's not obvious to me what we should do for pseudo-versions under that condition.

If there is a go.mod file, then we currently do not allow a major-version mismatch (but perhaps we should in case of replacements; I'll file a separate issue for that). Whatever we do, we probably should not generate a pseudo-version with a particular major-version component if we would not allow that same major version for the same contents with an explicit tag.

Answer 10 · 2019-04-25T13:28:39.000Z

@leitzler For your example in #27171 (comment), Bryan commented in #27171 (comment) on the expected result, but what do you currently see as the actual result for your specific example with Go 1.12 or tip?

In other words, with the current implementation, do you see a pseudo-version that sorts just after one of these, and if so, which one:

v0.2.2 - the most recent in creation date, that is merged into master

v0.2.1 - the most recent, that is tagged on master

(Possibly related to #31673 or #31287).

Answer 11 · 2019-04-25T17:55:23.000Z

Go 1.12.4

# go version
go version go1.12.4 linux/amd64
# cd $(mktemp -d) && go mod init foo
go: creating new go.mod: module foo
# go get -v github.com/leitzler/tagtests
go: finding github.com/leitzler/tagtests v0.2.2
go: downloading github.com/leitzler/tagtests v0.2.2
go: extracting github.com/leitzler/tagtests v0.2.2
Fetching https://github.com?go-get=1
Parsing meta tags from https://github.com?go-get=1 (status code 200)
# cat go.mod
module foo

go 1.12

require github.com/leitzler/tagtests v0.2.2 // indirect
# cat go.sum
github.com/leitzler/tagtests v0.2.2 h1:iHmrV3vzYCN0qAo+SSyKUBtPQX4uvo65IAmo3Hh5Qw0=
github.com/leitzler/tagtests v0.2.2/go.mod h1:wK4dd9tQFb9oteDJOe6SYbpZj44o+r94PUWEw+ZFpg4=

Answer 12 · 2019-04-25T18:01:58.000Z

However, if I ask for @master it will give me a pseudo version based on v0.2.1:

# go list -u -m all
foo
github.com/leitzler/tagtests v0.2.2
# go get -v github.com/leitzler/tagtests@master
go: finding github.com/leitzler/tagtests master
go: downloading github.com/leitzler/tagtests v0.2.2-0.20190424071028-5aaa858a59e2
go: extracting github.com/leitzler/tagtests v0.2.2-0.20190424071028-5aaa858a59e2
Fetching https://github.com?go-get=1
Parsing meta tags from https://github.com?go-get=1 (status code 200)
# go list -u -m all
foo
github.com/leitzler/tagtests v0.2.2-0.20190424071028-5aaa858a59e2 [v0.2.2]

Answer 13 · 2019-04-25T18:08:11.000Z

@leitzler OK, Go 1.12 seems to give the wrong answer for @master in your simple example according to what Bryan said in #27171 (comment), which is perhaps the same issue as #31673.

In #27171 (comment) above, I had tried to differentiate between what seemed to be at least three different underlying problems for the different examples cited in this issue. It would at least seem from the outside that your example and #31673 are likely a fourth category of problem.

Answer 14 · 2019-04-26T21:02:33.000Z

Change https://golang.org/cl/174061 mentions this issue: WIP: Change RecentTag in modfetch/git to use for-each-ref instead of describe

Answer 15 · 2019-05-09T23:07:34.000Z

Change https://golang.org/cl/176417 mentions this issue: cmd/go: move two vcs test repos to vcs-test.golang.org

Answer 16 · 2019-05-22T13:13:27.000Z

Here is a reproducible demonstration of GOPATH version pollution at work (use this command to run the script).
It shows that a previous go get command can affect the results of a subsequent one, which it should not.

# with an unpolluted GOPATH, go get gets the correct version
env GOPATH=$WORK/gopath1
mkdir m1
cd m1
go mod init example
go get github.com/labstack/gommon@34167a09256a
grep v0.2.9-0.20181219175904-34167a09256a go.mod

# first make a module and pollute GOPATH with a "wrong" version.
env GOPATH=$WORK/gopath2
mkdir m1
cd m1
go mod init example
go get github.com/labstack/gommon@v0.0.0-20181219175904-34167a09256a

# then make another module and observe the pollution at work.
cd $WORK
mkdir m2
cd m2
go mod init example
go get github.com/labstack/gommon@34167a09256a
grep v0.2.9-0.20181219175904-34167a09256a go.mod

When I run the script, I see this result, indicating that the last go get command resolved to v0.0.0-20181219175904-34167a09256a not the correct v0.2.9-0.20181219175904-34167a09256a.

# with an unpolluted GOPATH, go get gets the correct version (1.875s)
# first make a module and pollute GOPATH with a "wrong" version. (1.903s)
# then make another module and observe the pollution at work. (0.055s)
> cd $WORK
$WORK
> mkdir m2
> cd m2
$WORK/m2
> go mod init example
[stderr]
go: creating new go.mod: module example

> go get github.com/labstack/gommon@34167a09256a
> grep v0.2.9-0.20181219175904-34167a09256a go.mod
[go.mod]
module example

go 1.13

require github.com/labstack/gommon v0.0.0-20181219175904-34167a09256a // indirect

FAIL: /tmp/testscript114881046/0/script.txt:22: no match for `v0.2.9-0.20181219175904-34167a09256a` found in go.mod
error running /tmp/m.script in /tmp/testscript114881046/0

Answer 17 · 2019-05-22T13:27:46.000Z

@rogpeppe does #27173 look related to your most recent report here?

Answer 18 · 2019-05-22T13:36:40.000Z

@thepudds Yes, definitely related, but the description of that issue isn't very clear to me, so I'm not sure if it's a strict dupe.

The underlying problem here , I think, is that the go command is using existing information in $GOPATH to determine the version resolution, but instead it really needs to go to the source always, because the correct answer to the question "what is the latest version for this commit" can change over time.

Answer 19 · 2019-05-22T15:05:42.000Z

Hi @rogpeppe, agreed not a strict dup. Part of the relationship is #27173 might show how one can put the wrong semver tag within a pseudoversion, and it still works without the go tool changing it to be a correct semver tag. That said, #27173 happens to not be 100% clear to me either.

The underlying problem here , I think, is that the go command is using existing information in $GOPATH to determine the version resolution, but instead it really needs to go to the source always, because the correct answer to the question "what is the latest version for this commit" can change over time.

There might be performance implications always of going to the source as part of something like go build or go list, etc.? You could say go get should always go to the source to verify any pseudoversions are correct, but go get foo@<version> mostly has the same behavior as editing the go.mod to read require foo <version> and then doing a go build or go test or similar, which means it might not make sense for go get either. However, maybe go mod tidy could be changed to always go to the source to verify any pseudoversion is correct, given it is already more-or-less documented to be more expensive in terms of module resolution, and go mod tidy is a rarer operation, and it is called tidy. But not 100% sure if that makes sense.

Answer 20 · 2019-05-22T15:56:17.000Z

@rogpeppe, thanks for the clear reproducer. That confirms my suspicions about the mechanism of the bug.

I tend to agree that we should go to version control (or re-query the proxy) to resolve commits to versions. It's not clear to me whether we should try the local cache first when GOPROXY is set to off, although I suspect we should not.

Answer 21 · 2019-05-22T15:58:21.000Z

@thepudds, there should be no performance implication for go build or go list in the steady state: once we have resolved the commit to a specific pseudo-version, we store that pseudo-version in the go.mod file and we can use that rather than re-resolving the underlying commit.

(And, in fact, it may be important not to re-resolve the version as the underlying tags evolve, since that can affect the MVS result and impact reproducibility.)

Answer 22 · 2019-05-22T16:04:10.000Z

It's not clear to me whether we should try the local cache first when GOPROXY is set to off, although I suspect we should not.

I'd second this.

Answer 23 · 2019-05-22T16:16:02.000Z

The underlying problem here , I think, is that the go command is using existing information in $GOPATH to determine the version resolution, but instead it really needs to go to the source always, because the correct answer to the question "what is the latest version for this commit" can change over time.

I think I misunderstood @rogpeppe's above suggestion then.

Based on @bcmills comment, is the suggestion instead:

always go back to source when doing a module query such as go get github.com/labstack/gommon@34167a09256a
but do not always go back to source when doing something like go get github.com/labstack/gommon@v0.0.0-20181219175904-34167a09256a (which is a syntactically valid pseudoversion, but has the "wrong" semver tag within it)?

If that is the suggestion, then I withdraw my comment #27171 (comment) on performance / correctness concern.

Answer 24 · 2019-05-29T15:35:05.000Z

The pseudoversion for golang.org/x/text in #27171 (comment) seemed to be sensitive to what was in the cache. From looking at that briefly several months ago and mentioned in #27171 (comment), it seemed to be due to different results for the tag returned by git describe on later runs of go get.

Trying to recreate the problem from #27171 (comment) with current tip no longer seems to exhibit the problem:

go get golang.org/dl/gotip && gotip download
export GOPATH=$(mktemp -d)
cd $(mktemp -d)
gotip mod init tempmod
gotip list -m -json golang.org/x/text@master
gotip list -m -json golang.org/x/text@master
gotip list -m -json golang.org/x/text@master

which consistently returns:

{
        "Path": "golang.org/x/text",
        "Version": "v0.3.2",
        "Time": "2019-04-25T14:42:06-07:00"
}

Using the commit hash from the initial report in #27171 (comment) also now seems consistent:

export GOPATH=$(mktemp -d)
cd $(mktemp -d)
gotip mod init tempmod
gotip list -m -json golang.org/x/text@e6919f6577db
gotip list -m -json golang.org/x/text@e6919f6577db
gotip list -m -json golang.org/x/text@e6919f6577db

which consistently returns:

{
        "Path": "golang.org/x/text",
        "Version": "v0.3.1-0.20181227161524-e6919f6577db",
        "Time": "2018-12-27T08:15:24-08:00"
}

I am not sure if that is due to changes in the x/text repo, or other ecosystem changes, or perhaps due to changes in the go tool (e.g., maybe CL https://golang.org/cl/174061, which means it now no longer uses git describe in the section of code that was previously getting an unexpected tag for x/text).

@hyangah I would be curious to hear if you able to reproduce your original report from #27171 (comment), or perhaps a variation?

Answer 25 · 2019-05-29T17:22:18.000Z

@thepudds I just confirmed that my reproducer in #27171 (comment) still shows this issue with Go tip (as of commit e883d00).

Perhaps you could check that to see if it works for you?

Answer 26 · 2019-05-30T15:11:56.000Z

@thepudds I can't reproduce my report that uses golang.org/x/text either. I tested it again after manually reverting the cl/174061 and still couldn't reproduce the issue. So it's possible there was a different underlying issue, and now it's not reproducible.

Answer 27 · 2019-05-30T18:04:02.000Z

golang.org/x/text got a new v0.3.2 tag recently, and it points to the same commit as the current master branch. That's probably why that example no longer reproduces.

Answer 28 · 2019-05-30T18:37:13.000Z

In comment #27171 (comment) from several months ago, I had attempted to tease apart what the different underlying problems might be. At that time, the count was three.

At this point, there are probably at least 5-6 underlying problems reported in this issue, which I will attempt to enumerate as:

The initial report by @rogpeppe that created this issue in #27171 (comment).
- Unclear if this was a "hangover" from vgo not doing the right thing, or maybe an early case of item 5 below ("cache pollution"), or maybe something else entirely.
Docker and other repos that have tags that match a glob pattern of v[0-9]*.[0-9]*.[0-9]* can incorrectly end up with v0.0.0 pseudo-version.
- Fixed by https://golang.org/cl/174061.
In #29262, rsc.io/quote at master has two different major versions following the Major Subdirectory approach, and go get rsc.io/quote@master requests a v1 version but does not find a v1 based semver tag because it first finds the v3 semver tag.
- Not fixed.
golang/xtext in #27171 (comment) seems to get different results for the tag returned by git describe on later runs of go get.
- Not reproducing currently (#27171 (comment)), but might not be fixed. Bryan guesses not yet fixed in #27171 (comment).
Tags on other branches were not handled properly (#31673, #31287, plus related comments here from @leitzler and myself such as #27171 (comment))
- Fixed by https://golang.org/cl/174061.
Cache pollution where a bad pseudoversion lives on, as reported by @rogpeppe in #27171 (comment).
- Not fixed (and not yet fully investigated to my knowledge anyway).
Arguable if #27173 deserves to be in this list given it is still an open separate issue, but it might be related to "cache pollution" given that #27173 issue illustrates pushing a series of "bad" psuedoversions into a go.mod that are then trusted and not updated to a more correct psuedoversion. That issue though also touches on other things, including that there might not be a single correct canonical answer for a pseudoversion.
- Not fixed.

Given the variety above, it would not be shocking if there were additional underlying issues not yet teased out.

Answer 29 · 2019-05-31T19:56:34.000Z

Change https://golang.org/cl/179857 mentions this issue: cmd/go/internal/modfetch: use the resolved version to search for tags in (*codeRepo).convert

Answer 30 · 2019-06-12T22:07:36.000Z

Change https://golang.org/cl/181881 mentions this issue: cmd/go: validate pseudo-versions against module paths and revision metadata

Answer 31 · 2019-06-13T15:09:25.000Z

Change https://golang.org/cl/182178 mentions this issue: cmd/go/internal/modfetch: re-resolve commit hashes in readDiskStat

Answer 32 · 2019-06-18T18:43:07.000Z

I've seen several examples in the module index (https://index.golang.org) of invalid pseudo-versions starting with v0.0.0-0.. If they originated with a go command bug it seems fine to allow them, but looking through the history of the code I'm not able to find any such bug.

Does anyone have any examples of a real go.mod file containing a v0.0.0-0.[…] version string in the wild? If so, do you know how it got there?

(Perhaps @marwan-at-work, @rogpeppe, @myitcv, or one of the other folks with third-party module-related tools has a clue?)

Answer 33 · 2019-06-18T20:21:53.000Z

has a clue?

No clue here.

Answer 34 · 2019-06-18T20:56:59.000Z

I noticed one of my repos in a listing you pasted on slack (or in another issue? Can’t really remember where I saw it). Anything regarding https://github.com/modsdemo is my fault. Have been playing around on purpose with it.

Answer 35 · 2019-06-18T21:05:20.000Z

@leitzler, thanks for the heads-up. (The comment you saw it in was #27173 (comment).)