cmd/go: checksum mismatch for x8s.io/client-go@v8.0.0+incompatible

Question

cmd/go: checksum mismatch for x8s.io/client-go@v8.0.0+incompatible

jharshman opened this issue 6 years ago · 40 comments

jharshman commented 6 years ago

What version of Go are you using (`go version`)?

Go 1.11

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

darwin
amd64

What did you do?

go build ./...
reproduction is difficult because including the package k8s.io/client-go causes incompatible dependencies to be pulled in which is something I have to solve manually.

What did you expect to see?

A repeat of successful builds from the day before.

What did you see instead?

A checksum mismatch this morning after cloning the project to my personal laptop and attempting a build.

go: verifying k8s.io/client-go@v8.0.0+incompatible: checksum mismatch
        downloaded: h1:7Zl+OVXn0bobcsi4NEZGdoQDTE9ij1zPMfM21+yqQsM=
        go.sum:     h1:2pUaSg2x6iEHr8cia6zmWhoCXG1EDG9TCx9s//Aq7HY=

Answer 1 · 2018-09-28T19:07:25.000Z

Okay so the workaround is to manually go get the correct version and then run the build again. This is a little unpredictable IMO. I'm guessing this stuff will clear up as more packages add module support and semantic versioning.

Answer 2 · 2018-09-28T19:15:36.000Z

@jharshman is there further investigation that needs to be done for this issue? Is there documentation that you are suggesting should be updated? If not, it sounds like this issue can be closed.

Answer 3 · 2018-09-28T19:53:15.000Z

+incompatible versions should not cause checksum mismatches.

Is it possible that you were running more than one build concurrently? (#26794)

Answer 4 · 2018-09-28T20:19:46.000Z

@katiehockman I would say this particular issue shouldn't be closed because this seems to be a symptom of a larger issue and needs to be addressed. As @bcmills mentioned, +incompatible versions should not cause checksum mismatches but in this case it has.

And @bcmills to answer your question, no this was the only build running. There were no other builds going on concurrently.

Answer 5 · 2018-10-01T23:09:12.000Z

Also seeing this problem but not sure if this is due to GOOS mismatch. I am using WSL (Ubuntu x64) on Windows 10 to run go mod tidy and generate the corresponding go.sum file. I am building and testing using Gitlab CI using a Go 1.11 Alpine x64 container and I am seeing:

go: verifying k8s.io/client-go@v8.0.0+incompatible: checksum mismatch
	downloaded: h1:7Zl+OVXn0bobcsi4NEZGdoQDTE9ij1zPMfM21+yqQsM=
	go.sum:     h1:2pUaSg2x6iEHr8cia6zmWhoCXG1EDG9TCx9s//Aq7HY=

go version on both the ci machine and my own machine outputs: go version go1.11 linux/amd64.

I also recreate the go.sum and go.mod files using an Go 1.11 alpine container (the same one used in CI) and still see the same problem.

Answer 6 · 2018-10-04T21:59:57.000Z

@katiehockman please remove the WaitingForInfo label unless there is more information that you are requesting.
@bcmills is there a solution in the pipe for this? It's very reproducible so I can image other users are experiencing it as well.

The best solution I have found is to go mod vendor once you have a working build.

Answer 7 · 2018-10-04T22:10:30.000Z

@jharshman Could you try a Go build from source at commit 1bca6ce and see if it still reproduces?

I wonder if this is a side-effect of #27153.

Answer 8 · 2018-10-05T05:04:26.000Z

Im having similar issue with golang master branch on my travis build. What's resolution for this?

Answer 9 · 2018-10-05T12:37:13.000Z

@leakingtapan, try a go built from source at the above-mentioned commit, and let us know whether it helps.

For CI-related issues, though, you're more likely running up against #26794.

Answer 10 · 2018-10-05T20:25:38.000Z

I tried 1bca6ce with no luck. I also tried 8081862 which was working on my Travis build, but now even this is failing.

go: verifying k8s.io/kubernetes@v1.11.2: checksum mismatch
        downloaded: h1:AAwZncqPvpeV6DPF8/z2g05/79cumA6IunGDByI1bOs=
        go.sum:     h1:2/lmzYbN17Mr23mX/p4ODMm/MBWTeu2Q1Bvsw82YC40=

Answer 11 · 2018-10-08T19:35:27.000Z

Same issue here:

go: downloading k8s.io/client-go v9.0.0+incompatible
go: downloading k8s.io/api v0.0.0-20181006043042-ded30614edd7
go: verifying k8s.io/client-go@v9.0.0+incompatible: checksum mismatch
        downloaded: h1:/PdJjifJTjMFe0G4ESclZDcwF1+bFePTJ2xf+MXjcvs=
        go.sum:     h1:NXWpDuPFeVB5lYP1fTqJUtwigjtmRXJNtndnN53ldGI=

Answer 12 · 2018-10-09T02:43:40.000Z

@leakingtapan @Cosmonawt @jharshman @F21

If you are running 1bca6ce your h1 hashes will, unfortunately, have to be updated once. Both kubernetes/kubernetes and kubernetes/client-go use export-subst which is disabled by this change:

If you are running 1bca6ce you should get the same hash moving forward. Just delete the old hash and use the new one. Again, you should only have to do this once.

I imagine a lot of folks will run into this issue since k8s repos appear to use export-subst in various places.

If you are seeing inconsistent hashing results I recommend:

Build go from source after 1bca6ce
Delete the old h1 hash for k8s.io/kubernetes k8s.io/client-go and any other repo using export-subst.
Build your project and record the new hash for k8s.io/kubernetes k8s.io/client-go etc

If the hash changes for you after that then please report back.

Answer 13 · 2018-10-09T18:57:09.000Z

@jasonkeene thx for providing the solution. In order to not break people who is using go1.11 to build our package, I decided to stay with go1.11 for a while. This makes my travis build and mac build happy. However, when I tried build on a linux machine the issue even happen on go1.11, which is surprising. Here is the output:

mac

>> go version
go version go1.11 darwin/amd64

>> echo $GO111MODULE
on

>> make
make
mkdir -p bin
go build -o bin/aws-ebs-csi-driver ./cmd/aws-ebs-csi-driver

linux

>> go version
go version go1.11 linux/amd64

>> echo $GO111MODULE                                                                                                                                                                                                            
on

>> make
mkdir -p bin
go build -o bin/aws-ebs-csi-driver ./cmd/aws-ebs-csi-driver
go: verifying k8s.io/kubernetes@v1.11.2: checksum mismatch
        downloaded: h1:AAwZncqPvpeV6DPF8/z2g05/79cumA6IunGDByI1bOs=
        go.sum:     h1:2/lmzYbN17Mr23mX/p4ODMm/MBWTeu2Q1Bvsw82YC40=
make: *** [aws-ebs-csi-driver] Error 1

Answer 14 · 2018-10-09T20:27:28.000Z

Yes this might be the problem 1bca6ce was intended to solve. Can you diff the contents of the two zip files that produce the different hashes?

Answer 15 · 2018-10-10T21:40:25.000Z

Thanks for the solution @jasonkeene , but building Go from source is the verification, not a long term solution. We need to be able to build projects reliably cross teams, platforms, and in CI.

It sounds like a bug-fix release needs to be cut from this.

Answer 16 · 2018-10-10T23:47:39.000Z

@jharshman For sure compiling from source was just to test to make sure this was your issue.

As far as the bug fix release, pretty sure @bcmills is on it here: #28094

I would still like you to try the above steps to make sure you are not running into another issue and that the bug fix will actually fix your problem.

Answer 17 · 2018-10-11T16:34:22.000Z

@jasonkeene sure let me verify that I don't hit any other issues 👍

Answer 18 · 2018-10-15T17:35:02.000Z

I'm not sure if it's related, but this may be related to an issue we've been having with modules: #27952

We've had situations where the incomplete project didn't compile, but we've also had times that it compiled, but had the wrong hash because files were missing.

Answer 19 · 2018-10-16T18:33:31.000Z

@jasonkeene verified the fix by compiling from source after 1bca6ce. Works after removing the older hashes.

Answer 20 · 2018-10-17T00:06:58.000Z

Awesome, good to know!

Answer 21 · 2018-10-25T15:38:00.000Z

OK, fixed by 1bca6ce.

Answer 22 · 2018-11-07T22:02:08.000Z

This does not appear to be fixed in Go 1.11.2. I removed go.sum, generated a new one with go mod tidy using Go 1.11.2, and I still encountered a checksum error when building the project in a Docker container from image golang:1.11.2.

go: verifying k8s.io/client-go@v9.0.0+incompatible: checksum mismatch
	downloaded: h1:2kqW3X2xQ9SbFvWZjGEHBLlWc1LG9JIJNXWkuqwdZ3A=
	go.sum:     h1:NXWpDuPFeVB5lYP1fTqJUtwigjtmRXJNtndnN53ldGI=

Answer 23 · 2018-11-09T14:27:59.000Z

@jayschwa Did you run go clean -modcache from your host machine? The fix that is in 1.11.2 only applies to creating archives of modules when they are downloaded. If you already had the zipfile created and is in your cache it will need to be removed.

Answer 24 · 2018-11-12T17:08:00.000Z

@jasonkeene, running go clean -modcache seemed to do the trick. Thanks.

Answer 25 · 2018-11-16T05:02:29.000Z

I have a project with a private dependency that looks like a regression in regards to this issue between 1.11.1 and 1.11.2.

Host: Darwin
CI: GCP Cloud Builder w/ Alpine image

If I build the container locally on Darwin, things work as expected, but GCP Cloud Builder CI demonstrates the following behavior between go versions:

1.11: checksum mismatch
1.11.1: works are expected
1.11.2: checksum mismatch

These results hold whether or not go.sum is checked in, which means cleaning the modcache won't fix the issue in this case.

Another possible example is in this thread: kubernetes/kubernetes#69040 (comment)

Answer 26 · 2018-11-19T21:10:23.000Z

@buckhx Could the GCP Cloud Builder be using a module cache or the proxy functionality? I do not use this service.

Answer 27 · 2018-11-20T19:53:09.000Z

Even using go clean -modcache with go 1.11.2 is a bad user experience for me. I get an error about checksums even when running the command to clear checksums (GO111MODULE=on go clean -modcache).

go: downloading k8s.io/client-go v9.0.0+incompatible
go: verifying k8s.io/client-go@v9.0.0+incompatible: checksum mismatch
	downloaded: h1:2kqW3X2xQ9SbFvWZjGEHBLlWc1LG9JIJNXWkuqwdZ3A=
	go.sum:     h1:NXWpDuPFeVB5lYP1fTqJUtwigjtmRXJNtndnN53ldGI=

To prevent this, I actually had to disable modules and run the same command (GO111MODULE=off go clean -modcache).

But even then when the clean command has completed, I still get checksum errors when I attempt to run GO111MODULE=on go mod vendor:

go: downloading k8s.io/client-go v9.0.0+incompatible
go: downloading k8s.io/apimachinery v0.0.0-20181108192626-90473842928c
go: finding google.golang.org/grpc/keepalive latest
go: verifying k8s.io/client-go@v9.0.0+incompatible: checksum mismatch
	downloaded: h1:2kqW3X2xQ9SbFvWZjGEHBLlWc1LG9JIJNXWkuqwdZ3A=
	go.sum:     h1:NXWpDuPFeVB5lYP1fTqJUtwigjtmRXJNtndnN53ldGI=

I am on MacOS Mojave, by the way.

Answer 28 · 2018-11-21T10:09:03.000Z

Both 1.11.1 and 1.11.2 are failing for me in client-go. Tried all above workarounds.

Answer 29 · 2018-11-21T10:12:54.000Z

1.11.1:

rm go.sum
go build

works

Answer 30 · 2018-11-26T15:32:51.000Z

@integrii @buckhx Can you diff the contents of the two zipfiles? This will show you why they have different hashes and will give you a clue to what the problem is.

Also, it might be worth opening a new issue with the zipfile diff since this one has been marked as resolved. Feel free to mention me so I don't loose track of this.

Answer 31 · 2018-11-26T21:44:52.000Z

In my case, I found that I had misplaced go.mod and go.sum files in my home directory, which go get was finding automatically and doing very strange things with. I deleted those files and things worked great. Thanks!

Answer 32 · 2018-12-18T16:20:09.000Z

Great to see this is fixed! Any idea when it will be released? Go 1.11.4 is still affected. I've just wasted half a day because GoLand on OS X is updating the sum file automatically with checksums that don't work in Linux containers. Turning this off at least makes it possible to compile again, but without module support / code completion.

OS X (Go 1.11.4, Git 2.17.2)
github.com/kylelemons/go-gypsy v0.0.0-20160905020020-08cad365cd28 h1:SzHLOWB819X9aUXh2ct6krDIoIujoTsj5J1rDMjkRyo=

Linux (Go 1.11.4, Git 2.17.1)
github.com/kylelemons/go-gypsy v0.0.0-20160905020020-08cad365cd28 h1:mkl3tvPHIuPaWsLtmHTybJeoVEW7cbePK73Ir8VtruA=

Answer 33 · 2018-12-18T16:42:16.000Z

@lastzero you're most probably hitting #29278 (comment)

Answer 34 · 2018-12-18T18:38:10.000Z

@simonpasquier Thank you! Looks like the follow-up "bug". For those also affected: I had to change settings in GoLand (didn't automatically detect latest Go version) plus delete Go module caches locally and on Travis CI.

Answer 35 · 2019-02-07T11:58:38.000Z

Using 1.11.5 and still have this impact

OS: Darwin/Mojave

Edit: apparently it works after I do:

rm go.sum
go mod vendor

Answer 36 · 2019-02-28T01:17:46.000Z

If your machine is macOS or WSL and your CI is Linux, you can add this command to the CI config.

sed -e '/^k8s.io\/client-go /d' -i go.sum

Here is an example for CircleCI:

version: 2
jobs:
  build:
    docker:
      - image: circleci/golang:1.11.2
    steps:
      - checkout
      # workaround for https://github.com/golang/go/issues/27925
      - run: sed -e '/^k8s.io\/client-go /d' -i go.sum
      - run: go vet

Answer 37 · 2019-04-26T04:04:49.000Z

d:\xxx>go version
go version go1.12.4 windows/amd64

d:\xxx>go mod tidy
go: downloading github.com/kevinburke/go-bindata v3.11.0+incompatible
verifying github.com/kevinburke/go-bindata@v3.11.0+incompatible: checksum mismatch
downloaded: h1:RcC+GJNmrBHbGaOpQ9MBD8z22rdzlIm0esDRDkyxd4s=
go.sum: h1:GiPs9jxaG2xY1B5Dt/d/yHUOMlTk14uS35VcmHrdo4I=

Answer 38 · 2019-04-26T05:32:28.000Z

verifying github.com/kylelemons/go-gypsy@v0.0.0-20160905020020-08cad365cd28: che
cksum mismatch
downloaded: h1:mkl3tvPHIuPaWsLtmHTybJeoVEW7cbePK73Ir8VtruA=
go.sum: h1:SzHLOWB819X9aUXh2ct6krDIoIujoTsj5J1rDMjkRyo=

Answer 39 · 2019-04-26T07:19:50.000Z

@runner-mei Did you try go clean -modcache?

Answer 40 · 2019-04-26T07:43:03.000Z

@lastzero oooo, it is ok!

What version of Go are you using (go version)?