GoogleCloudPlatform/berglas

Checksum in sum.golang.org is different from download without proxy/checksum database (GOPRIVATE=*)

piotrkubisa opened this issue · 7 comments

piotrkubisa commented

References

The GOPRIVATE environment variable controls which modules the go command considers to be private (not available publicly) and should therefore not use the proxy or checksum database.
-- https://pkg.go.dev/cmd/go

What?

Whether GOPRIVATE=* environment has been set, the checksum in go.sum for berglas dependency is different than if I would not set GOPRIVATE (standard Go installation). This issue makes unable to have defined custom GOPRIVATE across the team and/or CI environments, because Go compiler will deny any further operations.

While, I rely on numerous go-modules (including various Gcloud SDKs) I have only encountered this problem in berglas. The fact that the berglas manages secrets, this issue might be a security-related problem, because default sum.golang.org offers different version of berglas than actually is distributed over github.com.

How to reproduce?

In a secret gist I have prepared a sample project that imports github.com/GoogleCloudPlatform/berglas and prints out the go.sum, depending whether GOPRIVATE=* was or wasn't defined.

In the diff(1) output presented below, I compare the computed go.sum file with GOPRIVATE=* set and without respectively.

diff out/with-goprivate.txt out/no-goprivate.txt 
57c57
< github.com/GoogleCloudPlatform/berglas v0.6.2 h1:GeTSHX7B/eIONuwnVuh92I3f0yAmdSz6RdAYP5/efQI=
---
> github.com/GoogleCloudPlatform/berglas v0.6.2 h1:mgcdWF53ltZA0hBGWrpBAjuKh5VnQPxHRlFRoHd5tpA=

If you would like to test it on your machine, please take a look in my gist - https://gist.github.com/piotrkubisa/4dbba8d36bf3748944489dc523eefead. I hope example presenting sample code will be more clear for you.

main.go

package main

import (
	"fmt"
	"os"

	"github.com/GoogleCloudPlatform/berglas/pkg/berglas"
	"github.com/sirupsen/logrus"
)

var _ *berglas.ListResponse = nil

func main() {
	fmt.Println("Hello World!")

	logrus.Println("Testing")

	contents, err := os.ReadFile("go.sum")
	if err != nil {
		panic(err)
	}
	fmt.Println(string(contents))
}

Dockerfile

FROM golang:1.17 AS builder

WORKDIR /src
COPY main.go .

# comment/uncomment line below to see a difference in go.sum
# ENV GOPRIVATE=* 

RUN go mod init my-app
RUN go mod tidy --compat=1.17
RUN cat go.mod
RUN cat go.sum

RUN CGO_ENABLED=0 go build -o /bin/app

FROM gcr.io/distroless/static:latest
COPY --from=builder /bin/app /bin/app
COPY --from=builder /src/go.sum /go.sum
ENTRYPOINT ["/bin/app"]
sethvargo commented

I have no idea where to even begin to debug something like that.

sethvargo commented

I tried running the script in your gist on 3 different networks (home, cellular, coffeeshop) to try and reproduce.

First, I tried with no other dependencies besides berglas (removed "github.com/sirupsen/logrus"). On all three networks, I go no diff.

Next, I tried adding "github.com/sirupsen/logrus" back in, but got the same result.

My suspicion is that your proxy, network, ISP, or something higher up the chain is modifying the response.

piotrkubisa commented

Logrus has nothing to do with the issue. It presents a completely other dependency that does not seem to have different checksum, whether the GOPRIVATE=* is set or not. It also takes part in verification if my network setup has some proxy or doesn't. Because, the checksum is exact for both scenarios, I think I don't have any unknown proxy of my ISP.

Also, because we start blaming someone's other network, which likely has nothing to do with my problem, I have prepared you two GitHub workflows, one with GOPRIVATE=* and other without. I find GitHub-managed workflow runners as a neutral ground and good place for more investigation. Here are results, which confirms my thesis - I am not only one, who can be haunted by this problem:

Without GOPRIVATE

github.com/GoogleCloudPlatform/berglas v0.6.2 h1:mgcdWF53ltZA0hBGWrpBAjuKh5VnQPxHRlFRoHd5tpA=
github.com/GoogleCloudPlatform/berglas v0.6.2/go.mod h1:LEKpytS5wf+P8OGenFlsVmi/uwqcjkFQTH8wZABaCgI=

With GOPRIVATE=*

github.com/GoogleCloudPlatform/berglas v0.6.2 h1:GeTSHX7B/eIONuwnVuh92I3f0yAmdSz6RdAYP5/efQI=
github.com/GoogleCloudPlatform/berglas v0.6.2/go.mod h1:LEKpytS5wf+P8OGenFlsVmi/uwqcjkFQTH8wZABaCgI=
sethvargo commented

Alright, I reworked your example a bit to eliminate as many variables as possible and was still able to reproduce it (workflow).

I downloaded the cached binary from https://proxy.golang.org/github.com/%21google%21cloud%21platform/berglas/@v/v0.6.2.zip, unzipped it, and compared it to the v0.6.2 tag. It looks like none of the examples/ are included:

HEAD detached at v0.6.2
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	deleted:    examples/appengine/go/.gcloudignore
	deleted:    examples/appengine/go/.gitignore
	deleted:    examples/appengine/go/README.md
	deleted:    examples/appengine/go/app.yaml
	deleted:    examples/appengine/go/go.mod
	deleted:    examples/appengine/go/go.sum
	deleted:    examples/appengine/go/main.go
	deleted:    examples/appengineflex/go/.gitignore
	deleted:    examples/appengineflex/go/Dockerfile
	deleted:    examples/appengineflex/go/README.md
	deleted:    examples/appengineflex/go/app.yaml
	deleted:    examples/appengineflex/go/go.mod
	deleted:    examples/appengineflex/go/go.sum
	deleted:    examples/appengineflex/go/main.go
	deleted:    examples/cloudfunctions/go/README.md
	deleted:    examples/cloudfunctions/go/fn.go
	deleted:    examples/cloudfunctions/go/go.mod
	deleted:    examples/cloudfunctions/go/go.sum
	deleted:    examples/cloudrun/go/Dockerfile
	deleted:    examples/cloudrun/go/README.md
	deleted:    examples/cloudrun/go/go.mod
	deleted:    examples/cloudrun/go/go.sum
	deleted:    examples/cloudrun/go/main.go
	deleted:    examples/kubernetes/README.md
	deleted:    examples/kubernetes/deploy/sample.yaml
	deleted:    examples/kubernetes/deploy/webhook.yaml
	deleted:    examples/kubernetes/go.mod
	deleted:    examples/kubernetes/go.sum
	deleted:    examples/kubernetes/main.go

This makes me think that the proxy doesn't include that directory, either because it declares its own go.sum/mod files, or because it is named examples (which seems weird).

Nonetheless, I don't see anything malicious here. If you believe the Go module proxy is modifying packages in ways that it shouldn't, I would recommend using the contact information at the bottom of https://proxy.golang.org/.

  • 👍1
piotrkubisa commented

Okay, that sounds great!

I have strong suspicion towards examples/ directory, which you have named as possible root of the problem. I have done two little changes to the berglas repository in my fork:

  • v0.6.3 - rm -rf examples/ completely. The go.sum file started to look exactly the same in both scenarios.
  • v0.6.4 - I have reverted changes from v0.6.3 and just renamed the examples dir (mv examples renamed-examples-dir). The workflow says, the checksum is identical in both scenarios too.
$ export GOPRIVATE=*
$ go mod download -json -x github.com/piotrkubisa/berglas@v0.6.4
{
        "Path": "github.com/piotrkubisa/berglas",
        "Version": "v0.6.4",
        "Info": "/home/piotr/go/pkg/mod/cache/download/github.com/piotrkubisa/berglas/@v/v0.6.4.info",
        "GoMod": "/home/piotr/go/pkg/mod/cache/download/github.com/piotrkubisa/berglas/@v/v0.6.4.mod",
        "Zip": "/home/piotr/go/pkg/mod/cache/download/github.com/piotrkubisa/berglas/@v/v0.6.4.zip",
        "Dir": "/home/piotr/go/pkg/mod/github.com/piotrkubisa/berglas@v0.6.4",
        "Sum": "h1:d6xGda/YPfMBD3q6Qbpgqi5slRlQ4ULJ5w/8nPW4fK4=",
        "GoModSum": "h1:LEKpytS5wf+P8OGenFlsVmi/uwqcjkFQTH8wZABaCgI="
}
$ unset GOPRIVATE
$ go mod download -json -x github.com/piotrkubisa/berglas@v0.6.4
{
        "Path": "github.com/piotrkubisa/berglas",
        "Version": "v0.6.4",
        "Info": "/home/piotr/go/pkg/mod/cache/download/github.com/piotrkubisa/berglas/@v/v0.6.4.info",
        "GoMod": "/home/piotr/go/pkg/mod/cache/download/github.com/piotrkubisa/berglas/@v/v0.6.4.mod",
        "Zip": "/home/piotr/go/pkg/mod/cache/download/github.com/piotrkubisa/berglas/@v/v0.6.4.zip",
        "Dir": "/home/piotr/go/pkg/mod/github.com/piotrkubisa/berglas@v0.6.4",
        "Sum": "h1:d6xGda/YPfMBD3q6Qbpgqi5slRlQ4ULJ5w/8nPW4fK4=",
        "GoModSum": "h1:LEKpytS5wf+P8OGenFlsVmi/uwqcjkFQTH8wZABaCgI="
}

Would the rename from examples/ to demo/ will fix this problem?

github-actions commented

This issue is stale because it has been open for 14 days with no
activity. It will automatically close after 7 more days of inactivity.

sethvargo commented

Would the rename from examples/ to demo/ will fix this problem?

It's possible, but I "examples" is a pretty common name. I really think a bug should be opened against the module registry for this.

  • 👍1