proposal: cmd/go: allow GOPRIVATE to provide source repository URI

Proposal: Allow GOPRIVATE to provide source repository URI

Last updated: 2021/04/24

Rationale:

In current state of Go modules' tooling, working with private repositories without setting up a private proxy infrastructure serving modules over https is possible but cumbersome; and still is confined to git VCS only.

For git repositories it is usually done like that:

go env -w GOPRIVATE=example.com
git config --global url."git@example.com:".insteadOf "https://example.com"

This configuration works, but it has to be global (per machine/user). To my knowledge no other VCS allows for such workaround.

Proposal:

Allow GOPRIVATE to point to the source repository for a given module path matching pattern. Let user provide vcs type, transport, and credentials.

GOPRIVATE=pattern["["vcs+transport://[userid[:password]@]host[:port][/path]"]"][,...]

GOPRIVATE=example.com/[fossil+ssh://internal.example.com]
GOPRIVATE=example.com/[git+ssh://git@internal.example.com]
GOPRIVATE=example.com/[fossil+https://internal.example.com]
GOPRIVATE=example.com/[git+https://testing.example.com/test]

If pattern ends with slash, allow URI to be constructed using parts of matched module path represented by five percent-letter variables:

   matched.pattern/path/to/module
                           ^^^^^^ %z end element of module path
                   ^^^^           %s start element after the match
                   ^^^^^^^^^^^^^^ %p part after the match
   ^^^^^^^^^^^^^^^                %m matched by (excluding ending slash) 
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ %w whole module path

example.com[git+file:///repos/%p.git]          # get from bare repos archive
example.com[fossil+file:///repos/%s-%z.fossil]    # or a bag of user fossils
github.com/myuser/[git+file:///gopath/src/%w/] # use old GOPATH as source
example.com/[git+ssh://%s@citest.example.com:2222/%p.git] # per CI domain

Reserved characters of RFC 3986 that should be literal part of produced URI must be given in percent-encoding form. Literal characters ":", "[", "]", "", and "@" can also be provided after a backslash character.

edits:

2021/04/24 add reference to #‍40189; add "reserved characters encoding" note.

Note it has always been possible to use your.module/path.vcs as your canonical module path, directly specifying both the module root and the VCS system to use to retrieve it

Note to the readers: (answering doubts raised at irc discussion) This proposal is just about simplifying private repository setup for solo developers and small teams. As per docs GOPRIVATE affects GONOPROXY and GONOSUMDB. Ie. modules still are "downloaded" (here copied) to the GOMODCACHE as if coming from any other source.

This proposal is entirely separate to #44347.

Also, the "keep GOPATH for private repositories" attitude will better be served by #‍44649 proposal.

CC @bcmills @jayconrod

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@ohir I think #40189 and this issue are two different concerns. This issue seems to be about making GOPRIVATE be smarter when authenticating with a VCS backend. While #40189 is about having a sub command that manages authenticating with a GOPROXY.

Although I agree that authenticating with private Go Modules has always been a point of friction, the syntax here seems a bit overloaded and potentially even harder for people to configure correctly.

Go, keeping a small surface area in how it interacts with VCS systems like git is a good thing in my opinion so that it's up to the user to choose the authentication mechanism (git has many ways to do so such as .netrc, config files, and credentials stores). So Go shouldn't be concerned there.

I think for that reason, a GOPROXY is a better answer which means only one server has to configure their VCS credentials correctly and everybody else wouldn't even need to have a VCS program installed on their machine to fetch private Go Modules.

However, the current state of things means that the authentication mechanism with a GOPROXY is almost as cumbersome as the authentication mechanism with a VCS sub system, hence the reason for #40189

Thanks

@marwan-at-work
Thank you for your insights.

the syntax here seems a bit overloaded and potentially even harder for people to configure correctly.

This is pretty much (except for vcs+ part) syntax used by most CLI net tools (wget, curl, ftp, even git itself). Anyway, I got an answer I asked for, thank you :)

GOPRIVATE is really for encoding one specific piece of data. Trying to add more data to the encoding does not seem like it is the right approach. It seems like there's a more general problem here to solve, but I'm not sure what the exact problem statement is and how much to generalize it.

/cc @matloob @jayconrod @bcmills

@rsc

for encoding one specific piece of data [...]

This specific piece of data is enough to opt-out of public, but not enough to work in private.

what the exact problem statement is.

Setting up private repositories is cumbersome for small enterprises and for solo developers. Going private needs either a fragile and error-prone configuration dispersed among at least three env variables (GOPRIVATE GOPROXY GONOPROXY), and possibly local vcs configuration — or it needs a private goproxy server like Athens to be configured and accessible.

This proposal conceives that GOPRIVATE should be the single place where small teams and solo developers can setup all things "private" — without need to think about GOPROXY order and providing external authentication tools (as proposed in #‍40189).

Then tools can access vcs repository and add missing private modules to GOMODCACHE /cache/downloads obeying goproxy protocol.

It's clear we need an answer for the underlying problem that both this issue and #40189 are trying to solve, but neither really solves it completely. We don't have a coherent plan right now. This is something the go command team hopes to look at in the next cycle. Putting on hold until then.

Placed on hold.
— rsc for the proposal review group

@rsc @ohir

Could the extension of go.mod, described in #39536 solve the described issue with setting up and managing private repositories per project?

I've been reading a lot of these issues recently. I don't know much at all about the go internals, but why is the http protocol the first protocol tried, and why doesn't it just fallback to other protocols?

Speaking specifically about Github I would wonder why use the http protocol at all if we know Github supports the git protocol and git+ssh? Plus, Github generally pushes users to use SSH keys.

Is there a proposal to just change the default for github to git+ssh instead of https? Or at least allow users to override this setting?

Update: OK someone pointed out the reason that https is preferred. If you're not a githubber and don't have SSH set you can pull any public repository without auth. Generally that is a major win.

Which is kind of covered #26134, but this whole private repo discussion is going on across a ton of different tickets.

Relevant resources:

Github repository roots forced to HTTPS

go/src/cmd/go/internal/vcs/vcs.go

Lines 1431 to 1437 in d255203

    
           { 
        
           	pathPrefix: "github.com", 
        
           	regexp:     lazyregexp.New(`^(?P<root>github\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`), 
        
           	vcs:        "git", 
        
           	repo:       "https://{root}", 
        
           	check:      noVCSSuffix, 
        
           },

If I weren't using github I think I would be able to specify the protocol?

go/src/cmd/go/internal/vcs/vcs.go

Lines 219 to 253 in d255203

    
           var vcsGit = &Cmd{ 
        
           	Name:      "Git", 
        
           	Cmd:       "git", 
        
           	RootNames: []string{".git"}, 
        
           	CreateCmd:   []string{"clone -- {repo} {dir}", "-go-internal-cd {dir} submodule update --init --recursive"}, 
        
           	DownloadCmd: []string{"pull --ff-only", "submodule update --init --recursive"}, 
        
           	TagCmd: []tagCmd{ 
        
           		// tags/xxx matches a git tag named xxx 
        
           		// origin/xxx matches a git branch named xxx on the default remote repository 
        
           		{"show-ref", `(?:tags|origin)/(\S+)$`}, 
        
           	}, 
        
           	TagLookupCmd: []tagCmd{ 
        
           		{"show-ref tags/{tag} origin/{tag}", `((?:tags|origin)/\S+)$`}, 
        
           	}, 
        
           	TagSyncCmd: []string{"checkout {tag}", "submodule update --init --recursive"}, 
        
           	// both createCmd and downloadCmd update the working dir. 
        
           	// No need to do more here. We used to 'checkout master' 
        
           	// but that doesn't work if the default branch is not named master. 
        
           	// DO NOT add 'checkout master' here. 
        
           	// See golang.org/issue/9032. 
        
           	TagSyncDefault: []string{"submodule update --init --recursive"}, 
        
           	Scheme: []string{"git", "https", "http", "git+ssh", "ssh"}, 
        
           	// Leave out the '--' separator in the ls-remote command: git 2.7.4 does not 
        
           	// support such a separator for that command, and this use should be safe 
        
           	// without it because the {scheme} value comes from the predefined list above. 
        
           	// See golang.org/issue/33836. 
        
           	PingCmd: "ls-remote {scheme}://{repo}", 
        
           	RemoteRepo: gitRemoteRepo, 
        
           	Status:     gitStatus, 
        
           }

	{
	pathPrefix: "github.com",
	regexp: lazyregexp.New(`^(?P<root>github\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`),
	vcs: "git",
	repo: "https://{root}",
	check: noVCSSuffix,
	},

	var vcsGit = &Cmd{
	Name: "Git",
	Cmd: "git",
	RootNames: []string{".git"},

	CreateCmd: []string{"clone -- {repo} {dir}", "-go-internal-cd {dir} submodule update --init --recursive"},
	DownloadCmd: []string{"pull --ff-only", "submodule update --init --recursive"},

	TagCmd: []tagCmd{
	// tags/xxx matches a git tag named xxx
	// origin/xxx matches a git branch named xxx on the default remote repository
	{"show-ref", `(?:tags\|origin)/(\S+)$`},
	},
	TagLookupCmd: []tagCmd{
	{"show-ref tags/{tag} origin/{tag}", `((?:tags\|origin)/\S+)$`},
	},
	TagSyncCmd: []string{"checkout {tag}", "submodule update --init --recursive"},
	// both createCmd and downloadCmd update the working dir.
	// No need to do more here. We used to 'checkout master'
	// but that doesn't work if the default branch is not named master.
	// DO NOT add 'checkout master' here.
	// See golang.org/issue/9032.
	TagSyncDefault: []string{"submodule update --init --recursive"},

	Scheme: []string{"git", "https", "http", "git+ssh", "ssh"},

	// Leave out the '--' separator in the ls-remote command: git 2.7.4 does not
	// support such a separator for that command, and this use should be safe
	// without it because the {scheme} value comes from the predefined list above.
	// See golang.org/issue/33836.
	PingCmd: "ls-remote {scheme}://{repo}",

	RemoteRepo: gitRemoteRepo,
	Status: gitStatus,
	}

Proposal: Allow GOPRIVATE to provide source repository URI

Rationale:

Related:

Proposal: