proposal: cmd/go: module semantics in $GOPATH/src

[bcmills]

(Caveat: I'm not sure whether this is actually a good idea, and I'm really not sure whether it's feasible in time for the 1.11 experiment.)

In this golang-dev message, @rsc explains why GO111MODULE=auto doesn't work inside $GOPATH/src:

in auto mode we're trying not to break the existing meaning of working in the non-module-aware GOPATH/src/A: it's no good to redefine what the B half of it means.

I think there is a way to make modules work within $GOPATH/src without changing the existing meaning of $GOPATH/src.

When GO111MODULE=on or GO111MODULE=auto,

1. When the user runs go get mod/pkg@version within $GOPATH/src, update the contents of $GOPATH/src to match the versions implied by that dependency.

   • Download (to $GOPATH/src) any modules needed to satisfy the (transitive) imports of mod/pkg, just like the old go get but taking versions into account.
     • If any of those packages have been modified, go get should fail (the same as it does today if you run go get -u pkg with modified dependencies).
   • If any other modules in the build list are present in $GOPATH/src, update their contents too — even if they are not needed to satisfy imports.
   • Leave the $GOPATH/src contents unchanged for any module that was removed from the build list entirely.

2. If a build within a module in $GOPATH/src imports a package from a module that is not present in $GOPATH/src, copy that module into $GOPATH/src (as go get would do).

3. Whenever any package within any module in $GOPATH/src is built, update its go.mod file to reflect the actual contents of $GOPATH/src. That way, any future go commands within that module will produce the same contents as the current build.

   ⚠ This is the difficult part of this proposal!

   • For each module involved in the build, compare the contents of that module's go.mod file with the contents of its (transitive) dependencies in $GOPATH/src, ignoring any modules that are not present (e.g., because they were not needed to satisfy transitive imports).
   • Keep the original require directives from that module's go.mod file. Add replace directives for any (transitive) dependencies that do not match the versions implied by that module's go.mod. If the source code stored in $GOPATH still matches a committed (and non-excluded) version, use that version as the replacement; otherwise, use the local filesystem path.
   • Since replace directives do not affect other modules, ignore changes to them when comparing the contents of dependencies, including via go.sum checksums.

4. If a build within a module in $GOPATH/src imports a package that is present in $GOPATH/src but does not have an associated module, add require and replace directives (pointing into $GOPATH) to the go.mod files for all affected modules, as if there were a top-level go.mod for $GOPATH/src itself.

   module github.com/bcmills/module-with-unsatisfied-imports
   require GOPATH/src v0.0.0
   replace GOPATH/src => /home/bcmills/src
   

I believe that those steps maintain the essential properties of both $GOPATH and modules. Namely:

1. The source files in $GOPATH/src are exactly the files used for builds within $GOPATH/src.
2. All source files used for builds within $GOPATH/src are present in $GOPATH/src.
3. The go.mod files within $GOPATH/src accurately describe the sources used for builds.
4. The go.mod files within $GOPATH/src include requirements for all packages imported during builds.

(CC @FiloSottile @myitcv @natefinch)

[myitcv]

In this golang-dev message, @rsc explains why GO111MODULE=on doesn't work inside $GOPATH/src:

@bcmills do you mean GO111MODULE=auto here?

Because GO111MODULE=on does work within GOPATH, it simply enforces that everything be a module.

Instead I think with GO111MODULE=on having module mode be enabled within GOPATH by the presence of go.mod file is probably more natural.

My unease with the proposal above is the complexity in terms of implementation and understanding, the latter of which is probably most "critical".

[bcmills]

@bcmills do you mean GO111MODULE=auto here?

Yes, thanks! (Edited to fix.)

[bcmills]

Because GO111MODULE=on does work within GOPATH, it simply enforces that everything be a module.

That's true, but it breaks the previous invariants of working in GOPATH — namely, that the contents of packages throughout $GOPATH/src match what goes into the build.

For example, users might expect that they can use modules-in-GOPATH as a way to keep module definitions up-to-date while still being able to use tools that haven't been updated with module support. That doesn't actually work today: those tools see contents in GOPATH that might not actually match the active modules, and if users want to fix that they'll have to check out every active module explicitly.

I suppose we could build a standalone tool that does that, and go mod -vendor to some extent already does, but part of the point of version support in the go tool is to make module maintenance relatively seamless for users already accustomed to the go tool.

[myitcv]

That's true, but it breaks the previous invariants of working in GOPATH — namely, that the contents of packages throughout $GOPATH/src match what goes into the build.

But if this is a documented behaviour, is breaking this invariant so bad? Because using "on" is an override in the first place. It's a good point about tooling accidentally "working" (but not) but again with this setting we can assume the user knows what they are doing and document as much, i.e.the requirement for using module aware tooling?

[myitcv]

And given we're moving away from GOPATH, breaking the invariant feels even less bad!

[myitcv]

Just spotted @bcmills, but think the issue title needs updating too?! 😄

[bcmills]

But if this is a documented behaviour, is breaking this invariant so bad? Because using "on" is an override in the first place.

I, for one, am fairly likely to set GO111MODULE=on in my .bashrc or .profile. It's an override, but not an intrusive one. (It's not like, say, needing to pass an explicit flag to every command.)

It's good to document potentially-confusing behavior, but better still to make the behavior less confusing. I hope this proposal would do the latter, although it's possible that implicitly changing go.mod files throughout $GOPATH/src could be equally confusing.

It's a good point about tooling accidentally "working" (but not) but again with this setting we can assume the user knows what they are doing and document as much, i.e.the requirement for using module aware tooling?

I think you underestimate my ability to forget about configuration options I've applied. 🙂

[bcmills]

And given we're moving away from GOPATH, breaking the invariant feels even less bad!

I'm still not 100% convinced that getting rid of GOPATH entirely is a good idea.

GOPATH currently allows you to assemble a lot of different “modules” into something that resembles a “monorepo” and iterate locally on that. For example, you can add a feature in module A, use it in module B, and fold the result into module C for testing, all before you commit to a stable API for module A. That's especially important when refactoring module boundaries: for example, if you split one module into two modules with cyclic dependencies, you ought to test them together before you commit them.

That's possible with the current GO111MODULE support, but requires a lot of manual intervention: you have to add the appropriate replace directives in B and C pointing to the local copies of A and B, and you have to remember to back them out and replace them with real versions before you commit those changes upstream.

Or, perhaps you instead use -getmode=vendor and make changes in the /vendor directory — but then you have to remember to extract those changes and patch them back into the upstream modules, and that introduces more opportunities for error (e.g., applying the patch to the wrong baseline version).

---

So instead of getting rid of GOPATH in the module world, perhaps we could use it to group together a set of interdependent modules. If we do the version lookups carefully, that could even add the very nice property that the updated versions used for local development match the tags to be committed upstream.

For example, if I add a local v1.1.0 tag to the local repo for module C, I'd like to be able to test that the require C v1.1.0 in B's go.mod file can actually import the result before I push them both upstream. (For a concrete example, see rsc.io/quote v2.0.0, which accidentally failed to update tho module path before push.)

[rsc]

For Go 1.11, GO111MODULE=auto really needs to have no effect on commands people run in GOPATH. Otherwise we break people who have not opted in to modules. Also GO111MODULE is meant only as a transition mechanism and will go away like GO15VENDOREXPERIMENT did: we can't add anything to it that we're not comfortable throwing away in Go 1.13.

The plan described here is aiming at trying to provide a file system checkout of dependencies that can be edited to affect the operation of the original. That's an important workflow to enable but overloading GOPATH is probably the wrong approach: really any such tree of files should be scoped to working on one specific module instead of the more diffuse "this is the whole world" of GOPATH.

There was an interesting discussion on golang-dev this morning, and I'm sure we'll learn more about what we need along those lines as we use modules more.

Closing this issue, though.