cmd/go: flags to control changes to go.mod, go.sum

Question

cmd/go: flags to control changes to go.mod, go.sum

jayconrod opened this issue 6 years ago · 91 comments

Proposed changes

I propose we add three new flags to go subcommands that deal with modules.

-modfile=go.mod - instead of reading and writing go.mod from the current directory or a parent directory, the go command would read and write the specified file. The original go.mod would still determine the module root directory, so this file could be in any directory (perhaps /tmp). It would be an error to use this flag when no actual go.mod file is present.
-sumfile=go.sum - instead of reading and writing go.sum from the directory containing go.mod, the go command would read and write the specified file. It would be an error to use this flag when no original go.mod file is present.
-g - "global mode" - the go command would behave as if no go.mod file were present.

These flags would be allowed when GO111MODULE is set to on or auto (the current default) and rejected when GO111MODULE is set to off. -modfile and -sumfile both require an actual go.mod to be present, so modules must be enabled when they're used. -g does not require an actual go.mod to be present, and in auto mode, it implies that modules are enabled.

Background

The go command updates go.mod and go.sum after any command that needs to find a module for a package not provided by any module currently in the build list. This ensures reproducibility: if you run the same command twice, it should build (or list or test) the same packages at the same versions, even if new versions have been published since the first invocation.

For example, if you run go build ., and a .go file in the current directory imports example.com/m/pkg which is not provided by any known module, the go command will add a requirement on the latest version of the module example.com/m to go.mod. Future runs of go build . will produce the same result.

While these updates are usually helpful, there are many situations where they're not desirable.

Selected issues

gopls (various issues)

gopls loads information about source files in a workspace using golang.org/x/tools/go/packages, which invokes go list. gopls may also run go list directly. In either case, gopls may trigger changes to go.mod and go.sum. This may be caused by user actions that seem unrelated to building anything, for example, opening a file. go.mod appears to change mysteriously on its own, and users don't realize gopls is triggering it.

It's not usually important that the information gopls loads is reproducible; files it operates on are frequently changing. However, it is important that when it resolves an unknown import path to a module, it doesn't need to do so repeatedly since this can add a lot of latency, especially on slow connections.

gopls could set -modfile and -sumfile to temporary copies of the original go.mod and go.sum. The original go.mod and go.sum would not be modified (until the user explicitly runs a command like go build). Resolved module requirements would stay in the temporary files so they would not need to be resolved again.

#25922 - clarify best practice for tool dependencies

Developers need a way to express module requirements that aren't implied by package imports. This is especially useful for tools invoked by go generate. Authors can add tool requirements to go.mod manually or with go get, but these requirements are erased by go mod tidy.

The current recommendation is to create a tools.go file, tag it with // +build tools, then import main packages of needed tools. tools.go will never be built because of the tag, but go mod tidy will read the imports and preserve the requirements. This feels like a hacky workaround rather than a best practice. It also pushes requirements which may not otherwise be needed on downstream modules.

A better solution would be to keep a separate go.tools.mod file with tool requirements, then point to that with -modfile=go.tools.mod when running commands that require tools.

#26640 - allow go.mod.local to contain replace/exclude lines

This is a feature request to keep some go.mod statements out of source control. It's frequently useful for module authors to check out dependencies and point to them with replace statements for local development and debugging. These statements shouldn't necessarily be exposed to users or other developers on the same project though.

Setting -modfile=go.local.mod and -sumfile=go.local.sum would solve this problem, at least partially. The two files could be copied from the regular go.mod and go.sum files and added to .gitignore. Note however, that these local files are used instead of the regular files, not in addition to, so some synchronization might be required.

#30515 - offer a consistent "global install" command

Developers want to be able to install tools from any directory, regardless of the requirements of the current module. go get tool@version may update the current go.mod, so users need to change to a temporary directory without a go.mod file to run commands like this. Tool authors need to be careful when writing installation instructions because of this.

The -g flag would address this issue. It would tell the go command to run as if it were outside any module. Tool authors could write go get -g tool@latest in their installation instructions: this would install the latest version of the tool, regardless of the current directory.

Note that "missing go.mod" is being reconsidered (#32027), so the actual semantics of -g may change: this issue is just about ignoring the current module.

#33710 - module mode removes concept of global docs

In module mode, go doc example.com/pkg prints documentation for the packages named on the command line at the same version they would be built with. Like go build, go doc may add or update requirements in go.mod. This may be undesirable, especially if you're using the documentation to decide whether you want to depend on a package that is not currently imported.

The -g flag would partially solve this. The current module would be ignored, and "global" documentation would be shown.

Note that go doc does not currently work in "missing go.mod" for packages outside std. #33710 would need to be fixed, but -g would provide a useful way to access that mode.

Other related issues

There are a large number of open issues about unexpected and unwanted go.mod changes. The flags suggested here won't solve all these problems, but they provide useful context.

#26977 - go mod why adds a go.mod line
#29452 - go list has too many (more than zero) side effects
#29869 - 'go list' should not resolve or record modules that are not relevant to the requested output fields
#31372 - 'mod verify' does not respect -mod=readonly
#31999 - x/tools/gopls: support go.mod files
#32027 - rethink "missing go.mod" mode
#33326 - go install: don’t fail when go.mod can’t be updated on a read-only system
#34450 - go mod download support -readonly

Answer 1 · 2019-09-24T20:18:37.000Z

cc @bcmills @rsc @stamblerre @ianthehat

Answer 2 · 2019-09-24T20:33:19.000Z

-g - "global mode" - the go command would behave as if no go.mod file were present.

Per #32027 (comment), we'll probably want to make the go command less willing to resolve dependencies when no go.mod file is present.

If we do that, presumably the -g flag will allow the go command to resolve the transitive dependencies of the modules containing the packages listed on the command line. Should -g also allow the go command to resolve missing dependencies found in the import statements of those packages?

Answer 3 · 2019-09-24T20:36:11.000Z

The original go.mod would still determine the module root directory,

but

These flags would be accepted when GO111MODULE is set to on and rejected when GO111MODULE is set to off.

If GO111MODULE is on but there is no go.mod file above the current working directory, what would we use as the module root?

Answer 4 · 2019-09-24T21:51:28.000Z

Per #32027 (comment), we'll probably want to make the go command less willing to resolve dependencies when no go.mod file is present.

If we do that, presumably the -g flag will allow the go command to resolve the transitive dependencies of the modules containing the packages listed on the command line. Should -g also allow the go command to resolve missing dependencies found in the import statements of those packages?

I think -g should just make the go command do whatever it would do without a go.mod file. So if go run x.go resolves transitively imports and succeeds, go run -g x.go would do the same within a module. But if we change go run x.go to fail, then go run -g x.go would also fail.

Answer 5 · 2019-09-24T21:57:08.000Z

If GO111MODULE is on but there is no go.mod file above the current working directory, what would we use as the module root?

I contradicted myself a bit here. Updated the text. -modfile and -sumfile require an actual go.mod to be present in order to set the module root directory.

Answer 6 · 2019-09-25T01:20:19.000Z

Regarding the file flags: I think having to specify both -modfile and -sumfile is cumbersome. I can't think of any compelling reason to want to share go.sum with any other module, since go mod tidy will throw away the unnecessary lines anyway. So I would suggest that at a minimum, -sumfile be derived from -modfile if it's not set. A more extreme option would be to specify a single prefix, say -modprefix, and add .mod and .sum to it as needed. That may be too strange, though.

This is also a good precedent to set in case there's ever a third file, since nobody will know to override its location before it exists.

Answer 7 · 2019-09-25T11:29:32.000Z

@jayconrod regarding the "gopls (various issues)" motivation. I'm not entirely clear on what the issue here is and hence why -g/-modfile/other is a solution.

Is the problem that people are not used to cmd/go having side effects in module mode?

Because for me, I would expect that anything I do inside my editor, e.g. adding an import for a package whose module is not in my go.mod, could have side effects on the main module.

Could you give a bit more background?

Answer 8 · 2019-09-25T11:50:21.000Z

#30515 - offer a consistent "global install" command

While not part of the original issue, the consensus for a while seemed to be that a "global" install should also obey replace directives. At least, this is what @ianthehat said in #30515 (comment). Has the team's position generally changed to not obeying replace directives at all?

I'm also a bit confused by the multiple flags, like @heschik. How about a -modroot=path flag? It would roughly be equivalent to cd path && go <args>.

Answer 9 · 2019-09-25T11:57:47.000Z

Possibly worth reiterating that replace directives come in two forms:

directory-based
non-directory-based

~~I think the consensus that @mvdan refers to above was on the latter being applied.~~

See @mvdan's comment following this one.

Answer 10 · 2019-09-25T12:28:05.000Z

Sorry, I should have clarified. There were initially some comments about applying non-directory replace directives, but the consensus seemed that we either apply all of them or none of them - to not add a third build mode. @ianthehat's comment that I linked seemed to lean towards applying all replace directives.

Answer 11 · 2019-09-25T13:18:51.000Z

Personally, I'm in favour of applying directory replacements only when the source is in a user-controlled directory (as opposed to in global mode where the source is in the module cache). But I feel strongly that installs in global mode should respect other replacements, FWIW.

Answer 12 · 2019-09-25T13:54:15.000Z

@mvdan, @rogpeppe: see #30515 (comment) and #30515 (comment).

Furthermore, given that the proposed semantics of the -g flag are to do whatever we would do if outside of a module, the question of whether or how to apply replace directives seems orthogonal (and a bit off-topic). #31173 is probably a more appropriate venue for that discussion.

Answer 13 · 2019-09-25T13:57:24.000Z

It would be an error to use this flag when no actual go.mod file is present.

We should probably make that even stronger: the module directive in the replacement go.mod file should specify the same module path as the actual go.mod file. (Otherwise, if we're building packages within the module, we will end up resolving what should be module-local imports by looking for an external module.)

Answer 14 · 2019-09-25T14:06:10.000Z

regarding the "gopls (various issues)" motivation. I'm not entirely clear on what the issue here is and hence why -g/-modfile/other is a solution.

Is the problem that people are not used to cmd/go having side effects in module mode?

Because for me, I would expect that anything I do inside my editor, e.g. adding an import for a package whose module is not in my go.mod, could have side effects on the main module.

@myitcv, @ianthehat and @stamblerre have told me there's no canonical issue for this, but they've had to close a lot of issues as "working as intended", pointing to #29452, and they explain this frequently on Slack.

They mentioned one egregious example where someone in a clean workspace switched branches, then tried to switch back but were unable to because go.mod had uncommitted changes. Their editor (and gopls) was open in the background. It had detected changes in open files, run go list indirectly, and modified go.mod as a result.

-modfile would have made these changes to a temporary go.mod file instead of the go.mod in the main repo. #31999 is about supporting go.mod files in gopls, and while there aren't any details there yet, part of the plan is to provide easy ways to make changes and upgrades. It would be difficult to do that without -modfile.

Answer 15 · 2019-09-25T14:08:25.000Z

Regarding the file flags: I think having to specify both -modfile and -sumfile is cumbersome. I can't think of any compelling reason to want to share go.sum with any other module, since go mod tidy will throw away the unnecessary lines anyway. So I would suggest that at a minimum, -sumfile be derived from -modfile if it's not set. A more extreme option would be to specify a single prefix, say -modprefix, and add .mod and .sum to it as needed. That may be too strange, though.

This is also a good precedent to set in case there's ever a third file, since nobody will know to override its location before it exists.

I like the simplicity of just saying what both files should be, but I couldn't actually come up with a scenario where you'd want to set -modfile without -sumfile.

How about this: there's no -sumfile flag. If -modfile is set to M, then the sum file is strings.TrimPrefix(M, ".mod") + ".sum".

Answer 16 · 2019-09-25T14:13:31.000Z

I'm also a bit confused by the multiple flags, like @heschik. How about a -modroot=path flag? It would roughly be equivalent to cd path && go .

I don't think that solves the same set of problems. -modfile would still use the location of actual go.mod file to set the module root directory, not its argument. -modfile just redirects reads and writes, providing a way to control changes.

Answer 17 · 2019-09-25T14:25:55.000Z

While not part of the original issue, the consensus for a while seemed to be that a "global" install should also obey replace directives. At least, this is what @ianthehat said in #30515 (comment). Has the team's position generally changed to not obeying replace directives at all?

After thinking through all the weird edge cases and trying out a bunch of things, I came to the conclusion that applying replace directives is a confusing and dangerous ball of scary, and the only sane thing to do is not to apply them, in fact the only sane thing to do is to never ever check replace directives in to your repository in the first place.
Part of the benefit of this proposal is it allows for a workflow that makes that a much easier goal, by allowing an alternate go.mod that has the replace directives in it but is not used by default.

I'm also a bit confused by the multiple flags, like @heschik. How about a -modroot=path flag? It would roughly be equivalent to cd path && go <args>.

That would not allow most of the useful patterns (having multiple .mod files in the same directory that you can choose between, or a cache directory with modified .mod files for a bunch of different modules etc)

It would also mean that you have to specify what happens in a bunch of interesting edge cases (thinks like relative paths, are they from the original module or the alternate root?)

I think that specifying the .sum file would/should be incredibly rare, leaving it as the original one would be fine for the majority of use cases.

Answer 18 · 2019-09-25T14:26:49.000Z

@mvdan @rogpeppe

On replacements:

We should apply all of them or none of them. A third mode would cause confusion. (#30515 (comment)).
In order to apply file path replacements, we'd need to check out a whole repo. That's very different from what go get does now (especially when a proxy is in use), and there are many ways it could fail. We also won't get reproducible builds if replacement paths point outside the repository.

Personally, I think the downsides of (2) outweigh the benefits, and it's better to have something very simple like -g.

Answer 19 · 2019-09-25T14:33:54.000Z

How about this: there's no -sumfile flag. If -modfile is set to M, then the sum file is strings.TrimPrefix(M, ".mod") + ".sum".

Maybe split the difference? If the original is go.mod and -modfile is M.mod, first check whether M.sum exists: if so, use (and update) it, and otherwise use (and update) the original go.sum.

As far as I can tell, that handles both the tools.go.mod case and the /tmp/some-gopath-dir/go.{mod,sum} case.

Answer 20 · 2019-09-25T15:03:08.000Z

Maybe split the difference? If the original is go.mod and -modfile is M.mod, first check whether M.sum exists: if so, use (and update) it, and otherwise use (and update) the original go.sum.

As far as I can tell, that handles both the tools.go.mod case and the /tmp/some-gopath-dir/go.{mod,sum} case.

@bcmils Could work, but it seems a little subtle. If a module only depends on std and has no go.sum file, a naïve tool might copy go.mod to /tmp without creating an empty go.sum file. It would then unexpectedly modify the temp go.mod and the original go.sum.

Answer 21 · 2019-09-25T15:19:46.000Z

@jayconrod re #34506 (comment)

-modfile would have made these changes to a temporary go.mod file instead of the go.mod in the main repo

But what about the situations where you do want gopls to have these side effects? i.e. adding an import for a package whose module is not in my go.mod.

Answer 22 · 2019-09-25T15:30:08.000Z

I think -g should just make the go command do whatever it would do without a go.mod file.

Without a go.mod file, go get example.com/some/really/old/tool (that is, some tool without its own go.mod file) should probably fail, rather than re-resolving the latest transitive imports of that tool and discarding the result.

On the other hand, I think it is probably reasonable to expect go get -g example.com/some/really/old/tool to succeed, even if it is consistently slow.

Answer 23 · 2019-09-25T16:47:17.000Z

But what about the situations where you do want gopls to have these side effects? i.e. adding an import for a package whose module is not in my go.mod.

@myitcv It would be up to gopls and the editor to provide a sensible way to do that. Perhaps it could show a warning that go.mod is incomplete and provide a quick fix to add the missing requirement. Or it could update go.mod when a file is saved if it adds new imports.

There are situations where modifying go.mod is not wanted, and -modfile provides gopls with a knob that controls when those modifications happen.

Answer 24 · 2019-09-25T17:03:58.000Z

Without a go.mod file, go get example.com/some/really/old/tool (that is, some tool without its own go.mod file) should probably fail, rather than re-resolving the latest transitive imports of that tool and discarding the result.

On the other hand, I think it is probably reasonable to expect go get -g example.com/some/really/old/tool to succeed, even if it is consistently slow.

@bcmills There's a lot of nuance here. Assuming no go.mod is present, should go get example.com/tool when example.com/tool does have a go.mod file? What if the go.mod file is missing some requirements?

Should -g have this variation in behavior for go get only or for other commands, too?

Answer 25 · 2019-09-25T21:16:31.000Z

Assuming no go.mod is present, should go get example.com/tool when example.com/tool does have a go.mod file? What if the go.mod file is missing some requirements?

I'm not sure. Probably:

Without -g and outside of a module:

go get should resolve the latest version of the requested packages, compute the resulting build list, try building the requested packages, and error out if any of the import statements it encounters cannot be resolved using that build list.

That way, the number of latest lookups (and, potentially, fetches) is O(N) with the arguments passed to go get, and on successive runs everything else can be resolved using the local module cache.
Without -g and inside of a module:

go get should resolve the latest version of the requested packages, add them to the main module's build list, and resolve and add any additional dependencies as needed during the build, recording the result in the main module's go.mod file.

That still keeps the number of latest lookups down to O(N) with the arguments in the steady state, since any subsequent go get will use the versions recorded in the go.mod file (which will be in the local cache).
With -g and inside of a module:

The user has explicitly told us not to record the result of version resolution.

If the latest version of the module(s) containing the requested packages does not specify all needed dependencies, we have two possible options:
1. We could reject the go get request and ask the user to retry within a module. Most likely they'll just set -modfile=$(mktemp) instead of -g and run the command again, in which case we've added no value.
2. We could resolve the full import graph and throw away the results. But I think in most cases the user won't be running go get -g on the same module repeatedly, so that's probably fine. (In contrast, the user may reasonably run go run foo.go on the same source file repeatedly, so it's more important not to discard dependency information there.)
With -g and outside of a module:

It's the same situation as inside of a module, but instead of -modfile=$(mktemp), they'll likely do pushd $(mktemp -d); go mod init ugh; go get […]; rm -r .; popd.

Answer 26 · 2019-09-25T21:20:28.000Z

Should -g have this variation in behavior for go get only or for other commands, too?

Do we need to support the -g flag at all for other commands? I was assuming it would be a get-specific flag.

Answer 27 · 2019-09-26T13:13:43.000Z

It would be up to gopls and the editor to provide a sensible way to do that. Perhaps it could show a warning that go.mod is incomplete and provide a quick fix to add the missing requirement. Or it could update go.mod when a file is saved if it adds new imports.

This creates too much of a disconnect with the other go commands to my mind. By way of analogy, this would be equivalent to go test failing because go.mod is incomplete, and then requiring the user to run some specific command to fix it before again running go test.

~~Hence @bcmills's comment makes sense to me:~~

~~Do we need to support the -g flag at all for other commands? I was assuming it would be a get-specific flag.~~

Updated: @bcmills was talking about -g and @jayconrod was referring to -modfile

Answer 28 · 2019-09-27T14:58:23.000Z

@jayconrod will your proposed change still be compatible with the advice given at:

https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module

(I'm guessing yes, but just want to make sure).

Some projects have migrated to that approach such as this:

https://github.com/atlassian/smith/blob/master/tools.go

But there are other projects which still do go get in a Makefile inside the module directory to fetch tools.

Answer 29 · 2019-09-27T15:09:21.000Z

The proposal has no effect on that approach, but...

That approach is a horrible hack with all sorts of horrendous caveats and problems. I would argue strongly that it was okay as a temporary measure in go1.11 but it is already bad advice today and would become much worse advice when the features in this proposal are available.

Answer 30 · 2019-09-27T15:30:45.000Z

That approach is a horrible hack with all sorts of horrendous caveats and problems.

[citation needed]

#33926 is the only issue I'm aware of describing a concrete problem with the // +build tools approach, and the concrete problem cited there (#33926 (comment), an incompatibility between gopls and one of its dependencies) was due to a breaking change made in one of the tool's non-semantically-versioned dependencies.

Answer 31 · 2019-09-27T17:14:07.000Z

Some notes from a few out-of-band conversations with @bcmills @ianthehat and @myitcv:

-modfile

In evaluating this feature request, it would be helpful to have a better understanding of how gopls would use it. Clearly, there are situations where it should and should not modify go.mod. Some extra logic will be required to manage and synchronize both the actual and temporary go.mod files.
- @ianthehat @stamblerre Could you comment here with ideas on when this feature would be used and how that would work in gopls? What would the user experience look like, ideally? Is any of this a v1.0.0 blocker?
It seems like most of the time, we want a separate file that is used in addition to the actual go.mod, not instead of. gopls needs to manage additional requirements. For local replacements, we still want the original set of requirements. For tool dependencies, we want additional requirements.
- Having -modfile always augment the actual go.mod seems like it loses flexibility though. What if you actually want something different?
- An idea: we could add a statement (say include) that pulls in statements from another go.mod file.
- A statement like this would only work in the main go.mod file though. Like replace and exclude, it would be ignored in remote modules. It would add a lot of complication to the proxy protocol and sumdb logic to support include in general.
- Maybe it could be done as part of the flag? -modfile=+go.local.mod or -modfile=go.local.mod,go.mod would both mean "read both go.mod and go.local.mod and write changes to go.local.mod".

-g

Note that -modfile and -g are separate features that solve different issues, though they are both related to go.mod modifications. I probably should have filed two issues for these though.
It's hard to evaluate -g because we haven't decided what "missing go.mod" mode should do (#32027).
- At the moment, @bcmills and I are thinking that it should be possible to build packages in std and ad-hoc packages comprising .go files listed on the command line that only import packages in std. Any build, run, list, test, doc command that requires resolving an import path to a module would fail outside of a module (I'll post a comment on that issue).
- If we go with that solution, there's no point in those commands supporting -g, since they won't usually be able to do anything useful.
- So -g may as well be a special flag for go get that is tailored to fix #30515.
We haven't been able to agree on a solution for #30515 though. The question is what to do with replace directives.
- replace directives should be used for 1) temporarily forking a module to depend on a fix or feature that hasn't been released upstream, and 2) local development, especially of changes that affect multiple modules. In the context of #30515, only (1) is important.
- Technically, we could apply replace directives that replace modules with other modules. However, we cannot safely apply replace directives that replace modules with directories, since paths may point outside the module or outside the current repository. See #30515 (comment) and the comments below. We don't want to introduce a "third mode" where some replace directives are applied but not others, since this adds complication for module authors (increased test surface) and in the go command (lots of opportunity for bugs).
- An idea: go get -g could apply module replace directives in the module providing packages named on the command line. It would reject file replace directives with an error. So replace directives would never be partially applied.
- gorelease could warn module authors if they are about to publish a version that includes file replace directives in the main go.mod. File replace directives could be kept in a separate go.local.mod, referenced with -modfile.
- This seems like a lot of pieces to fit together, so perhaps initially for 1.14: go get -g rejects all replace directives with an error. We could enable module replace directives later on without breaking anyone.

Answer 32 · 2019-10-07T18:12:03.000Z

@myitcv

But what about the situations where you do want gopls to have these side effects? i.e. adding an import for a package whose module is not in my go.mod.

@jayconrod

Some notes from a few out-of-band conversations with @bcmills @ianthehat and @myitcv:

-modfile

* In evaluating this feature request, it would be helpful to have a better understanding of how gopls would use it. Clearly, there are situations where it should and should not modify go.mod. Some extra logic will be required to manage and synchronize both the actual and temporary go.mod files.
  
  * @ianthehat @stamblerre Could you comment here with ideas on when this feature would be used and how that would work in gopls? What would the user experience look like, ideally? Is any of this a v1.0.0 blocker?

I don't think it's obvious gopls should ever modify my go.mod and go.sum. To me, gopls is an extension of my editor, and my editor should not change any files I didn't ask it to. To me, it's tedious to constantly have to keep an eye out for irrelevant go.mod/go.sum changes before checking in new code.

Of course, if I run go commands from inside my editor (either directly or indirectly through a "run", "build" or similar editor command), then my go.mod/go.sum files are subject to change. But in those cases I'm no longer editing, I'm using the editor as a convenient command line shell.

So in conclusion, I see gopls always using -modfile pointing to its own go.mod file as the right decision.

Answer 33 · 2019-10-07T18:26:34.000Z

@eliasnaur so what about when go test modifies your go.{mod,sum} - is that also an error from your perspective? Because in that instance you haven't specifically asked it to modify your go.{mod,sum}, you've simply asked to run some tests. go mod edit is the specific command for editing go.mod. If go test did not operate in this way, having to run an additional command (which would require some manual effort) to add the relevant require directives would be tiresome to my mind because that's almost always what I want to do.

Per @jayconrod's request to @ianthehat and @stamblerre above, I think we need to establish what the workflow here would be from a user's perspective before doing something that to my mind goes against what the go tool is already doing.

Answer 34 · 2019-10-07T18:35:49.000Z

Another use case for this is go-fuzz. go-fuzz has to augment the source tree with its own dependencies during compilation, but there is ~zero value to the user in having the module containing those dependencies be present in their go.mod/go.sum. (cc @thepudds)

Answer 35 · 2019-10-07T18:38:03.000Z

To offer some input from a heavy gopls user: I'm very happy to let gopls modify my go.mod file as it pleases, since that means I can paste in import paths and have them actually resolve, even if I have to do cleanup duty later if go.mod contains something I didn't expect.

If it didn't have this ability, they'd break, and then more things end up breaking when I have to go run go in my terminal and gopls doesn't see that something has changed (leading to some bad state only a reload can fix, which is already far too common). I can see a future where gopls wouldn't have the ability to modify go.mod, but it'd really need to better handle watching for changes, and would be kinda disappointing to go from the workflow of "paste in an import and it just works" back to the old workflow of needing to make sure that everything is there before trying to use it.

Answer 36 · 2019-10-07T19:30:51.000Z

@eliasnaur so what about when go test modifies your go.{mod,sum} - is that also an error from your perspective?

No. I meant to cover go test by:

Of course, if I run go commands from inside my editor (either directly or indirectly through a "run", "build" or similar editor command), then my go.mod/go.sum files are subject to change. But in those cases I'm no longer editing, I'm using the editor as a convenient command line shell.

In other words, go commands modify my go.* files, as they should. I'm only talking about gopls and editing in general, which shouldn't.

To offer some input from a heavy gopls user: I'm very happy to let gopls modify my go.mod file as it pleases, since that means I can paste in import paths and have them actually resolve, even if I have to do cleanup duty later if go.mod contains something I didn't expect.

I agree, and that's why I haven't complained about gopls before this proposal. With -modfile, gopls can keep a private go.mod file and operate exactly as if it had modified your go.mod file.

Answer 37 · 2019-10-08T05:41:39.000Z

In other words, go commands modify my go.* files, as they should

My point was that for some people go test and other build commands (e.g. go list, go build etc) should not have these side effects: #29452. i.e. an explicit go mod edit commands should be required before running go test etc.

I'm not arguing for that position, just using it to point out that many people have normalised the fact that cmd/go build commands modify go.{mod,sum} in just the same way that I've normalised gopls modifying go.{mod,sum}.

With -modfile, gopls can keep a private go.mod file and operate exactly as if it had modified your go.mod file.

What would the workflow look like when I do want gopls to modify my go.{mod,sum}?

Having a second go.{mod,sum} introduces a second source of truth; the "real" one is used by cmd/go, the "fake" is used by gopls. How do we ensure they remain in sync? How do we point out conflicts?

Answer 38 · 2019-10-08T08:12:37.000Z

In other words, go commands modify my go.* files, as they should

My point was that for some people go test and other build commands (e.g. go list, go build etc) should not have these side effects: #29452. i.e. an explicit go mod edit commands should be required before running go test etc.

I'm not arguing for that position, just using it to point out that many people have normalised the fact that cmd/go build commands modify go.{mod,sum} in just the same way that I've normalised gopls modifying go.{mod,sum}.

Sure, I've also normalized that behaviour, by monitoring my go.* files like a hawk :)

Note that #29452 seems to make a distinction between building commands such as go test, go install, go build and query commands such as go list, go mod. So even with a hypothetical fix to #29452, your go test would still do what you expect.

With -modfile, gopls can keep a private go.mod file and operate exactly as if it had modified your go.mod file.

What would the workflow look like when I do want gopls to modify my go.{mod,sum}?

May I ask when and why you want gopls to modify you go.* files? And for each of the cases you do want its modifications, why aren't your inevitable go test, go run, go install or even go mod tidy good enough checkpoinst for actually changing your go.* files?

The only legitimate case I can see is for adding imports that you're going to keep, but how would gopls know the keepers from your spelling mistakes? I can't count the number of times gopls has added irrelevant dependencies to my go.* files just because I code completed or misspelled a package.

Having a second go.{mod,sum} introduces a second source of truth; the "real" one is used by cmd/go, the "fake" is used by gopls. How do we ensure they remain in sync? How do we point out conflicts?

My go.* files are the source of truth, while gopls' own files are for caching queries and pleasing its underlying go commands. If conflicts occur, its conflicting (or all?) changes to its internal go.* files should be wiped out.

Answer 39 · 2019-10-09T13:54:05.000Z

Note that #29452 seems to make a distinction between building commands such as go test, go install, go build and query commands such as go list, go mod. So even with a hypothetical fix to #29452, your go test would still do what you expect.

It's worth noting that go list is a build command; it takes build flags. go mod edit is I think the only go mod subcommand that is literally concerned with mechanically editing go.mod. All the other subcommands will cause a full resolution of dependencies that might result in changes to go.{mod,sum}.

The reason I bring up the go test example is because in another world we could easily have decided that go test should fail if go.{mod,sum} didn't contain the relevant entries for go test to proceed. In such a scenario, a go mod edit or similar mechanical update would have been required as a separate step before running go test again. As it is, go test does as little as it needs to to go.{mod,sum} for the build configuration under test. This feels natural to me, because it avoids the painful error message "you need to run X before continuing".

May I ask when and why you want gopls to modify you go.* files?

Because I want the type checking and analysis that gopls carries out to proceed in much the same way that go test does without me needing to intervene. Similarly, when gopls gains the power to help with things like code generation or other commands, I don't want to be having to manually intervene to run go mod edit to fix my go.mod or click "accept" on a code suggestion to add something to my go.mod.

The only legitimate case I can see is for adding imports that you're going to keep, but how would gopls know the keepers from your spelling mistakes?

I notice you skilfully sidestepped my question on the workflow of when I do want gopls to update my go.{mod,sum} 😉 I'm guessing however you want to manually "copy" the "right" changes from gopls's copy to the real copy?

I can't count the number of times gopls has added irrelevant dependencies to my go.* files just because I code completed or misspelled a package.

This could happen in any scenario: even with go test for example. go mod tidy is your friend here, and I think Rebecca added support for a formatting of go.mod files (which amounts to a go mod tidy from memory). I don't think manually copying/accepting the "right" changes is necessary here, the tools can do the work for us.

If conflicts occur, its conflicting (or all?) changes to its internal go.* files should be wiped out.

I'm certainly not clear what this conflict resolution algorithm would look like: can you sketch it out ?

Answer 40 · 2019-10-09T15:11:34.000Z

Note that #29452 seems to make a distinction between building commands such as go test, go install, go build and query commands such as go list, go mod. So even with a hypothetical fix to #29452, your go test would still do what you expect.

It's worth noting that go list is a build command; it takes build flags. go mod edit is I think the only go mod subcommand that is literally concerned with mechanically editing go.mod. All the other subcommands will cause a full resolution of dependencies that might result in changes to go.{mod,sum}.

I suppose that's what #29452 is about: go list is a build command, but feels like a query command.

However, in the context of gopls its use of go list is an implementation detail.

May I ask when and why you want gopls to modify you go.* files?

Because I want the type checking and analysis that gopls carries out to proceed in much the same way that go test does without me needing to intervene. Similarly, when gopls gains the power to help with things like code generation or other commands, I don't want to be having to manually intervene to run go mod edit to fix my go.mod or click "accept" on a code suggestion to add something to my go.mod.

That's a great example. I want to consent to go.mod changes because adding a dependency is not something I do lightly. From Rus Cox' Our Software Dependency Problem:

"A package, for this discussion, is code you download from the internet. Adding a package as a dependency outsources the work of developing that code—designing, writing, testing, debugging, and maintaining—to someone else on the internet, someone you often don’t know. By using that code, you are exposing your own program to all the failures and flaws in the dependency. Your program’s execution now literally depends on code downloaded from this stranger on the internet. Presented this way, it sounds incredibly unsafe. Why would anyone do this?"

In other words, completing or auto-generating code for me is ok to do more or less automatically, but or adding or changing my dependencies is not.

I notice you skilfully sidestepped my question on the workflow of when I do want gopls to update my go.{mod,sum} wink I'm guessing however you want to manually "copy" the "right" changes from gopls's copy to the real copy?

Whenever you consent to the changes. Whether that is a go build, go test on the command line, or clicking "run" or "accept" in a IDE. And perhaps there is a setting for "I accept all changes" if you really want to accept whatever gopls gives you during edits.

I can't count the number of times gopls has added irrelevant dependencies to my go.* files just because I code completed or misspelled a package.

This could happen in any scenario: even with go test for example. go mod tidy is your friend here, and I think Rebecca added support for a formatting of go.mod files (which amounts to a go mod tidy from memory). I don't think manually copying/accepting the "right" changes is necessary here, the tools can do the work for us.

Yes, but I consider go test and friends consenting to changes. It's not perfect (#29452) and I would personally have GOFLAGS=-mod=readonly if govim supported it.

If conflicts occur, its conflicting (or all?) changes to its internal go.* files should be wiped out.

I'm certainly not clear what this conflict resolution algorithm would look like: can you sketch it out ?

Keep an internal copy of go.* and use -modfile to point to them. Whenever the original go.* files changes, replace the copy.

Answer 41 · 2019-10-11T00:11:24.000Z

I'm curious as to whether #34829 would satisfy the gopls use-case less invasively. How important is it to resolve dependencies in uncommitted code, vs. tolerating (and possibly suppressing) errors for unresolved dependencies?

Answer 42 · 2019-10-11T18:03:56.000Z

Contemporaneous notes from a conversation with @ianthehat and @stamblerre:

When you add an import, either explicitly or by completion, gopls should update go.mod or offer to do so. The exact behavior needs evaluation.
If changes to go.mod don't happen automatically, they'd probably be suggested Quick Fixes on imports that aren't provided by any module in the build list.
If changes to go.mod happen automatically, they should happen when the file is saved, probably not as soon as the import is added (right after a completion).
A frequent problem: a user opens a file that transitively imports a package not provided by any module in the build list. gopls (via go list) adds a requirement for a new module. The user hasn't edited anything yet and is surprised that go.mod has changed.
How far could we get if go list -e -mod=readonly worked better (#34829)?
- It doesn't let us type check, since we can't load packages that transitively depend on missing packages.
- It doesn't let us suggest fixes (updates to go.mod). Everything is just broken.
- -modfile would let us add the module requirement to a separate go.mod, then suggest a change with the difference. We could type check everything.
How should -modfile be passed to go list?
- golang.org/x/tools/go/packages.Config.BuildFlags.
- Drivers are expected to know all the flags for go list, including -modfile. Drivers for older versions of the Go command and other build systems should filter this flag out or find something equivalent.

Answer 43 · 2019-10-11T20:49:06.000Z

@eliasnaur

go list is a build command, but feels like a query command.

I can sympathise with that point of view: it's certainly not obvious.

However, in the context of gopls its use of go list is an implementation detail.

This is true but it's a critical implementation detail. Whilst go/packages is the abstraction, go list is the very means by which the concept of modules, packages etc are defined for cmd/go-based build systems.

(That's not to say however that we're bound only by what cmd/go offers today, indeed that's what we're discussing here.)

I would personally have GOFLAGS=-mod=readonly if govim supported it.

I think this is a great suggestion for an option in govim. Indeed I just did a bit of experimentation and it already works if GOFLAGS=-mod=readonly is passed to the environment of gopls by govim: you get no modifications to go.{mod,sum} as a result of using Vim/govim, and changes made via cmd/go are correctly picked up by gopls by virtue of govim simulating file watch changes (whilst we wait for full gopls file watching support). We could add this today with a bit of a hack on the govim side, pending support for a workspace config option in gopls at a later date. I've created govim/govim#555 to track this for govim.

Whenever you consent to the changes. Whether that is a go build, go test on the command line, or clicking "run" or "accept" in a IDE.

I'm not clear how go build or go test is any more "safe" than things happening via gopls/go/packages/go list - they will end up having exactly the same side effects (if we consider them being run at the same point in time) and you're still blind to them unless you check your go.{mod,sum}.

Keep an internal copy of go.* and use -modfile to point to them. Whenever the original go.* files changes, replace the copy.

As you mention above, you'd either look to "accept" the changes by:

explicitly running go build, go test and friends
prompting the user

For option 1, this would mean that there is a possibility the subsequent go build/go test will result in a different changes to the go.{mod,sum} than those made in the -modfile copy. Because none of these commands (with the exception of go get) specifies a version of a dependency. This doesn't seem ideal.

For option 2, presumably you'd prompt for every change that gets made to the -modfile go.mod to "sync" it back to the original if the user accepts? If so, this feels like a very noisy workflow to me, especially when I don't get any such prompts when using go test etc.

Per @jayconrod's request to Rebecca and Ian (#34506 (comment)) and subsequent follow up in #34506 (comment), I think it's worth looking at the scenarios when a go.{mod,sum} change can occur:

incomplete go.{mod,sum} - i.e. go mod tidy not run
new import added in main module and code action run to run goimports equivalent (this happens on save in govim if so configured)
... others?

And also the use cases that motivated this discussion in the first place:

UC1 - people changing Git branches (as I understand it, this generally causes all sorts of problems)
UC2 - people just browsing projects, i.e. with no intention of making changes
UC3 - people who do not want to have gopls touch go.{mod,sum}
... others?

UC3 is covered above and and in govim/govim#555.

UC1 requires a significantly broader solution to my understanding, because changing files like this "underneath" the editor has all sorts of problems.

UC2 is totally valid to my mind, and building on the suggestion discussed between Jay, Rebecca and Ian, it might make sense (assuming the user is not interested in UC3) to not make any changes to go.{mod,sum} until the first user-initiated change is made to a file in the main module. This however creates a weird UX, because all type check errors that might show would then disappear with the addition, say, of a space to a comment. Alternatively, if it's detected the go.{mod,sum} is not tidy, then the user could be prompted to fix this as the workspace is opened.

Just to note, I'm very much in support of fixing workflow issues like this. My reservations are around a lack of clarity on the UI/UX, use cases that we're looking to fix, whether we will end up actually fixing those use cases, and potential issues with conflicts if we were to use -modfile.

Answer 44 · 2019-10-11T20:59:23.000Z

scenarios when a go.{mod,sum} change can occur

As I mentioned above, this happens with go-fuzz. We inject a dependency and execute a build, which then changes go.mod in a way that the user doesn't care about at all.

Answer 45 · 2019-10-11T21:12:27.000Z

@josharian - sorry, I totally missed that. My comments were more focussed on use cases from a gopls/editor perspective. Although I'll admit I don't really know whether there is overlap or not with go-fuzz in that respect.

Answer 46 · 2019-10-11T22:39:46.000Z

On Fri Oct 11, 2019 at 1:52 PM Paul Jolly wrote: > Keep an internal copy of go.* and use `-modfile` to point to them. Whenever the original go.* files changes, replace the copy. For option 2, presumably you'd prompt for every change that gets made to the `-modfile` `go.mod` to "sync" it back to the original if the user accepts? If so, this feels like a very noisy workflow to me, especially when I don't get any such prompts when using `go test` etc.

Do you mean that accepting every change is noisy? I read @jayconrod's comment

-modfile would let us add the module requirement to a separate go.mod, then suggest a change with the difference. We could type check everything.

as there would be a summary of all the changes which I could accept in bulk if/when I pleased. If I waited to accept changes, or (more likely) never accepted them, `gopls` would hopefully keep working.

Answer 47 · 2019-10-17T01:56:01.000Z

Here's another use case for this flag.

As part of #34867, I want to use a small Go program to do some fairly trivial json parsing. At the moment that I need that Go program to run, the local go.mod file is in a bad state. (I'm running the Go program as part of a script to make the go.mod file functional.) As a result, I can't use go run. If I could do go run -modfile=/sometempfile, then I could temporarily get past the broken go.mod and run the file.

(Yes, there are other ways to accomplish this, like setting GO111MODULE=off, or copying the Go program into a temp dir, manufacturing a trivial go.mod for it, building it, and running it. But this flag also provides a nice, easy simple workaround.)

Answer 48 · 2019-10-22T19:01:44.000Z

Change https://golang.org/cl/202564 mentions this issue: cmd/go: add -modfile flag that sets go.mod file to read/write

Answer 49 · 2019-10-22T20:39:55.000Z

A quick update:

I've implemented the -modfile flag in CL 202564. There will be no -sumfile flag. Instead, the name of the go.sum file will be derived from the name of the go.mod file by trimming the .mod suffix (if present) and add .sum, as @heschik suggested.

I'm planning to add a -g flag that would apply to go get. It will cause go get to behave as if it were outside a module. After changes made in #32027, it doesn't really make sense for -g to apply to other commands.

This interpretation of -g means there would be no main module, and replace and exclude directives will be ignored. We've discussed the tradeoffs of doing it this way here and in #30515. I'm starting to think that we should support both approaches. Perhaps it should be possible to pass a -g=main flag. This would restrict command line arguments to one main package (or perhaps packages from one "main" module) and would apply replace and exclude directives from that module. File replace directives would be rejected. WDYT?

Answer 50 · 2019-10-24T20:57:38.000Z

@jayconrod just so I'm clear, what decision on replace/exclude directives was taken in https://go-review.googlesource.com/c/go/+/203279?

Answer 51 · 2019-10-24T20:58:18.000Z

Change https://golang.org/cl/203279 mentions this issue: cmd/go: add go get -g flag to install tools in global mode

Answer 52 · 2019-10-24T21:14:36.000Z

@myitcv In https://golang.org/cl/203279, -g will mean "act as if there is no main module", and it will apply to go get only.

I alluded to having a -g=main flag in my earlier comment, which would have the semantics you wanted: command line arguments could only come from one module, and that module would be treated as the main module for the purpose of replace and exclude directives.

I think it makes sense to have two separate modes. It doesn't seem like everyone will be happy with either one. However, I don't think we'll be able to get -g=main done before the freeze (5 working days from now). The implementation will be more complicated than -g, some design work is needed, and we're thinking about some changes to replace for 1.15 which might affect this (proposal not ready yet).

Answer 53 · 2019-10-25T16:06:47.000Z

However, I don't think we'll be able to get -g=main done before the freeze

From the user's perspective, -g=main is what they'll want most of the time when doing "global installs" of programs. So I think adding -g now, without any replace directives, could be a bad decision in the long-term; the behaviour would be pretty much set in stone.

If we really want "act as if there is no main module" today, how about a simpler -modfile=none?

Answer 54 · 2019-10-25T18:20:45.000Z

From the user's perspective, -g=main is what they'll want most of the time when doing "global installs" of programs.

I respectfully disagree. From a user's perspective, what they'll want most of the time is a binary that builds in exactly the same configuration no matter where you build it from — that is, one that does not rely on checked-in replace or exclude directives at all.

When that condition holds, -g and the possible -g=main are equivalent.

That being the case, -g sets the right default by encouraging tool authors to get the changes they rely on merged upstream, or to explicitly fork packages for which they require divergent behavior.

Moreover, -g as drafted is much simpler to reason about: it means “pretend that I am not in a module” — nothing more, and nothing less.

Answer 55 · 2019-10-25T18:20:50.000Z

From the user's perspective, -g=main is what they'll want most of the time when doing "global installs" of programs.

@mvdan I'm not convinced this is true long-term. As module adoption increases and go command bugs are fixed, I hope we'll see fewer tools that rely on replace directives as fewer problems need to be worked around. replace directives should rarely be necessary in tagged releases.

-g=main will also cause problems for problems for projects that rely on replace directives with file paths. We cannot support those in downloaded modules. So for example, go get -g=main golang.org/x/tools/gopls would break today because there's a file replace directive for golang.org/x/tools.

If we really want "act as if there is no main module" today, how about a simpler -modfile=none?

So -modfile=none instead of -g? That could work, but -g seems like a simpler command line to me. Based on how we've resolved #32027, -modfile=none would only be meaningful for go get, so I'm not convinced -g should have that spelling.

Answer 56 · 2019-10-25T18:25:48.000Z

I think -modfile=none would be a mistake.

-modfile means “find the go.mod file at the root of my module and replace it with this other file.”

-modfile=none would therefore mean “find the go.mod file at the root of my module and replace it with none”. But what does it mean to replace a go.mod file with an empty one? What does that do to the module path and the semantics of the current directory?

-g, in contrast, means “pretend that I am not in a module”, which has a well-defined (if evolving) behavior.

Answer 57 · 2019-10-25T18:26:37.000Z

In particular, you could imagine combining -g and -modfile to mean “ignore the module containing the current directory, resolve the versions I'm passing you on the command line, and record your decision in this go.mod file.”.

Answer 58 · 2019-10-25T18:27:58.000Z

Somewhat tangential, but I don't think -g=main is a good idea for a flag. Like normal flags flags, the flag should either be boolean or not boolean. Otherwise -g main won't work as expected. (Discussed recently in the context of go test -shuffle over at #28592 (comment).)

Answer 59 · 2019-10-25T18:38:54.000Z

I agree with @mvdan that -g=main is always the functionality I want when installing global tools. If the upstream author needed to use replace and exclude directives to build a tool and tagged it in a release I can't imagine why anyone would then want to go get -g binary@version and have it ignore those directives.

Yes authors should get changes merged upstream in time for their release to avoid the replace when possible, but why punish the consumers of that binary? I realize there are technical challenges around this (failing on filesystem based replaces seems fine to me) but I really don't understand the use case for the current implementation of the -g flag. I'd much rather the current go get -g refuse to build a binary if the version its getting has any replaces or excludes rather then to ignore them. Then I'm not silently using the wrong version of the tool. If it fails, fine then at least I know I can git clone to build it.

Answer 60 · 2019-10-25T19:18:49.000Z

Directory-based (file path in the parlance above) replace directives are, as I have previously agreed, a different beast to non-directory replace directives. In a different world they could have been named dirreplace or only allowable in a go.mod.local. Therefore I agree with @jayconrod's position that a directory-based replace directive in a released go.mod could even be considered an error. It means nothing to the end user.

So working on the assumption they are an error, we won't seem them in a released tool's go.mod (this would need to be documented of course).

To my mind therefore applying non-directory replace directives does not introduce significant dissonance. Nor do I think that in the case we're discussing (i.e. -g, or whatever name we give it, as a way of a user installing a "global" tool) it's inconsistent with the cmd/go documentation:

Exclude and replace apply only in the main module's go.mod and are ignored in dependencies.

Because we're not talking about a dependency: it's effectively the main module and could be documented as such.

Nor is applying non-directory replace directives an acceptance that a change will not be upstreamed ever. Indeed the very existence of a replace directive is a very strong indicator that's exactly what the tool author is in the process of doing.

I totally accept that it's simpler to understand replace/exclude directives not being applied, but I don't think that leaves us in a better net position. Here's another perspective: not applying replace directives forces tool authors towards hard forking. This makes it harder to upstream the fix which might have exactly the opposite intended effect, i.e. people upstream less.

A replace directive (as overloaded/badly named as it may be) signifies a tool author's intent to try and carry on whilst they simultaneously try and upstream a change. Is replace being abused here? I don't think so - it's still being used for the "temporary" solution.

This being the "reality" in which I think we will find most tools, I therefore agree with @mvdan's comment. Furthermore that having -g and -g=main just creates more confusion from the tool author's/end user's perspective

Answer 61 · 2019-10-25T22:47:38.000Z

Change https://golang.org/cl/203557 mentions this issue: [DO NOT SUBMIT] cmd/go: add -g=main flag for 'go get'

Answer 62 · 2019-10-28T20:30:14.000Z

Replying to a comment from @myitcv on CL 203279.

Just a thought regarding the discussion on -g, -g=main, -modfile, -modcacherw etc.

I haven't exactly remained close to all of these discussions but it seems we're adding lots of new flags here, decisions which are permanent.

In the case of -g and -g=main I'm not clear we've arrived at the "right" answer.

With -modfile, I can see use cases, but we haven't experimented with these in gopls for example before making the change. It might be the experiment fails and we no longer need the flag.

With -modcacherw I'm even further from the discussion/change so I defer on that.

Hence I wonder whether documenting these flags as experimental (or dropping the flags altogether and instead having GOEXPERIMENT_XYZ env vars) would be a more prudent next step?

I'd prefer not to have experimental flags in the Go command as part of a regular release. Despite being labelled experimental, these features have a way of becoming difficult to change or remove simply by being used, discussed, and documented.

If we're moderately confident in the design, I think it's reasonable for these flags to be in the 1.14 beta. If it turns out they have problems, we can remove them before the release.

If we're not confident in the design, we can defer these until a future release. However, we've been discussing variations on -g on and off since March (#30515), and I'd love for us to ship something in 1.14 that most people are happy with.

Answer 63 · 2019-10-28T20:31:05.000Z

About -modfile

This seems like the less controversial of the two changes discussed here, so I won't say too much more about it. It's submitted last week as CL 202564. It's important that we verify this solves the problems set out to solve, primarily preventing gopls from unexpectedly editing go.mod files.

@stamblerre, @ianthehat Would you be willing and able to prototype functionality related to -modfile in gopls? It would be great to have some feedback on this flag before the 1.14 release candidate is cut in early January. I don't expect everything can be done by then, but it would be good to know if the basic flow of copying the go.mod file to a temporary directory and monitoring changes there works as intended.

Anyone else interested in using this for tools or managing non-production dependencies, please try it out at tip. Let me know if you run into problems, either here or in a new issue.

Answer 64 · 2019-10-28T20:37:36.000Z

About -g

This is the flag that's more useful to everyone on a day-to-day basis, but it's also the one we're having trouble agreeing on. I'll try and recap.

-g sets out to solve two problems. First, we need a way to install binaries globally using go get without touching go.mod in the current directory. Second, we need a way to install binaries while respecting replace and exclude directives in authors' go.mod files. -g as proposed here was meant to solve the first problem, leaving the second until later, but after discussion, it seems like we should agree on a solution to both before proceeding. Note that since we've resolved #32027, -g will be specific to go get, so it's fine if it does something that's not meaningful to go list for example.

replace directives are really the hard part. We can't apply replace directives with file paths on the right side, since the go command operates on module zip files, not repositories, and zips do not include replacement files. We don't want to apply some replace directives but not others, since that adds complication to the go command and to module authors' release processes. So it should be all or nothing: for any go get command, either all replace directives are applied, or all are ignored, or the command fails with an error because some replace directive could not be applied. I think we're all in agreement up to this point.

Currently, go get does not apply replace directives when outside of a module. If we start applying replace directives (rejecting those that we cannot apply), some commands that could be built before will no longer be buildable. So this behavior needs to be opt-in with a new flag. The question is whether that flag is -g. That is, should we try to solve both problems together with one flag?

This is really a question of user intent and module author intent. It's subjective so it's hard to know what the "right" choice is.

Several people in this thread and on #30515 have argued convincingly that replace directives will continue to be an important part of people's workflows, and installing binaries with replace directives applies is the common case we should optimize the CLI for.

I'm starting to agree with this point of view. Short-term forks of upstream modules are not going to disappear. My main concern about this was that the behavior was very different from go get outside a module without -g, but perhaps that's desirable. We can advise module authors not to include file path replace directives in go.mod files at released version; perhaps these can be managed with -modfile or simply removed at tagged commits.

There are a lot of smaller use cases that this version -g would not address though: explicitly ignoring replacements; using newer versions of indirect dependencies; installing multiple binaries from multiple modules. Are these common enough cases to need a separate flag? All of them can be worked around by running go get without -g outside a module (as in 1.13) or by using -modfile.

Answer 65 · 2019-10-29T09:14:28.000Z

Thanks for the recap @jayconrod. To clarify, with my earlier comment I didn't mean to say that I have a fully fledged plan for -g. Just that I think adding it now without figuring out what we want to do with replace directives would be a mistake in the long run. Users are going to be confused by having both -g and -g=main, and most will default to just using -g.

I agree that having something in 1.14 would be good though, which is why I was trying to think of less invasive changes like -modfile=none. Or perhaps a standalone "ignore the current module" flag would still be useful in 1.14, but I think it's a mistake to spend the -g name on it.

We have a tools call later today, so I think it's a good opportunity to have a quick chat about it then.

Answer 66 · 2019-10-29T15:12:08.000Z

I would like to push back on something this subtle being -g. That takes a one-letter flag for something that we can't even explain.

If we do -modfile=none instead, that I can explain: it is exactly equivalent to having an empty go.mod file that is discarded after the command runs. The only gotcha with this is that the command will be slow, because it can't cache any of its decisions in the go.mod file. This is how GO111MODULE=on used to work without a go.mod file, and it led to even quite astute users incorrectly concluding that "modules are slow". It should be documented that -modfile=none is highly non-recommended if it goes in.

All the discussion about replace directives seems difficult to reconcile to me. What if I do go get -modfile=none A B and A and B both have different go.mod with conflicting replaces? It would be much better not to do replaces at all. That's the rule for dependencies in any event.

Answer 67 · 2019-10-29T15:26:20.000Z

If we do -modfile=none instead, that I can explain: it is exactly equivalent to having an empty go.mod file that is discarded after the command runs.

-modfile=none may be pretty subtle, too. It may also be surprising because -modfile works in all build commands. For example, running go build -modfile=none ./foo within a module would build the package in directory ./foo, resolving imports to latest versions of modules, even if those imports refer to packages in the same module.

What if I do go get -modfile=none A B and A and B both have different go.mod with conflicting replaces?

This example seems clearer to me. It would have the same behavior as go get A B today when run outside of a module. It would resolve the latest versions of A and B, then run MVS to find a combined build list, then build packages from that. replace would be ignored.

Only -g would enable replace directives, and -g can't be used together with -modfile. It would require that packages listed on the command line come from only one module, and replace directives would only be applied from that module.

Answer 68 · 2019-10-29T15:29:15.000Z

I still believe -g is a mistake. It is overfitting to a handful of use cases, adding complexity without a clear model behind it.

Go has never been about solving every possible use case.
Go is about providing easy-to-understand composable building blocks so that users can construct what they need the vast majority of the time, without our planning it for them.

-g does not seem to me like it fits into that.

Answer 69 · 2019-10-29T16:24:34.000Z

It is overfitting to a handful of use cases

From my perspective, -g restores the main purpose for which I use go get, previously removed as part of the migration to modules.

Installing tools globally is not a niche use case. See previous comments here and here.

Answer 70 · 2019-10-29T18:10:48.000Z

I still believe -g is a mistake. It is overfitting to a handful of use cases, adding complexity without a clear model behind it.

Go has never been about solving every possible use case.
Go is about providing easy-to-understand composable building blocks so that users can construct what they need the vast majority of the time, without our planning it for them.

-g does not seem to me like it fits into that.

@rsc I agree that -g is not a composable building block. However, I think this is a common use case, and it may be common enough to optimize (overfit) for.

I'm planning to spend some more time researching use cases to understand whether this makes sense. We should quantify how often replace comes up and whether there are other, better solutions.

Answer 71 · 2019-10-29T18:11:17.000Z

From my perspective, -g restores the main purpose for which I use go get, previously removed as part of the migration to modules.

@cespare I think we all agree that installing tools globally should be supported. The question is how to do it in a way makes the most sense for the ecosystem.

In GOPATH mode, replace directives are not available, but every project can have a vendor directory. Multiple vendor directories caused a lot of problems that prevented projects from being composed. We want to avoid composability problems with modules as much as we can.

Answer 72 · 2019-10-29T18:13:49.000Z

Based on discussion here and in the tools meeting today (notes and recording available later), I don't think we're confident enough in the design of -g to proceed in 1.14. Let's put it on hold for now and proceed with -modfile only.

There is still disagreement on whether replace directives should be applied.
- @myitcv, @mvdan, @rogpeppe, @pbx0, and others argue this will be important for many binaries long-term, and it's the behavior that most users want and expect.
- @bcmills and @rsc argue that this adds complexity, prevents modules from composing well, and interferes with a lot of other use cases.
- @bcmills and @ianthehat point out that it's difficult for authors to test their modules with -g.
@bcmills is working on a proposal enhancing replace directives for 1.15. If there are additional semantics in the future, it's difficult to decide what to do about replace now.
I'm planning to spend some time analyzing open source go.mod files to understand how often replace directives are needed in repositories with executables and what solutions make the most sense for them.
It's also possible that -modfile may make replace directives less relevant in the future. I hope that developers may put development-only requirements and replace directives in an alternate file (go.dev.mod or go.tools.mod) so those won't need to be checked into the main go.mod file.

Answer 73 · 2019-10-30T05:33:09.000Z

As a user new to modules I'm having trouble intuiting what the use case is for go getting a package outside a module other then to install global binaries. Can sometime explain this more?

If i'm installing global binaries then I feel it should be an error to build if replace directives can't be satisfied. If this behavior is inconsistent with current go get rules then it seems like global installation shouldn't be a flag of go get.

I know go install is off the table, but now that modules is out I'm also having trouble understanding when I would ever use go install anymore. I find the tooling changes the most confusing aspect of modules and I don't find any of the current behavior something I'm tied to since its all fairly different post-gopath.

Answer 74 · 2019-10-30T15:16:25.000Z

@pbx0 go get is also used to upgrade and downgrade dependencies within a module. With the -d flag, go get won't build anything at all.

When go get is used within a module to install an executable, as with other build commands, it respects requirements and replacements of the current module. This is useful if you want to install a tool used within your module, and you want to link against a newer version of some dependency or perhaps a replacement where you've fixed a bug or added a feature. Any changed requirements are written to go.mod.

go install serves a similar use case, but it doesn't have the same dependency management capabilities.

When go get is used to install an executable outside a module, as with other build commands (before #32027 was resolved), it does not respect replacements, since there is no main module. This encourages composability of modules, but it's pretty inflexible. (Some amount of inflexibility can be a good thing; we're happy with static types, after all).

There isn't really a good way to install an executable that won't build without its own module replacements, other than to clone its repository and run go install from there. Technically, it wouldn't be that difficult for the go command to apply these. However, there's a long list of similar things we can't support: file path replacements, checked-in vendor directories (possibly with local modifications), git submodules, generated code that's not checked in, ...

Answer 75 · 2019-10-31T12:41:49.000Z

Adding my 2 cents here in hope it helps clarify the need for a -g-parameter specifically: we have a project where we require goversioninfo, which is a tool we need to invoke before go build (through go generate), and is not a dependency of the actual module we are building. We currently run
GO111MODULE=off go get https://github.com/josephspurrier/goversioninfo
via bash to "just get" the tool and make its binary available under $GOPATH/bin without affecting any go.mod or go.sum files. For modularized projects we would probably use
(cd $HOME && GO111MODULE=on go get https://example.com/x/y)
because it is unlikely for a go.mod to be in $HOME and $HOME is also likely to not be inaccessible.
We believe
go get -g https://github.com/josephspurrier/goversioninfo (or similar)
would be a lot cleaner than having to first find a directory in the file system which is not part of a go module.

Answer 76 · 2019-10-31T14:24:40.000Z

@MMulthaupt For this use case, -modfile as proposed in this issue may be a better fit. You could have a go.tools.mod file which requirements on the minimum versions of tools you need, then run go generate -modfile=go.tools.mod ./...

Answer 77 · 2019-11-01T17:41:24.000Z

@jayconrod Thanks for the info. I guess the question I still have is what are the use cases for running go build and get outside a module in its current form?

I feel strongly that if the go tool ever supports a "install global binary" command that that command will error when it can't respect a replacement rather then build it anyway. It feels like the right action for a command purported to install global binary and anything else would be very surprising IMO. Is this something others feel similarly about given the purpose is to install a global binary at a certain version?

In some sense I'm advocating for more inflexibility, I don't want multiple subtly different actions to install binary commands outside of a module and I don't understand the other use-cases besides install a global binary which seems like it would be a popular one even if limited to projects without replacements, submodules, ect. I feel ambivalent to where the line is drawn in respecting replacements as long as its an error when it can't be done.

Answer 78 · 2019-11-01T19:55:26.000Z

I guess the question I still have is what are the use cases for running go build and get outside a module in its current form?

@pbx0 In 1.14 (after #32027), go build outside of a module will only be useful for small test programs that only depend on the standard library. go get will still be useful for installing binaries, as it is today (though replace directives are not honored).

Not much to add about go get -g and replace right not beyond what's already been said here and in #30515.

Answer 79 · 2019-11-11T20:57:13.000Z

Re-milestoning this for Go1.14 since -modfile is merged and seems likely to ship. I'll close this issue after we've validated -modfile in gopls.

I'd like to direct further discussion on -g or any other "global install" to #30515.

Answer 80 · 2019-11-21T23:56:46.000Z

Change https://golang.org/cl/208236 mentions this issue: cmd/go: add 'go generate' commands to modfile_flag test

Answer 81 · 2019-12-05T21:45:47.000Z

@jayconrod What's the status of this issue? Thanks.

Answer 82 · 2019-12-05T22:09:43.000Z

@ianlancetaylor -modfile is implemented. I'd like some of the gopls folks to evaluate it and post feedback after the beta ships. I expect to either close this issue or re-milestone for 1.15 before the RC.

Answer 83 · 2019-12-16T20:43:46.000Z

Change https://golang.org/cl/211538 mentions this issue: internal/lsp: Use the -modfile flag to update a different go.mod file

Answer 84 · 2019-12-19T18:02:56.000Z

Change https://golang.org/cl/212100 mentions this issue: cmd/go: in 'go list -m', print effective go.mod file

Answer 85 · 2020-01-27T19:05:36.000Z

Closing this issue since 1.14rc1 is imminent. -modfile support was added to gopls in CL 211538, and it seems to be working as intended.

@stamblerre @ridersofrohan @ianthehat Let me know if you have any other feedback on this feature.

Answer 86 · 2020-01-27T22:44:02.000Z

@jayconrod If I have a Makefile which does something like this:
https://github.com/libopenstorage/cloudops/blob/45bab4b444c4fe23c33eaca4146ce3b10dfd1a69/Makefile#L44

How should I change this Makefile when go 1.14 is released to use the -modfile flag?

Can I specify something like -modfile /dev/null?

Answer 87 · 2020-01-28T16:28:48.000Z

@rodrigc It looks like this Makefile is installing then running several tools at the latest versions in GOPATH mode.

I think the module-mode equivalent of this (using -modfile) would be to create a tools.mod file requiring the versions of the tools you want to use. You could build the tools with mkdir -p tools && go build -modfile=tools.mod -o tools/ example.com/tool.... Then run using executables in that tools subdirectory.

That would get you predictable versions of tools without interfering with your main go.mod file. It also wouldn't have side effects on the rest of your system since you aren't installing anything.

Answer 88 · 2020-01-28T17:36:00.000Z

@jayconrod Is this the recommended way to install tools now instead of the tools.go file? Will there be an update to https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module?

Answer 89 · 2020-04-10T00:43:23.000Z

@rodrigc It looks like this Makefile is installing then running several tools at the latest versions in GOPATH mode.

I think the module-mode equivalent of this (using -modfile) would be to create a tools.mod file requiring the versions of the tools you want to use. You could build the tools with mkdir -p tools && go build -modfile=tools.mod -o tools/ example.com/tool.... Then run using executables in that tools subdirectory.

That would get you predictable versions of tools without interfering with your main go.mod file. It also wouldn't have side effects on the rest of your system since you aren't installing anything.

This seems strange to me. I will maintain a file with the names and constraints of executables I need, which makes sense. But then, I want to just point the tool at that file when I use the command for "use this file to give me the things I enumerate". Instead, it seems like the recommendation is to pass each tool enumerated to a different go install invocation. I apologize if I'm misunderstanding what is meant by example.com/..., but I don't see how it can capture the need of downloading tools from several different repositories. So it seems like I only get value out of maintaining this file if I also maintain a script that parses out each dependency and invokes the tool with a different command than I would normally use.

Answer 90 · 2021-02-24T04:10:53.000Z

@jayconrod Could you summarize how to use it pls? Is the following still hold 100%?

Copy go.mod to go.local.mod. Add go.local.mod to .gitignore (or equivalent for your workspace).

Run go env -w GOFLAGS=-modfile=go.local.mod. This tells the go command to use that file by default.

Any any replace and exclude directives or other local edits.

Before submitting and in CI, make sure to test without the local file: go env -u GOFLAGS or just -modfile=. Probably also go mod tidy.

And what is the minimum requirement to use it pls.

Answer 91 · 2021-02-24T13:28:49.000Z

@suntong That's all still correct. -modfile was added in Go 1.14.

Right now, -modfile is mostly used in tools and editors (gopls), not so much directly by users on the command line. We're in the process of putting together a proposal that will make it easier to work on multiple modules at the same time. Stay tuned.