cmd/go: go install cmd@version errors out when module with main package has replace directive
james-lawrence opened this issue Β· 45 comments
What version of Go are you using (go version)?
go version go version go1.16 linux/amd64
Does this issue reproduce with the latest release?
yes
What did you do?
cd /tmp; go install bitbucket.org/jatone/genieql@latest
What did you expect to see?
go install finishing successfully
What did you see instead?
go install bitbucket.org/jatone/genieql@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
given that genieql is the main module this error appears to be erroneous.
other attempts I made all of which i would have expected to work properly (and use to):
cd /tmp; go install bitbucket.org/jatone/genieql/...@latest
go install bitbucket.org/jatone/genieql/...@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
cd /tmp; go install bitbucket.org/jatone/genieql/cmd/...@latest
go install bitbucket.org/jatone/genieql/cmd/...@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
cd /tmp; go install bitbucket.org/jatone/genieql/cmd/genieql@latest
go install bitbucket.org/jatone/genieql/cmd/genieql@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
L28 contains a replace directive
replace github.com/containous/yaegi => github.com/james-lawrence/yaegi v0.8.8-modules-enh
Which is not valid for go install
No module is considered the "main" module. If the module containing packages named on the command line has a go.mod file, it must not contain directives (replace and exclude) that would cause it to be interpreted
differently than if it were the main module
Closing as working as intended
yes obviously line 28 contains the directive. see the title. and then see this thread where I was told to open this issue.
that would cause it to be interpreted differently than if it were the main module
is the relevant line of that statement here. the fact is this directive wouldn't cause it to be interpreted differently than if it was the main module. and to any outside viewer, there is a main module, its the one being installed.
reopening, I would recommend rephrasing the issue in terms of a proposal to relax the behaviour rather than as a bug report
@seankhliao I'm just not sure if it is a bug or a proposal, the discussion around go installs (new) behavior was long, varied, and confused. =) this particular problem was pointed out during that discussion and seemingly ignored/disregarded/left open to be revisited. so here we are.
this particular problem was pointed out during that discussion and seemingly ignored/disregarded/left open to be revisited.
#40276 was indeed discussed at length. Indeed, over the years even prior to that specific proposal, this issue has been discussed in various forms, many times. To say that the question of replace directives was ignored/disregarded is incorrect. Indeed I specifically linked to a quote that pointed out the option for that particular point to be revisited. It suffices therefore to open an issue, as suggested, to start that discussion.
Deferring to @jayconrod on how best to take the discussion forward.
I meant it was at one of those 3 at various points in that (and the related) thread. I wasn't calling out any particular stage, sorry for that confusion.
I think there may be confusion around the concepts of a main module and the package main.
The main module is normally the one in which the go command was invoked (no matter what packages are being built or installed). Modules cannot be installed, only packages. When building in package@version mode there is no main module.
In the case in this issue genieql is the package main, and the way in which it would be built changes if the module of which it is a part is the main module because of the replace statement.
For example, if you did go install bitbucket.org/jatone/genieql in another module that had bitbucket.org/jatone as a dependency it would not obey the replace directive. If you checkout bitbucket.org/jatone and do go install ./genieql then it will obey the replace directives and produce a different result. It would also obey the versions specified in the main module, which would change the results even further.
Also note that it is not something that used to work, it is a brand new feature that limits the number of cases in which it is enabled.
Maybe we should change the terminology for main module to a different word?
Also note that it is not something that used to work
while technically true go get -u did work because it respected the vendor directory: go114 get -u bitbucket.org/jatone/genieql-go114/cmd/... see related issue #44841 and #40053 + the google groups thread in that issue.
cat ~/go/src/bitbucket.org/jatone/genieql-go114/vendor/github.com/containous/yaegi/interp/src.go | grep -i 'func pkgDir(ctx \*build.Context'
returns the patched function from the replace + vendor workflow, and if you started with a clean environment there would be no other sources of the library in question.
now did it work if trying to use a package as a dependency? no, and that was fine because it wasn't about using it as a library. many go programs are never intended to be used that way.
now the rest of your argument is attempting to explain away for current limitations after the fact that go get / go install commands were fundamentally changed to support go modules while simultaneously ignoring vendor workflows of the tools and making that argument with a distinction that never existed until the tools were changed.
now all that being said: installing a binary is different from managing dependencies. I believe this is an accepted fact and a reason go get was changed to be about managing dependencies and go install is being shift towards building and installing binaries. this fundamentally is a good change and I'm wholeheartedly behind it.
I think the question we need to be asking is if the distinction between main package of the main module during installation matter? I'm going to assert it does not and that this error is erroneous, but during dependency management where someone tried to add a dependency that had a replace directive this error would be 100% correct though I'd start out with a warning in that case vs a hard error.
Ah I believe I have a good explanation about why this should be supported.
the replace directive is fundamentally about compilation, its meant to inform the build system how to patch dependencies. this is true both during development where you may need to patch them while debugging a problem which is when you compile your program. and during installation of a main module (previously the main package) onto a system.
so during compilation there is no distinction between the main package and the main module, however, during dependency management replace directives have always been ignored and any software that was relying on a dependency where there was a replace directive was broken (conceptually if not in practice). and this is where we should be erroring out (i.e. in go get in this new world) when we see a replace directive. not during compilation (go install).
Closing this issue since it's working as described in #40276.
That being said, at some point in the future when we have more information, it will be reasonable to open a proposal to either ignore or apply replace and exclude directives. That proposal should be based on the discussion in #40276 (and previously #30515), it must include data backing up whichever position is supported, and it must consider tradeoffs for the other side.
I know this is a lot to ask (those issues are very long), but the main point of disagreement was what to do with replace directives, and we debated that for a year and a half. The outcome was a compromise: replace directives are not allowed for now, and we have the option of changing that in the future without making an incompatible change.
To restart that discussion, we need new information: how is the new go install cmd@version form being used, and would replace directives be helpful or harmful?
@jayconrod wrote:
To restart that discussion, we need new information: how is the new
go install cmd@versionform being used, and wouldreplacedirectives be helpful or harmful?
As a go user, I reasonably expect go install cmd@version to just work and install the binary cmd using the go.mod the authors of that package spent time getting exactly right.
Instead I get an extremely cryptic message like this
go install: github.com/rclone/rclone@latest (in github.com/rclone/rclone@v1.57.0):
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
As a user of said cmd, I have no idea how to fix that problem.
From the instructions for go install cmd@version: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies need amending
No module is considered the "main" module. If the module containing packages named on the command line has a go.mod file, it must not contain directives (replace and exclude) that would cause it to be interpreted differently than if it were the main module. The module must not require a higher version of itself.
Which says the same thing as the error message, more or less.
So it is consistent, but not helpful in my opinion!
It is very hard having a large open source project without a few soft forks while you are waiting for upstreams to take your patches.
I first noticed this problem with github's gh tool, and I think you'll find that there are lot more high profile Go binaries out there with this problem.
$ go install github.com/cli/cli/cmd/gh@latest
go install: github.com/cli/cli/cmd/gh@latest (in github.com/cli/cli@v1.14.0):
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
I read the proposal in #40276 and it concludes with:
The modules I'm most concerned about are those that use replace as a soft fork while submitting a bug fix to an upstream module; other problems have other solutions that I don't think we need to design for here. Modules using soft fork replacements are about 4% of the the modules with go.mod files I sampled (165 / 4519). This is a small enough set that I think we should move forward with the proposal above.
I am sad to learn that 4% is small for some, but it is still a big one when you are one of those inside 4%.
I think those soft forks are really important use case to support. We have this situation just now, we have a tool which depends on 3rd party library. That library have a data-loss bug users of our tool might encounter so we made a PR upstream with a fix and use replace to fix our tool. But now we cannot really release our tool but have to wait for upstream to merge it. In meantime, all our users might be loosing data. So now we have to wait for upstream to merge it.
You can imagine similar scenario for a security issue, too. So sometimes a soft fork is useful to get an important fix out soon. We can build a binary and release it so I am not sure why users cannot do go install ... @version to get exactly the same binary, just compiled for them.
@mitar Similarly I have a lot of soft-forks which I need replace directives for. The users can't install with "go install" because of the limitation on replace directives, but it would make way more sense if Go install just behaved as if I had run "go install" with cwd set to the root of the module containing the referenced package. Issue: go install -v github.com/aperturerobotics/bifrost/cmd/bifrost@master - fails w/ this error.
I hit this too.
$ go get github.com/estesp/manifest-tool/v2/cmd/manifest-tool@v2.0.0
go get: installing executables with 'go get' in module mode is deprecated.
Use 'go install pkg@version' instead.
For more information, see https://golang.org/doc/go-get-install-deprecation
or run 'go help get' or 'go help install'.
$ go install github.com/estesp/manifest-tool/v2/cmd/manifest-tool@v2.0.0
go install: github.com/estesp/manifest-tool/v2/cmd/manifest-tool@v2.0.0 (in github.com/estesp/manifest-tool/v2@v2.0.0):
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
Do I misinterpret that there's just no recourse here?
@jayconrod or other maintainers: given that this issue has a lot of traffic & there will be a lot of projects with replace directives:
Clearly people would like to be able to "go install" things with replace directives.
Ideally it would just respect the directives of the module containing the desired build target.
Given that both this issue & the ones linked to in the closing message are all closed, there's no path forward to this right now (or is there?) - could we re-open this at least?
Pinging this again, in case there's a canonical answer and we're all just missing it?
We have soft forks of tools due to various reasons and now we are blocked from upgrading from go 1.16 b/c of this issue. Given that Go 1.16 isn't being supported forever we'll soon be stuck with outdated software or rewrite a lot of module imports to make hard forks...
The same thing. We can't install 3rd party tools with go 18 any more.
I wish I could install it too.
I would like to be able to install it with go install as I would like many people to test it before I soft fork it and submit a pull request to the developer.
I wish I could install it too.
I would like to be able to install it with
go installas I would like many people to test it before I soft fork it and submit a pull request to the developer.
oh yes, this looks like a mandarory use-case
@jayconrod this seems to cause some significant inconveniences and if I understand the following passage from #40227 correctly, it should be possible to ease the restrictions and restore a smoother user experience:
Parts of this proposal are more strict than is technically necessary (for
example, requiring one module, forbidding replace directives). We could relax
these restrictions without breaking compatibility in the future if it seems
expedient. It would be much harder to add restrictions later.
This whole thing is especially tricky since everything needs to be fixed at the root of the issue and even then it's problematic.
If we look at openshift/hypershift#803 for example and its replace directives:
https://github.com/openshift/hypershift/blob/b5e8c84eefb207aaff9be8b9d4f2c606d9019f80/go.mod#L129
it's pointing to https://github.com/kubevirt/containerized-data-importer-api/blob/main/go.mod which has its module name different from its repository. However, a contributor cannot easily fix this because, the change with a following go mod tidy results in something like:
$ git diff
diff --git a/go.mod b/go.mod
index bd68c03..2b66eac 100644
--- a/go.mod
+++ b/go.mod
@@ -1,10 +1,11 @@
-module kubevirt.io/containerized-data-importer-api
+module github.com/kubevirt/containerized-data-importer-api
go 1.17
require (
k8s.io/api v0.23.5
k8s.io/apimachinery v0.23.5
+ kubevirt.io/containerized-data-importer-api v1.47.0
kubevirt.io/controller-lifecycle-operator-sdk/api v0.0.0-20220329064328-f3cc58c6ed90
)
diff --git a/go.sum b/go.sum
index 2c6f7f5..709ec7b 100644
--- a/go.sum
+++ b/go.sum
@@ -285,6 +285,8 @@ k8s.io/kube-openapi v0.0.0-20220124234850-424119656bbf/go.mod h1:sX9MT8g7NVZM5lV
k8s.io/utils v0.0.0-20210802155522-efc7438f0176/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA=
k8s.io/utils v0.0.0-20211116205334-6203023598ed h1:ck1fRPWPJWsMd8ZRFsWc6mh/zHp5fZ/shhbrgPUxDAE=
k8s.io/utils v0.0.0-20211116205334-6203023598ed/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA=
+kubevirt.io/containerized-data-importer-api v1.47.0 h1:fncmQ/2J8MVb3c2snNkUXlBQX472nqCjJJ2AduuMrDc=
+kubevirt.io/containerized-data-importer-api v1.47.0/go.mod h1:yjD8pGZVMCeqcN46JPUQdZ2JwRVoRCOXrTVyNuFvrLo=
kubevirt.io/controller-lifecycle-operator-sdk/api v0.0.0-20220329064328-f3cc58c6ed90 h1:QMrd0nKP0BGbnxTqakhDZAUhGKxPiPiN5gSDqKUmGGc=
kubevirt.io/controller-lifecycle-operator-sdk/api v0.0.0-20220329064328-f3cc58c6ed90/go.mod h1:018lASpFYBsYN6XwmA2TIrPCx6e0gviTd/ZNtSitKgc=
sigs.k8s.io/json v0.0.0-20211020170558-c049b76a60c6 h1:fD1pz4yfdADVNfFmcP2aBEtudwUQ1AlLnRBALr33v3s=
where kubevirt.io/containerized-data-importer-api v1.47.0 would either need to be changed by hand which might fail in CI or a follow up PR that means extra effort.
tl;dr: if I submitted a proposal to relax the behavior would that be something the team would generally consider?
I would like to add another use case for why applying replace is required:
Summary:
It is required to be able to use BuildInfo (debug.ReadBuildInfo()) as version embedding mechanism which is imo the go1.18 way to go (and will be even cleaner when go build package@version -o somePath will work #52898)
Long version:
I've recently switch version embedding from the previous pattern of ldflags -X foo.version=1.2.3 to using the wonderful new 1.18 BuildInfo and use go install ...@tag to create even the binary distributions of my projects
So for my projects go install project@version is now the only way to create the right binaries
I found a bug in golang.org/x/net that is fixed in a cl but not yet available so I patched it into a fork
https://github.com/fortio/proxy/blob/main/go.mod
replace golang.org/x/net => github.com/fortio/net v0.0.0-20220521233046-9a5c58aa1acd // Has https://go.dev/cl/407454
but I can't install it! so I can't have it show the right version
$ go install fortio.org/proxy@v0.8.3 # works
$ proxy -version # works
14:52:17 I proxy_main.go:67> Fortio Proxy 0.8.3 h1:KdayZCE9EV7NQwOENjnoGtP7gdtSfyIhJ/6hK73hv34= go1.18.2 arm64 darwin starting
0.8.3 h1:KdayZCE9EV7NQwOENjnoGtP7gdtSfyIhJ/6hK73hv34= go1.18.2 arm64 darwin
go go1.18.2
path fortio.org/proxy
mod fortio.org/proxy v0.8.3 h1:KdayZCE9EV7NQwOENjnoGtP7gdtSfyIhJ/6hK73hv34=
dep fortio.org/fortio v1.31.0 h1:5pwFhai6GNFTKiacKlWF3oRjoveFdMCCoqGoSDyFmIA=
dep github.com/fsnotify/fsnotify v1.5.4 h1:jRbGcIw6P2Meqdwuo0H1p6JVLbL5DHKAKlYndzMwVZI=
dep github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
dep golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI=
dep golang.org/x/net v0.0.0-20220425223048-2871e0cb64e4 h1:HVyaeDAYux4pnY+D/SiwmLOR36ewZ4iGQIIrtnuCjFA=
dep golang.org/x/sys v0.0.0-20220412211240-33da011f77ad h1:ntjMns5wyP/fN65tdBD4g8J5w8n015+iIIs9rtjXkY0=
dep golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
build -compiler=gc
build CGO_ENABLED=1
build CGO_CFLAGS=
build CGO_CPPFLAGS=
build CGO_CXXFLAGS=
build CGO_LDFLAGS=
build GOARCH=arm64
build GOOS=darwin
$ go install fortio.org/proxy@v0.9.0
go: fortio.org/proxy@v0.9.0 (in fortio.org/proxy@v0.9.0):
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
is there at least a --please-do-it-anyway flag or way to build the binary with the BuildInfo and with a go.mod replacement that I am missing?
in case this helps someone else; it turns out for things like x/net it's easy to use a fork and get go install to work:
change the go.mod in the fork (https://github.com/fortio/net/commit/cd8eba16ed62c07c4269b81d3715b30f18b28a03), eg:
- module golang.org/x/net
+ module github.com/fortio/netand use that fortio/proxy@0aa96c2
+ "github.com/fortio/net/http2"
- "golang.org/x/net/http2"it is a bit painful though because you have to change all the imports, and then revert that back when the upstream change shows up
Seems nobody has pointed this before, but if you follow the multi module repository documentation: https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories , you won't be able to install your application with go install. It would be good to either document this in the faqs or relax the rules for this case to be handle as described in the faqs.
Is there any proper workaround around this? Or now the installation of projects with a replace inside mod file is not feasible at all?
Unfortunately, I don't believe that there is. go install just simply doesn't work for some projects anymore. It's incredibly frustrating.
Is there any proper workaround around this? Or now the installation of projects with a replace inside mod file is not feasible at all?
The only work around I have figured out is to not use go install git... and manually git clone followed by manual go build/run/install. You could even write a replacement command that would work with go run, setup git clone in a temporary directory, run go build/install, remove the code. Basically hiding this problem from your users. It is lame, but should work and be close to what go actually does.
Why is this issue closed if there is no good solution?
In case there is a good solution, let me know. (I need it.)
It seems it was closed in March with the idea to gather more information. Considering the amount of references to this bug in the mean time, it might be a good idea to reopen the discussion.
For anyone here from a Google search for this error message, try running this command to install a package from a repo ("like npm install -g"):
GO111MODULE=off go get github.com/apenwarr/git-subtrac(replace github.com/apenwarr/git-subtac with your desired package; that's just an example of the one that led me here today)
It worked as desired:
β― ls $GOPATH/bin | grep git-subtrac
git-subtracNote my go version is go1.18.3 darwin/amd64 which is fairly outdated (latest is 1.19.2), so YMMV.
thanks to Alec over at StackOverflow. more explanation here: https://stackoverflow.com/a/52764550/3793499
For the Go developers, I thought I'd document how I found this error message:
I don't write Go, but I do occasionally run programs in it, and I have a reasonably up-to-date version on my system (go version go1.18.3 darwin/amd64). But the ecosystem is mostly unknown to me. So every time I need to install a package it's a bit of a chore to remember the right $GOPATH settings, check it's actually installed, remember it's go version not go --version, etc. And when something else goes wrong, I have to second guess all that configuration before I know if the error is my fault or not.
Today I found this thread because I wanted to install a global binary (https://github.com/apenwarr/git-subtrac) that I could use on my system. That readme quite reasonably suggests:
git-subtrac is a git extension written in the Go language using the lovely go-git library. If you have Go 1.12 or higher and a project that already uses go modules (contains a go.mod file), you can install the tool like this:
go install github.com/apenwarr/git-subtracIf you have an older version of go or don't care about go modules, you can install it like this instead:
go get -v github.com/apenwarr/git-subtrac
in reality, this is what my session looked like (starting from go version, which took a few attempts to get right π):
β― go version
go version go1.18.3 darwin/amd64ok, seems a bit old but not too bad, and this repo is 2 years old, so let's try the instructions from the readme:
β― go install github.com/apenwarr/git-subtrac
go: 'go install' requires a version when current directory is not in a module
Try 'go install github.com/apenwarr/git-subtrac@latest' to install the latest version add @latest? ok, I can try that... and I do like when an error message gives a suggestion :)
β― go install github.com/apenwarr/git-subtrac@latest
go: github.com/apenwarr/git-subtrac@latest (in github.com/apenwarr/git-subtrac@v0.0.0-20200907023842-6f55d3a89654):
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module. oh no :( I guess that's why it was only a suggestion
*various go help mashing trying to find the answer to how to install a global binary into $GOPATH/bin...*
β― go get --help
usage: go get [-t] [-u] [-v] [build flags] [packages]
Run 'go help get' for details.
β― go install --help
usage: go install [build flags] [packages]
Run 'go help install' for details.
β― go help get | grep -i global
β― go help install | grep -i global
it's not very helpful :(
*google some words*
Finally find https://stackoverflow.com/a/52764550/3793499 with the answer
To restart that discussion, we need new information: how is the new
go install cmd@versionform being used, and would replace directives be helpful or harmful?
Here's a case study: I have a project that relies on some Gtk4 bindings. Those bindings in turn rely on go4.org/unsafe/assume-no-moving-gc which uses build constraints that are manually added as each new Go version is released as a form of an assertion that Go never adds a moving GC. This causes problems, as it means that my app will not build with new versions of Go unless I wait for that module to be updated and then I also update my dependencies to use that new version. It also means that old versions of the app will never build with newer versions of Go.
I got tired of this and added a replace go4.org/unsafe/assume-no-moving-gc => ./internal/empty where internal/empty was a module with nothing in it as a way of getting rid of the assertion. This worked great, until someone tried to install my app using go install at which point it broke due to this issue. Due to the utter lack of viable workarounds, I was forced to remove the replace directive and go back to the manual update process every six months or so.
A more helpful error message I think:
The go.mod file for the module contains replace directives to potentially external code. Try cloning from source and using
go build.
I think this message is better because judging the comments in this thread:
- Its not clear to most why replace directives affect this command.
- For non go developers, its not obvious that cloning and trying
go buildis usually the next step when one sees this error.
FWIW I'm pro go install@version respecting in-repo replace directives whether its by just making that change directly or part of some rethink of supporting soft forks.
As of now, I am using the bingo tool to get around this problem. The design of this tool is very good, its a thin wrapper on top of go modules and you can use it to "go install" any binary dependency without getting into the replace directive problem.
I really hope to see a proposal or two here which integrates bingo's concepts directly into golang's tool chain
As of now i saw the below proposal from this issue in bingo but not sure about its implementation status
Someone flagged recent discussion on this closed issue to me privately. As a general note, we don't watch closed issues very carefully - they're closed.
Replace directives remain for local development. If you want to post software that others can go install, it needs to build without the replace directives. If we respected the replace directives during go install foo@latest, then that would create commands that can be installed that way but not using 'go get foo; go install foo' in a module where you are trying to track the version of foo you are using. We are also discussing integrating tools more explicitly with a module in #48429, and again there it would be a serious problem if tools existed that worked with 'go install foo@latest' but not with a regular build graph from another module. (And it is a non-starter for one tool in your module to force the use of its own replace directives. What if you don't want those replace directives? Or what if different tools have conflicting replace directives?)
If you are using dependencies that don't function without replace directives, that may be a sign to rethink the use of those dependencies, or to upstream fixes that make the replace directives unnecessary.
For the specific case of go4.org/unsafe/assume-no-moving-gc, there are two better answers than replace:
- (Short-term) Run 'go get go4.org/unsafe/assume-no-moving-gc@latest' in your own module, which will update beyond the version in your dependency, to get a copy that works with the latest version of Go.
- (Long-term) Upstream fixes to what is using that package to not use, or upstream fixes to that package to not break at each new Go release.
These work within the ecosystem and keep it healthy. Replace directives fragment the ecosystem and make it brittle. My post The Principles of Versioning in Go has a worked example. It looks at forced package downgrades, but replaces are analogous andhave all the same problems.
We are aware of the problem of go4.org/unsafe/assume-no-moving-gc specifically. In May we worked with the upstream author on a change that makes it no longer break at each new Go release. So if you run 'go get go4.org/unsafe/assume-no-moving-gc@latest' in your own module one last time, you should stop seeing breakages from that package, and you can remove the replace.
Hi @rsc,
Thankyou for the above, but it seems to avoid addressing the key use case I and others in this issue seem to want:
The particular problem is with the statement
Replace directives remain for local development. If you want to post software that others can go install, it needs to build without the replace directives.
which ignores a somewhat common practice:
- Some project is shipping a CLI executable written in Go.
- They discover some bug in a third party dependency. They fork it, fix the bug, and use a replace directive to use the forked version until a formally released fix is available.
- This works perfectly fine in most ways, as being a CLI executable, never depended upon as a library, their module is always the main module, so can use replace directives...
- ...except when users try to use
go install foo/bar/baz@version
It seems like you would class this as a misuse of replace directives based on what you said above?
The problem with that, is that the Go project has given developers an almost perfect tool - replace directives - for dealing with this, so it is unrealistic to think people would not use replace directives in this scenario. AFAIK there are no other options that don't involve needing to change import paths of the forked software in every source file that consumes it, which is inelegant in the extreme!
It's not like the users trying to use go install can go to the developers using the replace directives and ask them not to use them - they'll rightly point out that extreme inelegancy in forking without using replace directives, and decline to do that.
So you end up with unhappy users who have had the lovely convenience of go install dangled in front of them, and then snatched away.
If we respected the replace directives during go install foo@latest, then that would create commands that can be installed that way but not using 'go get foo; go install foo' in a module where you are trying to track the version of foo you are using.
@rsc: My understanding of this issue is that we care about the use case where you use go install outside of a go module. I think what we really are asking for is that go install should just be the same as doing git clone + go install inside the cloned repository. I think this is a well defined behavior. No surprises there. It would make the use case where you fix the program in your fork and wait for upstream to merge it possible.
Installing inside a go module is tricky as you describe. And there is no need to support it there because inside a go module you can just modify go.mod yourself and add the replace directive there.
So, could we get go install to work outside of go modules with repositories which have replace directives?
Some more anecdotal evidence as to why this is an issue:
We are using the k8s client libs, but recently this change was introduced in the openAPI Spec models:
kubernetes/kube-openapi#402
kubernetes/kube-openapi@38767cd
Now we can't upgrade k8s.io/api or k8s.io/client-go because of the model mismatches:
# k8s.io/client-go/applyconfigurations/meta/v1
vendor/k8s.io/client-go/applyconfigurations/meta/v1/unstructured.go:64:38: cannot use doc (variable of type *"github.com/google/gnostic/openapiv2".Document) as *"github.com/google/gnostic-models/openapiv2".Document value in argument to proto.NewOpenAPIData
So we are left with:
1.) Revert (and wait for transient patches)
- kubernetes/kubernetes#118340
- kubernetes-sigs/cli-utils#625
2.) Run a fork (and make the patch ourselves) π±
3.) Use a replace directive
replace k8s.io/kube-openapi => k8s.io/kube-openapi v0.0.0-20230501164219-8b0f38b5fd1f
Being a Lazy (Pragmatic...Β―_(γ)_/Β―) Programmer, I chose option 3, as I think most Go Devs would. Now I can upgrade the module I want, without any of the side effects. It just works.
...But now we can't go install the code because of the replace directive. Which is kind of annoying. I would expect go install to honor the replace directive, no different than if I clone the entire repo down and run go install from there.
I would also argue that although the replace directive may have been intended for local development purposes, it is widely used across go projects to circumvent issues exactly like the one I posted here. Dependencies can get messy, and being able to "strategically" use a replace directive when necessary is a nice way to work around (hack) those issues.
Regarding replace directives being "only for local development" reading back to the original vgo and modules proposals I find this about replace.
From vgo tour
If you do identify a problem in a dependency, you need a way to replace it with a fixed copy temporarily. Suppose we want to change something about the behavior of rsc.io/quote. Perhaps we want to work around the problem in rsc.io/sampler, or perhaps we want to do something else. (...)
From Minimal Version Selection
Minimal version selection is very simple. It achieves simplicity by eliminating all flexibility about what the answer must be: the build list is exactly the versions specified in the requirements. A real system needs more flexibility, for example the ability to exclude certain module versions or replace others.
(emphasis mine)
also from Minimal Version Selection, section Who controls your build?
The dependencies of a top-level module must be given some control over the top-level build. B 1.2 needs to be able to make sure it is built with D 1.3 or later, not with D 1.2. Otherwise we end up with the current go get's stale dependency failure mode.
(...)
In the design of module requirements, exclusions, and replacements, I've tried to balance the competing concerns of allowing dependencies enough control to ensure a succesful build without allowing them so much control that they harm the build. Minimum requirements combine without conflict, so it is feasible (even easy) to gather them from all dependencies. But exclusions and replacements can and do conflict, so we allow them to be specified only by the top-level module.
From the modules proposal:
This proposal aims to balance allowing dependencies enough control to ensure a successful build with not allowing them so much control that they break the build. Minimum requirements combine without conflict, so it is feasible (even easy) to gather them from all dependencies, and they make it impossible to pin older versions, as Kubernetes does. Minimal version selection gives the top-level module in the build additional control, allowing it to exclude specific module versions or replace others with different code, but those exclusions and replacements only apply when found in the top-level module, not when the module is a dependency in a larger build.
A module author is therefore in complete control of that moduleβs build when it is the main program being built, but not in complete control of other usersβ builds that depend on the module. I believe this distinction will make this proposal scale to much larger, more distributed code bases than the Bundler/Cargo/Dep approach.
(emphasis mine)
From the Q&A section of the modules proposal:
How does one patch a deep dependency without vendoring?
Response [#24301 (comment) by @kardianos.] By using a replace directive.
PS. I couldn't find any mention in the proposal or in the vgo posts about replace being only appropriate for local builds but AFAICT the proposal did not introduce the distinction between local vs non-local builds, AFAICT that was initially introduced as a consequence of implementation details of go get.
So what do you do if you encounter a replace? lol
I just installed go for the first time to use godo.
go get github.com/digitalocean/godo@v1.106.0
go: go.mod file not found in current directory or any parent directory.
'go get' is no longer supported outside a module.
To build and install a command, use 'go install' with a version,
like 'go install example.com/cmd@latest'
For more information, see https://golang.org/doc/go-get-install-deprecation
or run 'go help get' or 'go help install'.
go install github.com/digitalocean/godo@v1.106.0
go: downloading github.com/digitalocean/godo v1.106.0
go: github.com/digitalocean/godo@v1.106.0 (in github.com/digitalocean/godo@v1.106.0):
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
I don't get it. (get it?) What's go.mod's deal anyway?
@seanbethard you must git clone that project and run "go install ." within the directory.
@seanbethard you must git clone that project and run "go install ." within the directory.
This would be a lot less of a problem if the go command
- Told you to do this when you ran into the error.
- Had a command to clone the repo at a given version into a subdirectory of the current directory.
It would still be a problem, but it would be significantly less of one.
I think this bug intentionally exists to prevent people from leaning on replace directives instead of upstreaming their fixes.
As someone who is using a lot of replace directives, this is a quite annoying limitation for go install.
Example: https://github.com/aperturerobotics/bifrost#examples
Here I have to say - well, you could use go install, but we have replace directives, so instead clone it and build. It defeats the purpose of go install.