cmd/cue: get go panics
Closed this issue · 10 comments
Originally opened by @nwidger in cuelang/cue#668
What version of CUE are you using (cue version)?
$ cue version cue version 0.3.0-beta.3 darwin/amd64 $ go version go version go1.15.7 darwin/amd64
I can also reproduce the issue building from source using master at 9c5489f.
Does this issue reproduce with the latest release?
Yes
What did you do?
I am a complete CUE newbie. I listened to the recent Go Time podcast about CUE and thought I would try seeing whether cue get could generate stubs from one of my own Go packages. I barely know what I'm doing, I've scanned some of the CUE documentation but otherwise have never used CUE before. It might be that I'm doing something wrong, but I figured having cue panic with a nil pointer dereference wasn't expected.
go mod init blah
cue mod init blah.com
go get github.com/nwidger/lighthouse/@v0.4.2
cue get go github.com/nwidger/lighthouse/...
What did you expect to see?
Successfully generate cue files based on the Go packages in github.com/nwidger/lighthouse/....
What did you see instead?
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x171b900]
goroutine 1 [running]:
cuelang.org/go/cmd/cue/cmd.recoverError(0xc003803ec0)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:221 +0x95
panic(0x17cfba0, 0x1dabe00)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
cuelang.org/go/cmd/cue/cmd.(*extractor).extractPkg(0xc0003a7d40, 0xc000038044, 0x12, 0x0, 0x1e03520, 0x8)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/get_go.go:384 +0x60
cuelang.org/go/cmd/cue/cmd.(*extractor).extractPkg(0xc0003a7d40, 0xc000038044, 0x12, 0xc0032265a0, 0xc00348ee70, 0x1d)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/get_go.go:500 +0x1355
cuelang.org/go/cmd/cue/cmd.(*extractor).extractPkg(0xc0003a7d40, 0xc000038044, 0x12, 0xc00322f320, 0xc0002e0d0a, 0xf)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/get_go.go:500 +0x1355
cuelang.org/go/cmd/cue/cmd.extract(0xc0002cbe00, 0xc0001d1dd0, 0x1, 0x1, 0x11f9780, 0xc0001f1c88)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/get_go.go:363 +0x3d5
cuelang.org/go/cmd/cue/cmd.mkRunE.func1(0xc0002ecdc0, 0xc0001d1dd0, 0x1, 0x1, 0x0, 0x0)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:46 +0x6c
github.com/spf13/cobra.(*Command).execute(0xc0002ecdc0, 0xc0001d1da0, 0x1, 0x1, 0xc0002ecdc0, 0xc0001d1da0)
/Users/mpvl/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000091340, 0x0, 0x0, 0x0)
/Users/mpvl/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
/Users/mpvl/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
cuelang.org/go/cmd/cue/cmd.(*Command).Run(0xc0002cbe00, 0x1997280, 0xc0000a6008, 0x0, 0x0)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:206 +0x65
cuelang.org/go/cmd/cue/cmd.mainErr(0x1997280, 0xc0000a6008, 0xc0000a4050, 0x3, 0x3, 0x18d9dc8, 0xc0001f1f48)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:145 +0x8a
cuelang.org/go/cmd/cue/cmd.Main(0xc00008c058)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:127 +0x9c
main.main()
/Users/mpvl/Documents/dev/release/cue/cmd/cue/main.go:24 +0x25
From what I can tell, extractPkg get passed a nil *packages.Package, perhaps related to the net/http package. Despite cue exploding, I do see that it generates some files:
$ find .
.
./cue.mod
./cue.mod/usr
./cue.mod/gen
./cue.mod/gen/net
./cue.mod/gen/net/http
./cue.mod/gen/net/http/request_go_gen.cue
./cue.mod/gen/net/http/socks_bundle_go_gen.cue
./cue.mod/gen/net/http/http_go_gen.cue
./cue.mod/gen/net/http/header_go_gen.cue
./cue.mod/gen/net/http/client_go_gen.cue
./cue.mod/gen/net/http/sniff_go_gen.cue
./cue.mod/gen/net/http/doc_go_gen.cue
./cue.mod/gen/net/http/jar_go_gen.cue
./cue.mod/gen/net/http/server_go_gen.cue
./cue.mod/gen/net/http/transfer_go_gen.cue
./cue.mod/gen/net/http/transport_go_gen.cue
./cue.mod/gen/net/http/fs_go_gen.cue
./cue.mod/gen/net/http/method_go_gen.cue
./cue.mod/gen/net/http/h2_bundle_go_gen.cue
./cue.mod/gen/net/http/response_go_gen.cue
./cue.mod/gen/net/http/status_go_gen.cue
./cue.mod/gen/net/http/cookie_go_gen.cue
./cue.mod/gen/github.com
./cue.mod/gen/github.com/nwidger
./cue.mod/gen/github.com/nwidger/lighthouse
./cue.mod/gen/github.com/nwidger/lighthouse/lighthouse_go_gen.cue
./cue.mod/module.cue
./cue.mod/pkg
./go.mod
./go.sum
However, github.com/nwidger/lighthouse contains a number of sub-packages, so it looks like cue blows up before it gets to generating .cue files for them.
Please let me know if I can provide any other information!
Original reply by @myitcv in cuelang/cue#668 (comment)
Welcome to the CUE project, @nwidger!
You have, unfortunately, hit a bit of a rough edge. And there is some history associated with cue get go too: if you are interested, please take a look through #621.
The short version however is:
- there are a number of bugs we are in the process of fixing
cue get gowill ultimately be "renamed" tocue import go. This means that you the caller need to ensure the necessary dependencies can be resolved when callcue import go
Which brings us on to the specifics of your case.
As you are the author of this module, I will assume you actually want to provide CUE definitions alongside your Go code, so read on. If you truly intended to work with your module as a dependency, per your example above, see the heading "Using cue get go with ... for dependencies" at the end of this comment.
Providing CUE definitions alongside your Go code
As you are the author of this module, you probably want to have the CUE definitions exist alongside your Go code. In which case you want to do something slightly different to the steps about (we should improve the docs here).
For the record I'm using go version go1.15.7 linux/amd64.
Looking at your project:
cd $(mktemp -d)
git clone https://github.com/nwidger/lighthouse
cd lighthouse
git checkout v0.4.2
The first thing to say is that your module is not tidy:
$ go mod tidy
...
$ git diff --stat
go.mod | 11 ++++++++++-
go.sum | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 60 insertions(+), 1 deletion(-)
I'd suggest that for release versions it should be, otherwise users of your module will find "extra" dependencies added to their go.mod, because those dependencies are not captured by your module.
For what follows, I've run go mod tidy (pushed up the commit I'm using here; the diff vs your v0.4.2)
The next thing to note is that not all of your packages compile:
$ go test ./...
# github.com/nwidger/lighthouse/cmd/lhtogitlab
cmd/lhtogitlab/main.go:466:33: undefined: gitlab.OptionFunc
cmd/lhtogitlab/main.go:475:44: undefined: gitlab.OptionFunc
cmd/lhtogitlab/main.go:484:88: undefined: gitlab.OptionFunc
cmd/lhtogitlab/main.go:504:86: undefined: gitlab.OptionFunc
cmd/lhtogitlab/main.go:522:102: undefined: gitlab.OptionFunc
cmd/lhtogitlab/main.go:594:108: undefined: gitlab.OptionFunc
cmd/lhtogitlab/main.go:607:105: undefined: gitlab.OptionFunc
cmd/lhtogitlab/main.go:628:105: undefined: gitlab.OptionFunc
cmd/lhtogitlab/main.go:645:96: undefined: gitlab.OptionFunc
cmd/lhtogitlab/main.go:699:117: undefined: gitlab.OptionFunc
cmd/lhtogitlab/main.go:699:117: too many errors
ok github.com/nwidger/lighthouse (cached) [no tests to run]
? github.com/nwidger/lighthouse/bins [no test files]
? github.com/nwidger/lighthouse/changesets [no test files]
? github.com/nwidger/lighthouse/cmd/gittolh [no test files]
? github.com/nwidger/lighthouse/cmd/lh [no test files]
? github.com/nwidger/lighthouse/cmd/lh/cmd [no test files]
This might well not be a problem in practice, but I point it out because it's a factor that is contributing to the behaviour you are seeing.
Moving on to cue get go. If you want to make the CUE definitions available in the main module (which in this case is github.com/nwidger/lighthouse), you need to use the --local flag (see cue help get go):
$ cue get go --local ./...
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xb225e0]
...
Notice however we still get a panic. That's because of a bug described in #621 which means that error messages encountered by cue get go are not surfaced; instead the code generator tries to continue to generate CUE definitions from an invalid package and panics because it hits a nil pointer.
The invalid package in this case is the github.com/nwidger/lighthouse/cmd/lhtogitlab package which shows errors in the go test check above.
So what needs to be done here:
- we need to fix the bug that causes these errors to be ignored in
cue get go, which ultimately leads to the panic. That's tracked in #621 and will be fixed via https://cue-review.googlesource.com/c/cue/+/8021 cue get gocould potentially adoptgo testsemantics here of making progress where it can, rather than stopping entirely because of errors in one package. I've captured this in cuelang/cue#646, which is an issue that captures us renamingcue get gotocue import go(rationale in the linked comment if you are interested). However we could make this change prior to the rename
We are addressing the first of these two points for the next release because a number of people are being caught by this bug. We might also include a fix for the second point. In the interim, you can run cue get go with a set of packages that type checks, e.g.:
With the first of those two bugs fixed, there is actually yet another bug that your example trips over that I'm still in the process of debugging. This error will also be fixed as part of the next release.
So unfortunately for now I can't even provide you with a work around.
Using cue get go with ... for dependencies
If you did intend to try this with your module as a dependency, there is a slightly different challenge we will have to solve. The detail of that is captured in cuelang/cue#621 (comment). Namely that if we do move to a model of cue get go (or cue import go as it will be called) requiring that requirements are captured by the main module's go.mod, you will need some way to define dependencies on the packages github.com/nwidger/lighthouse/... within the module that requires github.com/nwidger/lighthouse. There is a proposal on how we could do that in the linked comment, but it might not be the best UX.
Let's generally fix cue get go first to iron out these bugs. Then we can return to this point if it's relevant in your case.
Original reply by @nwidger in cuelang/cue#668 (comment)
@myitcv Thank you so much for the very detailed reply! I feel a little sheepish as while I'm grateful for your very helpful reply, to be honest I never had any real intention to use CUE with my github.com/nwidger/lighthouse package. I simply tried using cue get go with it since it was a package I was familiar with it and thought it would work as well as any other for experimenting around with CUE. When I got the panic, I figured I'd be a good open source citizen and report it back upstream. :)
That being said, thank you for pointing out the issues in my package. That package is basically "done" as it has essentially served it's purpose (getting data out of an old issue tracker and into a new one) and hasn't been worked on in a number of years. It seems the github.com/xanzy/go-gitlab package never got added to the go.mod file for whatever reason and since then there have been breaking API changes. I've added the last working version of that dependency, ran go mod tidy, and published a new version v0.4.3 of my package. Both go test ./... and go build ./... are now happy using that version.
It looks like CUE still panics even with the v0.4.3 version of my package, both with cue get go github.com/nwidger/lighthouse/... and by running cue get go --local ./... within my package as you suggested. However, if I'm understanding your response correctly, this is to be expected.
$ pwd
/Users/niels/delme
$ go mod init blah
$ cue mod init blah.com
$ go get github.com/nwidger/lighthouse/@v0.4.3
$ cue get go github.com/nwidger/lighthouse/...
go: creating new go.mod: module blah
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x171b900]
goroutine 1 [running]:
cuelang.org/go/cmd/cue/cmd.recoverError(0xc001d63ec0)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:221 +0x95
panic(0x17cfba0, 0x1dabe00)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
cuelang.org/go/cmd/cue/cmd.(*extractor).extractPkg(0xc0000d0fc0, 0xc000038044, 0x12, 0x0, 0x1e03520, 0x0)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/get_go.go:384 +0x60
cuelang.org/go/cmd/cue/cmd.(*extractor).extractPkg(0xc0000d0fc0, 0xc000038044, 0x12, 0xc006ca0000, 0xc006ad0640, 0x0)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/get_go.go:500 +0x1355
cuelang.org/go/cmd/cue/cmd.(*extractor).extractPkg(0xc0000d0fc0, 0xc000038044, 0x12, 0xc006ca4d80, 0xc003a0c624, 0xf)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/get_go.go:500 +0x1355
cuelang.org/go/cmd/cue/cmd.extract(0xc0003097e0, 0xc0002c7520, 0x1, 0x1, 0x11f9780, 0xc00029bc88)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/get_go.go:363 +0x3d5
cuelang.org/go/cmd/cue/cmd.mkRunE.func1(0xc000328b00, 0xc0002c7520, 0x1, 0x1, 0x0, 0x0)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:46 +0x6c
github.com/spf13/cobra.(*Command).execute(0xc000328b00, 0xc0002c74f0, 0x1, 0x1, 0xc000328b00, 0xc0002c74f0)
/Users/mpvl/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000245080, 0x0, 0x0, 0x0)
/Users/mpvl/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
/Users/mpvl/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
cuelang.org/go/cmd/cue/cmd.(*Command).Run(0xc0003097e0, 0x1997280, 0xc00003a100, 0x0, 0x0)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:206 +0x65
cuelang.org/go/cmd/cue/cmd.mainErr(0x1997280, 0xc00003a100, 0xc0000300d0, 0x3, 0x3, 0x18d9dc8, 0xc00029bf48)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:145 +0x8a
cuelang.org/go/cmd/cue/cmd.Main(0xc00007e058)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:127 +0x9c
main.main()
/Users/mpvl/Documents/dev/release/cue/cmd/cue/main.go:24 +0x25
$ pwd
/Users/niels/go/src/github.com/nwidger/lighthouse
$ cue get go --local ./...
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x171b900]
goroutine 1 [running]:
cuelang.org/go/cmd/cue/cmd.recoverError(0xc004315ec0)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:221 +0x95
panic(0x17cfba0, 0x1dabe00)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
cuelang.org/go/cmd/cue/cmd.(*extractor).extractPkg(0xc016571cb0, 0xc000034044, 0x1e, 0x0, 0x1e03520, 0x0)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/get_go.go:384 +0x60
cuelang.org/go/cmd/cue/cmd.(*extractor).extractPkg(0xc016571cb0, 0xc000034044, 0x1e, 0xc00572d9e0, 0xc005c9ccd8, 0x0)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/get_go.go:500 +0x1355
cuelang.org/go/cmd/cue/cmd.(*extractor).extractPkg(0xc016571cb0, 0xc000034044, 0x1e, 0xc0057427e0, 0xc01465c4d4, 0xf)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/get_go.go:500 +0x1355
cuelang.org/go/cmd/cue/cmd.extract(0xc000309860, 0xc00033e020, 0x1, 0x2, 0x11f9780, 0xc00029bc88)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/get_go.go:363 +0x3d5
cuelang.org/go/cmd/cue/cmd.mkRunE.func1(0xc000328b00, 0xc00033e020, 0x1, 0x2, 0x0, 0x0)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:46 +0x6c
github.com/spf13/cobra.(*Command).execute(0xc000328b00, 0xc00033e000, 0x2, 0x2, 0xc000328b00, 0xc00033e000)
/Users/mpvl/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000243080, 0x0, 0x0, 0x0)
/Users/mpvl/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
/Users/mpvl/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
cuelang.org/go/cmd/cue/cmd.(*Command).Run(0xc000309860, 0x1997280, 0xc00003a100, 0x0, 0x0)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:206 +0x65
cuelang.org/go/cmd/cue/cmd.mainErr(0x1997280, 0xc00003a100, 0xc0000300b0, 0x4, 0x4, 0x18d9dc8, 0xc00029bf48)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:145 +0x8a
cuelang.org/go/cmd/cue/cmd.Main(0xc00007e058)
/Users/mpvl/Documents/dev/release/cue/cmd/cue/cmd/root.go:127 +0x9c
main.main()
/Users/mpvl/Documents/dev/release/cue/cmd/cue/main.go:24 +0x25
Anyway, I'm happy to try running any other experiments or to try out any fixes that might eventually surface. Just please know that I'm only playing around and not trying to get any real work done with CUE. :)
Original reply by @myitcv in cuelang/cue#668 (comment)
I figured I'd be a good open source citizen and report it back upstream. :)
Very kind of you to raise this issue.
A fix for #621 is going to land shortly, but we are going to leave solving this issue until a later commit. Reason being, we need to slightly rework the way in which Go standard library packages are handled by cue get go, per #648.
The bug you are ultimately running into here is that net/http imports golang.org/x/net/http2/hpack which actually resolves to a load of a special vendored copy, vendor/golang.org/x/net/http2/hpack.
With a fix for #648 we can address this trivially.
Original reply by @vikstrous2 in cuelang/cue#668 (comment)
I think I'm seeing the same issue with cue get go github.com/GoogleCloudPlatform/gke-managed-certs/pkg/apis/networking.gke.io/v1beta1 and cue get go github.com/kubernetes/ingress-gce/pkg/apis/backendconfig/v1beta1. In case you want more test cases :)
Original reply by @myitcv in cuelang/cue#668 (comment)
Note to self: another good test case here is goreleaser/goreleaser#2065
Original reply by @PierreR in cuelang/cue#668 (comment)
I believe I'm seeing the same issue with github.com/openshift/api/route/v1 (using go1.15.5)
Confirmed that this is still an issue as of v0.4.3.
Dropping the now-defunct v0.4.x milestone from this issue, but leaving the zGarden label such that we come round to considering what milestone this should sit in.
Note that I think this may be a duplicate of #2046, which was filed later, but has more recent activity. It's likely that both should be fixed by the same commit.
Confirming as a duplicate of #2046, which got fixed by https://review.gerrithub.io/c/cue-lang/cue/+/1173100. I no longer see the panic on master. Thanks @uhthomas!