x/mobile: build failing when using go modules
mirza-s opened this issue Β· 77 comments
What version of Go are you using (go version
)?
go version go1.11 darwin/amd64
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env
)?
GOARCH="amd64"
GOBIN="/Users/mbp/go/bin"
GOCACHE="/Users/mbp/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/mbp/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/mbp/wrk/src/libfp/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/nt/0hzfppd92jvbk99fhvnhy8ph0000gn/T/go-build526348344=/tmp/go-build -gno-record-gcc-switches -fno-common"
What did you do?
gomobile bind -target=android .
What did you expect to see?
Normal build as when building inside GOPATH
What did you see instead?
gomobile: go build -buildmode=c-shared -o=/var/folders/nt/0hzfppd92jvbk99fhvnhy8ph0000gn/T/gomobile-work-327189385/android/src/main/jniLibs/armeabi-v7a/libgojni.so gobind failed: exit status 1
go: finding github.com/dustin/go-humanize v0.0.0-20180713052910-9f541cc9db5d
go: finding golang.org/x/mobile v0.0.0-20180808221059-bceb7ef27cc6
can't load package: package gobind: unknown import path "gobind": cannot find module providing package gobind
Hi, what's going on this?
I have the same issue but with some different error message:
gomobile: /home/wangyun/dev/gopath/bin/gobind -lang=go,java -outdir=/tmp/gomobile-work-948078978 testaar failed: exit status 1
unable to import bind: cannot find package "golang.org/x/mobile/bind" in any of:
/home/wangyun/dev/software/go/src/golang.org/x/mobile/bind (from $GOROOT)
/home/wangyun/dev/gopath/src/golang.org/x/mobile/bind (from $GOPATH)
unable to import bind: cannot find package "golang.org/x/mobile/bind" in any of:
/home/wangyun/dev/software/go/src/golang.org/x/mobile/bind (from $GOROOT)
/home/wangyun/dev/gopath/src/golang.org/x/mobile/bind (from $GOPATH)
unable to import bind/java: cannot find package "golang.org/x/mobile/bind/java" in any of:
/home/wangyun/dev/software/go/src/golang.org/x/mobile/bind/java (from $GOROOT)
/home/wangyun/dev/gopath/src/golang.org/x/mobile/bind/java (from $GOPATH)
I'm using go 1.11 and updated gomobile, gobind to the latest version.
'''
env go111module=off gomobile bind
'''
it's will temporarily worked fine
cc @eliasnaur - some time has elapsed since this issue and I'm not sufficiently up to speed to know whether mobile is now modules-aware.
is there work currently being done on this front?
Not yet, at least not from my side. Partly because of time available, partly because I prefer to wait for module support to mature (GO111MODULE always on, the go/packages package feature complete and so on), but the main stumbling block is that I can't see the gomobile bind mode work as usual in module mode, at least not in its current form.
The problem is vendoring: gomobile bind generates a main package that in turn imports the packages you want to export to Java or Objective C. That works well in GOPATH mode, but not in module mode. Suppose gomobile bind in module mode generates a module containing a main package and your module as a dependency. Then, because vendored packages are only respected for the main module, any vendored packages from your module will be ignored by the go tool.
Perhaps some form of #26640 will help gomobile. Perhaps gomobile bind should work in a different way. I haven't thought that much about it.
I started a conversation about gomobile and modules here: https://groups.google.com/forum/#!topic/golang-dev/C0ip7pFC2Oc
One question I would like you to ponder on that thread is whether it would be acceptable to require gomobile users to first generate glue code with the gobind tool and then only let the gomobile tool do the final packaging.
Change https://golang.org/cl/167180 mentions this issue: all: add go.mod
Change https://golang.org/cl/167179 mentions this issue: all: set GO111MODULE=off explicitly in bind tests
Change https://golang.org/cl/167659 mentions this issue: cmd/gomobile: explicitly disable modules
Hi, I use modules heavily in my projects. Unfortunately, this makes using gomobile rather difficult. When do you expect gomobile to support modules? Any updates on this?
I was wondering whether packages that use major versions like /v2
are available with go mobile or not.
We also need this feature desperately!
Are there any workarounds for the meantime?
@balupton / @chrisprobst: I use go mod vendor
then switch GO111MODULE=off
and build it as a regular vendored project.
@balupton / @chrisprobst: We basically also use go mod vendor
. However I've created a small tool (GoUp) which automates this process for us.
Change https://golang.org/cl/181158 mentions this issue: cmd/gomobile: don't bind when not in gopath
I have problem, which probably related with it - MoonSHRD/p2chat-android#2. Please, help me, if someone already solved this problem...
CC @hyangah
As we discussed on emails, I decided to tackle this issue :)
My current understanding is that there are three action items roughly:
- Fix gobind to work with Go modules
- Fix gomobile-bind to work with Go modules
- Fix gomobile-build to work with Go modules
The item 1. and 3. seem straight forward. 2. might be more complicated. As Hana said to me, "That conflicts with the assumption of the module-based build system and requires redesign.". Let's discuss about 2. later.
I'm now working on 1., but I found a problem. The reverse binding like "Java/"
or "ObjC/"
doesn't work with GO111MODULE=on
: The below is one of the result. TestGobind/Java-Javapkg-WithModule
is the same test as TestGobind/Java-Javapkg
, but with GO111MODULE
on.
--- FAIL: TestGobind/Java-Javapkg-WithModule (9.37s)
gobind_test.go:91: gobind -lang java golang.org/x/mobile/bind/testdata/testpkg/javapkg failed: exit status 1: type-checking package "golang.org/x/mobile/bind/testdata/testpkg/javapkg" failed (/Users/hajimehoshi/mobile/bind/testdata/testpkg/javapkg/classes.go:8:2: could not import Java/java/beans (go/build: importGo Java/java/beans: exit status 1
can't load package: package Java/java/beans: unknown import path "Java/java/beans": cannot find module providing package Java/java/beans
))
I was wondering if reverse binding is still (actively) supported or not. As far as I remember, the example using reverse binding was deleted due to unsolvable circular dependencies: https://go.googlesource.com/mobile/+/2f2872eacd7fc12c252096592f87e9b23bd5be8b If we need to keep the reverse binding, I'll do so. My concern is that reverse binding does not match with the notion of modules. If we can remove the reverse binding, I'd want to remove the reverse binding first. I tend to want to remove it, but I am not sure it'd be fine.
Any opinions?
As we discussed on emails, I decided to tackle this issue :)
Thank you!
I was wondering if reverse binding is still (actively) supported or not.
I think the level of support is the same for reverse bindings as for the regular bindings. Reverse bindings are less mature however and have probably seen less use.
As far as I remember, the example using reverse binding was deleted due to unsolvable circular dependencies: https://go.googlesource.com/mobile/+/2f2872eacd7fc12c252096592f87e9b23bd5be8b
Note that circular references are a problem for both regular and reverse bindings. All gomobile bind programs must deal with the existence and breaking of reference cycles across the language barrier.
The deleted example you refer to was problematic because it tried to do an entire Android app in pure Go with the reverse bindings. It worked, but the result ended up with too many cross language cycles for that approach to be worth it for real programs.
The deleted example you refer to was problematic because it tried to do an entire Android app in pure Go with the reverse bindings. It worked, but the result ended up with too many cross language cycles for that approach to be worth it for real programs.
Thank you for elaborating! Now I understood that reverse binding still works as same as the regular binding, but it is not feasible to write an Android application only with reverse binding due to many cycles.
Then I think we should keep the reverse binding even with modules. My current idea is that imports starting with "Java/"
or "ObjC/"
are special as "C"
and will ignore the module information.
Change https://golang.org/cl/189597 mentions this issue: internal/importers: replace go/build usages with go/packages
Change https://golang.org/cl/190477 mentions this issue: cmd/gobind: remove the workaround for go/packages
Change https://golang.org/cl/190478 mentions this issue: bind/java: make the try-bots happy
Change https://golang.org/cl/190479 mentions this issue: cmd/gobind: fix issue at build tags
#33822 is possible related.
As #33807 says, Go with modules cannot understand reverse binding imports, starting with Java or ObjC, since they are not valid FQDNs. I think it is not obvious how to solve this. My current ideas are:
- Fix the Go compiler to recognize imports for reverse binding
- Make reverse binding valid only when GO111MODULE=off, and revisit this problem later
- Give up reverse binding :-/
I think 2. is the most feasible. 1. requires specification change and would be very controversial. @hyangah, @eliasnaur, what do you think?
@hajimehoshi let's go with 2. Go with modules can understand non-url like import path but only when they are either standard packages or packages in the main module which defines the workspace.
I will take a look at 1 this week.
Change https://golang.org/cl/192599 mentions this issue: bind/java, bind/objc: do not use module for reverse binding
@hyangah does this mean you're still looking for a solution to allow reverse-binding? Reverse binding is a very important feature for gomobile in my opinion.
Change https://golang.org/cl/203397 mentions this issue: cmd/gobind: replace go/bind with go/packages in gen.go
Change https://golang.org/cl/203399 mentions this issue: cmd/gobind: remove go/build and go/import usages from main.go
Change https://golang.org/cl/195937 mentions this issue: cmd/gobind: add tests enabling Go modules
Change https://golang.org/cl/206737 mentions this issue: cmd/gobind: enable TestDocs tested with Go modules
As gobind
should work with Go modules, now I'm working on gomobile bind
.
gobind
generates files at a temporary directory like [temp-dir]/src/gobind
. The temp-dir
is given by gomobile
. To access this temporary package from Go modules, I think we need to do these items:
- Replace
go/build
withgo/packages
as usual. - Create another temporary directory to emulate a Go proxy. This needs to zip the files and add
go.mod
files. See alsopackagestest
implementation. - Give a temporary and unique module name like
example.com/[random-token]
to the temporary module. - Overwrite
GOPROXY
like$(go env GOPROXY),file:///[new-temporary-directory]
- Use the temporary module name at
gomobile
whengo build
ing.
All are done in gomobile
and gobind
should not be modified from its current behavior.
@hyangah, @eliasnaur Any thoughts?
As
gobind
should work with Go modules, now I'm working ongomobile bind
.
gobind
generates files at a temporary directory like[temp-dir]/src/gobind
. Thetemp-dir
is given bygomobile
. To access this temporary package from Go modules, I think we need to do these items:* Replace `go/build` with `go/packages` as usual. * Create another temporary directory to emulate a Go proxy. This needs to zip the files and add `go.mod` files. See also [`packagestest` implementation](https://github.com/golang/tools/blob/master/go/packages/packagestest/modules.go#L54-L130).
Emulating a Go proxy with zip files, nice.
* Give a temporary and unique module name like `example.com/[random-token]` to the temporary module. * Overwrite `GOPROXY` like `$(go env GOPROXY),file:///[new-temporary-directory]`
It probably doesn't matter given the random token, but perhaps swap the order so the temporary directory is first, to match the intention of gomobile bind
?
* Use the temporary module name at `gomobile` when `go build`ing.
All are done in
gomobile
andgobind
should not be modified from its current behavior.@hyangah, @eliasnaur Any thoughts?
Sounds like a good plan to me.
Thank you!
It probably doesn't matter given the random token, but perhaps swap the order so the temporary directory is first, to match the intention of gomobile bind?
Yes. The second item and the third item will be done at the same time, so we'd need to create a temporary directory first.
Change https://golang.org/cl/206777 mentions this issue: cmd/gomobile: replace go/build with go/packages in bind
It sounds reasonable to me.
Change https://golang.org/cl/208059 mentions this issue: cmd/gomobile: replace go/build with go/packages in gomobile-build
Change https://golang.org/cl/208060 mentions this issue: cmd/gomobile: remove go/build usages from build.go
Change https://golang.org/cl/208077 mentions this issue: cmd/gobind: add -pkgpath option to gobind
- Replace go/build with go/packages as usual.
- Create another temporary directory to emulate a Go proxy. This needs to zip the files and add go.mod files. See also packagestest implementation.
- Give a temporary and unique module name like example.com/[random-token] to the temporary module.
- Overwrite GOPROXY like $(go env GOPROXY),file:///[new-temporary-directory]
- Use the temporary module name at gomobile when go building.
@hyangah and I have discussed the plan and we realized that these points:
- As the package generated by
gobind
is a main package, the module path doesn't have to start with FQDN. gobind
command needs to outputgo.mod
at[tmpdir]/src/gobind
. Thisgo.mod
will include appropriate dependencies along withe the given packages. This depends on the environment wheregobind
is executed. For example, ifgomobile bind example.com/foo
with ago.mod
specifyingexample.com/foo
version 1.2.3 is executed, the generatedgo.mod
would includeexample.com/foo v1.2.3
. Ifgomobile bind ../relative/path/to/localmodule
is executed, thego.mod
would includereplace
directive likereplace localmodule => /foo/bar/baz/relative/path/to/localmodule
.- We don't need to emulate a GOPROXY server. It is because
[tmpdir]/src/gobind
is a main package [1]. When executinggo build
for the generated[tmpdir]/src/gobind
package, just a relative path or changing the working directory should be fine.
Thus, the action items will be:
- Replace go/build with go/packages as usual.
- Fix
gobind
to generate an appropriatego.mod
. - Fix
gomobile
to compile the generated[tmpdir]/src/gobind
package without depending GOPATH.
@hyangah Is my understanding correct?
@eliasnaur @matloob What do you think?
[1] Even if [tmpdir]/src/gobind
is not a main package, a replace
directive would solve the issue.
Thank you for summarizing the discussion. The discussion was inhttps://go-review.googlesource.com/c/mobile/+/208077.
For step 1, I think you are almost done, right? Except the reverse binding. I realized that go/packages
depends on go list
underneath (unless a different driver is supported). Would be nice if gobind
users with different build systems can test and report issues if any.
For step 2, I'd argue that's part of gomobile bind
functionality, not gobind
's because gobind
users may want to have control over their go.mod
file, or may not need modules at all (bazel), or may want to check in the generated code with the existing module (in this case, checking in go.mod
in the generated code directory by mistake can result in a multi-module repo). It may change in the future, but for now how about starting from gomobile bind
?
For step 1, I think you are almost done, right?
Right, done :-)
For step 2, I'd argue that's part of gomobile bind functionality, not gobind's because gobind users may want to have control over their go.mod file, or may not need modules at all (bazel), or may want to check in the generated code with the existing module (in this case, checking in go.mod in the generated code directory by mistake can result in a multi-module repo). It may change in the future, but for now how about starting from gomobile bind?
I think go.mod
would be automatically defined by the arguments of gobind
and its environment, so offering the default go.mod
should not be a bad idea. I'm not sure the last case: what is a multi-module repo? Wouldn't go.mod
be deterministic? We would need to discuss this more anyway.
I agree making gomobile bind
care go.mod
would be fine for now.
I think go.mod would be automatically defined by the arguments of gobind and its environment, so offering the default go.mod should not be a bad idea.
Ok, how about making it an optional flag for gobind? So, gomobile bind passes the flag.
I'm not sure the last case: what is a multi-module repo? Wouldn't go.mod be deterministic? We would need to discuss this more anyway.
Multi-module repo is a repository containing multiple directories with go.mod. Then, the repo will be containing multiple repositories. Sometimes unavoidable, but often creates various problems. By accidentally dropping a go.mod in the gobind output directory, the directory will become a separate module.
Change https://golang.org/cl/209137 mentions this issue: cmd/gomobile: specify the path to $WORK/src/gobind directly instead of using $GOPATH
By accidentally dropping a go.mod in the gobind output directory, the directory will become a separate module.
I thought go.mod
would be dropped at $WORK/src/gobind
then a separate module should not be created.
Ok, how about making it an optional flag for gobind? So, gomobile bind passes the flag.
It's ok, but I'm not sure dropping go.mod
anytime would be really problematic. I'd want to avoid adding a flag if possible.
Change https://golang.org/cl/210477 mentions this issue: cmd/gomobile: make gomobile-init support Go modules.
I am still hitting issues while using modules
Is there a doc that tells clearly how to use gomobile bind with modules?
In my case it says its not able to find the package I am trying to build (module name in my go.mod file) in my GOPATH.
Is there a different way to give the bind command with modules? Also, I am using go1.13.
Is there a doc that tells clearly how to use gomobile bind with modules?
Go modules is not available yet. I'm now working on this :)
Ohh, I thought the above commit adds support for that!
Anyways, eagerly waiting for this! Thanks
Change https://golang.org/cl/210380 mentions this issue: cmd/gomobile: enable Go modules at gomobile-bind
Now gomobile-bind accepts Go modules (if the patch works correctly)! I'll be working on gomobile-build, and we can close this issue after that.
Oops, I found a relative path doesn't work. I'll fix this asap.
Change https://golang.org/cl/214497 mentions this issue: cmd/gomobile: preserve ./ when the given path is relative
Change https://golang.org/cl/214897 mentions this issue: cmd/gomobile: enable Go modules at gomobile-build
Change https://golang.org/cl/214498 mentions this issue: cmd/gomobile: enable Cgo
Change https://golang.org/cl/215419 mentions this issue: cmd/gomobile: output an error message at gomobile-init
Change https://golang.org/cl/215420 mentions this issue: cmd/gomobile: always use abolute paths to replace in go.mod
Change https://golang.org/cl/215421 mentions this issue: cmd/gomobile: always use abolute paths to replace in go.mod
Do replace directives work with the mod file?
I am hitting some issue where its not able to find the relative directories
I think https://golang.org/cl/215421 will fix the issue. With this, I've confirmed replace directives worked.
Now all the problems I found have been fixed. Let's close this issue (I don't have the right to close this issue though), and file other issues when we find other problems.
Great work @hajimehoshi !
Thanks a lot @hajimehoshi !
Forgive my lack of understanding - will this ship with a particular version of Go? I see itβs still part of an βUnreleasedβ milestone.
@themartorana x/mobile
is a repository and a module that you can use today. It's not shipped with Go releases, nor does it follow its release schedule. If a new version of gomobile is out, you can use it with Go 1.13.8, just like you can use it with 1.14rc1.
@hajimehoshi Oh wow, so happy to see this closed. Buuut :D, I think I found a tiny case that could use some love. gomobile
doesn't seem to cache binaries across builds. I have a dependency that takes 5-7 minutes to build for a single platform (https://github.com/ipsn/go-libtor), so for all 4, that's a whopping 30. That's acceptable if it gets reused for subsequent builds, but unfortunately currently it does not. Would it be possible to keep the build cache around?
Thanks.
gomobile
doesn't seem to cache binaries across builds.
That's odd since gomobile just invokes go commands as usual and then module cache and build cache should be used.
I was wondering if my fix was the culprit of the issue you reported. Hasn't there been this issue before my fix?
That's odd since gomobile just invokes go commands as usual and then module cache and build cache should be used.
Isn't gomobile using custom build and cache dirs? The packages are definitely downloaded into the tmp
folder under gomobile-work-xxxxxxxx
, new for every build.
Hasn't there been this issue before my fix?
I can't say, it's been a while since I picked up gomobile. Started doing it today. If you're looking for a painful repro, here's the code I just pushed https://github.com/coronanet/go-coronanet . Though it would possibly make sense to use something smaller to test than this monstrosity.
Isn't gomobile using custom build and cache dirs? The packages are definitely downloaded into the tmp folder under gomobile-work-xxxxxxxx, new for every build.
Aha, gobind, which is invoked by gomobile, generates a source code from Go packages for binding, and they are not cached indeed. Did you mean 'cache' of them?
I think the cache issue is not related to this issue #27234, so I'd be happy if you could file a new issue.
I think the cache issue is not related to this issue #27234, so I'd be happy if you could file a new issue.
@hajimehoshi Done #37902