plugin: panic: plugin was built with a different version of package

Question

plugin: panic: plugin was built with a different version of package

juhwany opened this issue 6 years ago · 36 comments

juhwany commented 6 years ago

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (`go version`)?

go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN="/Users/juhwany/go_workspace/bin"
GOCACHE="/Users/juhwany/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/juhwany/go_workspace"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/juhwany/gos/go_server/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/mk/2tyywdsn0jq6l0vl8094tss80000gp/T/go-build380269681=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Clone the blow repo in each directory. (not under GOPATH)
server (main program) : https://github.com/juhwany/go_server
plugin : https://github.com/juhwany/go_plugin
On same machine, build server & plugin in each directory
Build server : go build -o main
Build plugin : go build -buildmode=plugin -o plugin.so
Copy plugin.so to server directory and run
./main
See error message when main program try to open plugin.

panic: plugin.Open("plugin"): plugin was built with a different version of package github.com/juhwany/go_server/v2/apis

What did you expect to see?

Main program should load plugin.so successfully.
I could not understand why it fails even though the referenced package version is same between main program and plugin.
Am I using plugin in wrong way?

What did you see instead?

plugin was built with a different version of package error message

Answer 1 · 2018-09-19T16:07:07.000Z

Related (dup?) #26759 .

Answer 2 · 2018-09-20T06:28:07.000Z

@randall77

No. I think building environment is slightly different from #26759.
I didn't build main program and plugin under GOPATH and the above sample project uses Go module for specifying dependency package. Go version is also different.

Answer 3 · 2018-09-21T19:51:01.000Z

I do think they are all related though, including #27062. I think it all boils down to changing (absolute) paths at build time between the plugin build and the app build.

Answer 4 · 2018-09-22T03:41:35.000Z

CC @ianlancetaylor @cherrymui

Answer 5 · 2018-10-08T22:02:37.000Z

I'm not sure if anyone's already mentioned it, but putting the common(apis) package to a separate module outside the server one will work. It's definitely cumbersome, but can be used as a workaround until this issue is fixed.

Answer 6 · 2018-11-13T21:51:41.000Z

The command "go list -m -json all" shows version of go_server on go_plugin, but it's not defined on go_server itself. I have a similar problem using multiples modules on the same repository using "replace" directive. I don't know how to have the same version of the module inside and outside the repository.

Answer 7 · 2018-12-01T19:40:42.000Z

In #28983, @bcmills suggests to build plugins from within the main module.

// server/go.mod
module github.com/juhwany/go_server/v2
require github.com/juhwany/plugin v0.0.0
replace github.com/juhwany/plugin => ../go_plugin

$ go build -buildmode=plugin github.com/juhwany/plugin
$ go run main.go 
DoSomething called!

Answer 8 · 2019-07-10T00:39:24.000Z

Just throwing in my two cents here. Both the @tamasf97 and @oszika are workarounds.

However, @tamasf97's workaround only works if you can factor the shared code out without creating a circular dependency (not always possible).

@oszika's workaround using the "replace" directive is pretty inconvenient for clients of the shared code, as you need that local repo.

Answer 9 · 2020-02-28T20:30:08.000Z

We are unable to run our plugin-based service with gomod due to this issue.

Answer 10 · 2020-02-28T20:30:28.000Z

We are experiencing this issue after implementing go.mod in both the server and plugin

Answer 11 · 2020-02-28T20:34:39.000Z

@itsmontoya, @dhalman: what specific commands did you run to build the plugin?

(Did you follow the approach in #27751 (comment)? Did you build with -trimpath? Both of those may be necessary.)

Answer 12 · 2020-02-28T20:35:35.000Z

@bcmills we will try -trimpath and the other approach and get back to you ASAP! 👍

Answer 13 · 2020-03-01T06:43:30.000Z

added -trimpath and the error moved from our microservice dependency lib to now internal/cpu:

plugin.Open("plugins/thePlugin"): plugin was built with a different version of package internal/cpu

Answer 14 · 2020-03-01T10:43:36.000Z

@dhaimeen We've used the Go plugin API for quite a while now (over a year) and the error "different version of package internal/cpu" occurs when you've not used the -trimpath flag on both the shared object you build (plugin) and the main binary. Both bundle a copy of the Go runtime (although there's an open issue on the tracker about not packaging a copy of the Go runtime with plugin code). Hope this helps.

Answer 15 · 2020-03-02T23:31:23.000Z

The main bin solved it for us in a manual build process. Thank you!!

We are now running into an issue where it appears -trimpath isnt working as expected when running from a command in the .Exec(...) env

The .so files produced by the exec command differ from the files produced by go build in the cli

Answer 16 · 2020-03-03T00:00:11.000Z

Ok, I think we have determined the difference:

Successful:

cd <path-to-plugin>
go build -trimpath -buildmode=plugin -o <filename>

Unsuccessful:

go build -trimpath -buildmode=plugin -o <filename> <path-to-plugin>

Are there any ways to determine why the latter fails to produce the same output as the former?

Thanks!

Answer 17 · 2020-03-03T01:13:30.000Z

@dhalman I think you can try using go build's -x flag, e.g.

go build -x -trimpath -buildmode=plugin -o <filename>

to print out how the go command invokes the compiler and the linker, and how it differs.
Maybe you want to do a go clean -cache first.

Answer 18 · 2020-03-04T01:11:28.000Z

@dhalman, is the plugin in the same module as the main binary? If not, the module metadata could be what's throwing it off.

Do you have any cgo dependencies? That could also do it (#36072).

Answer 19 · 2020-05-26T10:10:56.000Z

I have a collection of Go plugins I need to package and distribute. When building and running on my machine everything works as expected. However plugins don't work on other machines.

Here is the command I execute:

$ GO111MODULE=on go build -trimpath -buildmode=plugin -o ./my-plugin.so ./my-plugin-directory

I tried running the parent application calling the plugins both with and without -trimpath. Nothing works.

How can I fix this behavior?

Answer 20 · 2020-07-04T23:43:34.000Z

@bcmills I got caught up on this one with modules - the problem can occur when you import a package from the same module in the "loader," and you import the same package but from a different module in the "plugin."

Answer 21 · 2020-07-05T20:55:05.000Z

We have developed an open source solution for syncing go.mod repositories across large projects with internal dependencies. We find it especially useful for managing large collections of plugins along with their private imports.

Happy to share it if requested, but would not want to advertise in this forum

Answer 22 · 2020-07-06T08:22:27.000Z

@paralin Do you have a working and a non-working example so I can try to reproduce your solution? I am not sure to fully understand.

Answer 23 · 2021-03-12T09:50:19.000Z

Any process?

Answer 24 · 2021-03-13T14:04:23.000Z

I encountered the same issue while trying out this repo: https://github.com/vladimirvivien/gosh

To reproduce it, this is the process:

go version      # go version go1.16 darwin/amd64
dlv version     # Version: 1.6.0

git clone https://github.com/vladimirvivien/gosh
cd gosh

go mod init github.com/vladimirvivien/gosh  # to fix the running issue

This is all good:

go build -buildmode=plugin -o plugins/splash_command.so plugins/00_splash.go
go build -buildmode=plugin -o plugins/sys_command.so plugins/syscmd.go
go run gosh.go  # all good

But when try debugging by setting the breakpoint at /gosh.go, line 59, I always get an error:

"plugin.Open("plugins/splash_command"): plugin was built with a different version of package runtime/internal/sys"

If I try this instead by adding the -trimpath:

go build -trimpath -buildmode=plugin -o plugins/splash_command.so plugins/00_splash.go
go build -trimpath -buildmode=plugin -o plugins/sys_command.so plugins/syscmd.go
go run gosh.go

I would get this error:

failed to open plugin splash_command.so: plugin.Open("plugins/splash_command"): plugin was built with a different version of package internal/unsafeheader
failed to open plugin sys_command.so: plugin.Open("plugins/sys_command"): plugin was built with a different version of package internal/unsafeheader

Why?

Answer 25 · 2021-04-08T12:35:16.000Z

We have developed an open source solution for syncing go.mod repositories across large projects with internal dependencies. We find it especially useful for managing large collections of plugins along with their private imports.

Happy to share it if requested, but would not want to advertise in this forum

Hey @dhalman,

could you post the link to the open source syncing solution of yours. I find developing any a bit non-trivial plugin for any a bit non-trivial app a total nightmare. Maybe your solution might help me out. :)

Cheers

Answer 26 · 2021-04-14T12:45:31.000Z

@kriptor i havent tried it in a while, but im here to help support if you do end up using it

https://github.com/gomuserver/gomu

Answer 27 · 2021-05-27T12:51:46.000Z

Hello,

UP on this subjecT?

Answer 28 · 2021-05-30T06:27:17.000Z

I posted some comments here about this issue previously, right now you can view the workarounds I've built as well as a work in progress hot reloading system (which is hampered by this bug) here: https://github.com/aperturerobotics/controllerbus/tree/master/example/plugin-demo

Answer 29 · 2021-05-31T16:16:53.000Z

Thank you @paralin , but my problem is ... very ennoyinh.

Package A include C version 1.1 ( go mod used ).

Package B ( the plugin ) include C version 1.1 go mod used.

Result, Plugin B was buillt with a different version of C.

My "workaround" was to remove the go.mod used in the B package to make it works. Very dirty solution in my opinion.

Answer 30 · 2022-12-22T07:01:43.000Z

In #28983, @bcmills suggests to build plugins from within the main module.

// server/go.mod
module github.com/juhwany/go_server/v2
require github.com/juhwany/plugin v0.0.0
replace github.com/juhwany/plugin => ../go_plugin

$ go build -buildmode=plugin github.com/juhwany/plugin
$ go run main.go 
DoSomething called!

I had initially built the plugin with the same go.mod file as the main module. I encountered the issue when I moved the plugin into its own module. I tried --trimpath to build both the plugin and main module but did not succeed. The reference is a place to look.

@juhwany Does this issue still exist for you?

Answer 31 · 2023-01-02T19:45:05.000Z

Apparently disabling optimizations and inlining -gcflags="all=-N -l" "resolves" the issue for me, but the output files are huge :(

go build -buildmode=plugin -gcflags="all=-N -l" -o plugins/test.so plugins/test.go

Answer 32 · 2023-04-06T07:27:22.000Z

I have basically tried everything and in the end I have this error: plugin was built with a different version of package internal/goarch". I am using exact same dependencies, go version, built environments, flags, etc...

Answer 33 · 2023-10-25T20:28:35.000Z

Well, I am still stuck with the same problem. Not sure if there will be any update on this issue. Exactly for this issue we had to choose another language for the modular system we were working on.

Answer 34 · 2024-02-12T17:45:04.000Z

I had to move Vroomy away from Plugins which was unfortunate.

Answer 35 · 2024-04-03T21:43:33.000Z

Any update on this issue

Answer 36 · 2024-04-03T22:02:30.000Z

Closing as I don't think there is any specific issue with the plugin package itself, it all appears to be application build environment differences.

What version of Go are you using (go version)?