runtime/cgo: unrecognized relocation with binaries shipped in go1.12.2
jayconrod opened this issue · 27 comments
What version of Go are you using (go version)?
go1.12.2
This specifically affects the binary distribution downloaded from golang.org. Bootstrapping 1.12.2 using 1.12.1 did not have this problem.
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env GOARCH="amd64" GOBIN="" GOCACHE="/home/bazel/.cache/go-build" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOOS="linux" GOPATH="/home/bazel/go" GOPROXY="" GORACE="" GOROOT="/home/bazel/go1.12.2" GOTMPDIR="" GOTOOLDIR="/home/bazel/go1.12.2/pkg/tool/linux_amd64" GCCGO="gccgo" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build845022667=/tmp/go-build -gno-record-gcc-switches"
This error is most easily reproduced with the gcr.io/bazel-untrusted/ubuntu1404:java8 docker image that Bazel uses for testing. Sorry, it's pretty big.
The important part of this is that it's on ubuntu14.04, which has an older linker.
bazel@269f5850547f:~$ /usr/bin/ld -v
GNU ld (GNU Binutils for Ubuntu) 2.24
bazel@269f5850547f:~$ /usr/bin/ld.gold -v
GNU gold (GNU Binutils for Ubuntu 2.24) 1.11
bazel@269f5850547f:~$ /usr/bin/gcc -v
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.8.4-2ubuntu1~14.04.4' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.4)
What did you do?
Download go1.12.2
Try to build the program below:
package main
// const int x = 42;
import "C"
import "fmt"
func main() {
fmt.Println(int(C.x))
}
~/go1.12.2/bin/go build hello.go
What did you expect to see?
Successful build.
What did you see instead?
# command-line-arguments
/home/bazel/go1.12.2/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-648962525/000005.o: unrecognized relocation (0x2a) in section `.text'
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
I'm seeing similar error when Bazel links binaries with cgo code.
external/go_sdk/pkg/tool/linux_amd64/link: running /usr/bin/gcc failed: exit status /usr/bin/ld.gold: error: /tmp/go-link-028957856/000005.o: unsupported reloc 42 against global symbol stderr
/usr/bin/ld.gold: error: /tmp/go-link-028957856/000006.o: unsupported reloc 42 against global symbol stderr
/usr/bin/ld.gold: error: /tmp/go-link-028957856/000007.o: unsupported reloc 42 against global symbol x_cgo_threadentry
/usr/bin/ld.gold: error: /tmp/go-link-028957856/000007.o: unsupported reloc 42 against global symbol x_cgo_inittls
/usr/bin/ld.gold: error: /tmp/go-link-028957856/000012.o: unsupported reloc 42 against global symbol stderr
gcc_fatalf.c:17: error: unsupported reloc 42
gcc_libinit.c:29: error: unsupported reloc 42
gcc_linux_amd64.c:86: error: unsupported reloc 42
gcc_linux_amd64.c:52: error: unsupported reloc 42
gcc_util.c:18: error: unsupported reloc 42
collect2: error: ld returned 1 exit status
Other thoughts
@ianlancetaylor told me that reloc 42 is R_X86_64_REX_GOTPCRELX. Support for this was added to gold in 2015, so it's presumably not supported by ubuntu14.04. Support was added to the Go linker for #13114. The Go toolchain should not be emitting these relocs. We may be incorporating .o files produced by a C compiler that emits these on the system that produces the binary distribution.
I was able to work around this by bootstrapping the Go 1.12.2 toolchain locally. I think the C compiler installed on this image does not emit these relocs.
The relocation appears in runtime/cgo for C code that refers to global variables.
The 1.12.2 binary distribution must have been built with a newer version of GCC than the 1.12.1 binary distribution. The only reasonable fix is to build it with an older version of GCC.
I'm not sure whether we should do anything about this. As a general rule, if you want the binary distribution to run on release N, you need to build it on release N, not a later release. This is one example of that, but there are many others. I don't remember whether we've ever documented anything about what systems the binary distributions run on.
We updated our linux-amd64 builder image from Debian Jessie to Stretch. That's surely the cause. We obviously didn't consider that it'd have any effect on release binaries. We also don't version our builder images and our release automation doesn't use different builders as a function of branch either. Maybe it should. We never thought about it.
Actually we do version our builder images a bit at a broad level. But we do update a number of them "in-place" without changing their names and we definitely don't select release builders as a function of branch.
just to note this is also the case for the go1.11.7 builds:
Whoah, all our Travis builds blew up:
For Travis CI errors, if you're willing to upgrade to Ubuntu 16.04, you can do so by changing dist: trusty to dist: xenial in your .travis.yml file.
This was reported again on #31336. We may need to plan to fix this.
Change https://golang.org/cl/171121 mentions this issue: dashboard, cmd/release: use Jessie for release builds
@gopherbot please backport to 1.11 and 1.12; the fix doesn't need to be cherry-picked, but it will be used to track the issue in the milestone.
Backport issue(s) opened: #31346 (for 1.11), #31347 (for 1.12).
Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.
1.12.3 and 1.11.8 have been built using Jessie and released. This should resolve the issue. Please let us know if there are any other problems and sorry for the trouble.
Thanks.
FWIW, this particular issue can be mitigated by passing "-Wa,-mrelax-relocations=no" when compiling. But if you do that, unless you set CGO_FLAGS to this every time you build, the new (ish) code in the go tool considers the distributed runtime/cgo.a stale and doesn't use it. In fact, given the build cache I'm not sure there is any real need to distribute runtime/cgo.a at all -- I stopped for my snap and as far as I can tell noone has noticed...
With Go 1.12.3 and 1.11.8 I'm still running into this issue with a slightly different error.
/home/travis/.gimme/versions/go1.12.3.linux.amd64/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-252521287/000006.o: unrecognized relocation (0x2a) in section `.text'
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
Yep, this still seems to be happening. Same error with the original repro instructions.
$ ~/go1.12.3/bin/go build hello.go
# command-line-arguments
/home/bazel/go1.12.3/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-328573434/000005.o: unrecognized relocation (0x2a) in section `.text'
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
Sorry to be the bearer of bad news, but I think we need to reopen this.
Change https://golang.org/cl/171124 mentions this issue: cmd/release: use Jessie for release builds, take 2
@bradfitz Are the 1.12.3 binaries being rebuilt after your "take 2" patch or is 1.12.4 coming shortly?
@aronatkins, new binaries are forthcoming. @andybons is on it. We're working on a test first, though.
I get the same error when compiling k8s's kubelet module, details here kubernetes/kubernetes#76259
Change https://golang.org/cl/171317 mentions this issue: cmd/release: sanity check relocations before release
this is happening on Centos6 with Go 1.12.3, when running: "go install":
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-606096497/000008.o(.text+0x74): unresolvable H??@?>H?FH??H??H??@?~?F?H??@?~H??8?H??H??0?FH??H??(?FH??H?? ?FH??H???FH??H???FH??H??F?fD relocation against symbol `stderr@@GLIBC_2.2.5'
/usr/bin/ld: BFD version 2.20.51.0.2-5.48.el6_10.1 20100205 internal error, aborting at reloc.c line 443 in bfd_get_reloc_size
Change https://golang.org/cl/171578 mentions this issue: doc/devel: add go1.12.4 and go1.11.9
@dbaroncelli, we didn't get a new release out today, but we should tomorrow.
Change https://golang.org/cl/171766 mentions this issue: [release-branch.go1.12] doc: document Go 1.12.4 and Go 1.11.9
Change https://golang.org/cl/171767 mentions this issue: [release-branch.go1.11] doc: document Go 1.11.9
I've confirmed with 1.12.4 and 1.11.9 that the relocations are back to their intended state:
$ readelf -r --wide go1.11.9.linux-amd64/go/pkg/linux_amd64/runtime/cgo.a 2>/dev/null | grep REL
0000000000000074 0000001300000009 R_X86_64_GOTPCREL 0000000000000000 stderr - 4
00000000000001dc 0000002400000009 R_X86_64_GOTPCREL 0000000000000000 stderr - 4
0000000000000003 0000001500000009 R_X86_64_GOTPCREL 0000000000000008 x_cgo_threadentry - 4
00000000000000ab 0000001d00000009 R_X86_64_GOTPCREL 0000000000000008 x_cgo_inittls - 4
0000000000000035 0000001600000009 R_X86_64_GOTPCREL 0000000000000000 stderr - 4
$ readelf -r --wide go1.12.4.linux-amd64/go/pkg/linux_amd64/runtime/cgo.a 2>/dev/null | grep REL
0000000000000074 0000001300000009 R_X86_64_GOTPCREL 0000000000000000 stderr - 4
00000000000001dc 0000002400000009 R_X86_64_GOTPCREL 0000000000000000 stderr - 4
0000000000000003 0000001500000009 R_X86_64_GOTPCREL 0000000000000008 x_cgo_threadentry - 4
00000000000000ab 0000001d00000009 R_X86_64_GOTPCREL 0000000000000008 x_cgo_inittls - 4
0000000000000035 0000001600000009 R_X86_64_GOTPCREL 0000000000000000 stderr - 4
$ readelf -r go1.11.9.linux-386/go/pkg/linux_386/runtime/cgo.a 2>/dev/null | grep GOT32
00000012 00001603 R_386_GOT32 00000000 stderr
00000225 00002703 R_386_GOT32 00000000 stderr
00000014 00001703 R_386_GOT32 00000004 x_cgo_threadentry
000000ac 00001e03 R_386_GOT32 00000004 x_cgo_inittls
00000043 00001903 R_386_GOT32 00000000 stderr
$ readelf -r go1.12.4.linux-386/go/pkg/linux_386/runtime/cgo.a 2>/dev/null | grep GOT32
00000012 00001603 R_386_GOT32 00000000 stderr
00000225 00002703 R_386_GOT32 00000000 stderr
00000014 00001703 R_386_GOT32 00000004 x_cgo_threadentry
000000ac 00001e03 R_386_GOT32 00000004 x_cgo_inittls
00000043 00001903 R_386_GOT32 00000000 stderr
These have been released on golang.org/dl. Apologies for the churn and thanks for everyone's patience. Hopefully we can close this for good now.