runtime: panic 'exitsyscall: syscall frame is no longer valid' in go tool

Question

runtime: panic 'exitsyscall: syscall frame is no longer valid' in go tool

mappu opened this issue 7 years ago · 16 comments

mappu commented 7 years ago

What version of Go are you using (`go version`)?

go version go1.9.2 linux/arm

Does this issue reproduce with the latest release?

Unsure.
It barely reproduces under the in-use version, either. This is mostly an FYI.

What operating system and processor architecture are you using (`go env`)?

A little bit strange one: Docker container 'arm32v7/golang:1.9.2-stretch on x86_64 host. That's an armhf rootfs where every binary executes under qemu-user.

GOARCH="arm"
GOBIN=""
GOEXE=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_arm"
GCCGO="gccgo"
GOARM="6"
CC="gcc"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build911040395=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

Compile a Go program.
It may be notable that the target source code was hosted on a Docker mapped volume,

What did you expect to see?

No error

What did you see instead?

The Go compiler crashed with the following message:

fatal error: exitsyscall: syscall frame is no longer valid

runtime stack:
runtime.throw(0x3d4579, 0x2d)
        /usr/local/go/src/runtime/panic.go:605 +0x70
runtime.exitsyscall.func1()
        /usr/local/go/src/runtime/proc.go:2634 +0x24
runtime.systemstack(0x1091ea80)
        /usr/local/go/src/runtime/asm_arm.s:264 +0x80
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1135

goroutine 5 [syscall]:
syscall.Syscall(0xc3, 0x10c0cb40, 0x10d26268, 0x0, 0x0, 0x3, 0x57ca9fb0)
        /usr/local/go/src/syscall/asm_linux_arm.s:17 +0x8 fp=0x10e052f8 sp=0x10e052f4 pc=0x85d14
syscall.Stat(0x3ecc94, 0x36, 0x10d26268, 0x10972c40, 0x0)
        /usr/local/go/src/syscall/zsyscall_linux_arm.go:1576 +0x70 fp=0x10e0532c sp=0x10e052f8 pc=0x85664
os.Stat(0x10b38230, 0x36, 0xf3cf8, 0x3720b8, 0x0, 0x623)
        /usr/local/go/src/os/stat_unix.go:32 +0x40 fp=0x10e05354 sp=0x10e0532c pc=0xa4ff0
syscall.Syscall6(0x36, 0xf3cf8, 0x3720b8, 0x0, 0x623, 0x0, 0x10d20760, 0x10b85900, 0x0, 0x1091e000)
        /usr/local/go/src/syscall/asm_linux_arm.s:73 +0x68 fp=0x10e05358 sp=0x10e05354 pc=0x85de8
os.StartProcess(0x10, 0x20, 0x21, 0x7, 0x3, 0xf0a74, 0x10b92fc0, 0xf667d000, 0x0)
        /usr/local/go/src/os/exec.go:94 +0x44 fp=0x10e05380 sp=0x10e05358 pc=0x9f1e8
created by cmd/go/internal/work.(*Builder).Do
        /usr/local/go/src/cmd/go/internal/work/build.go:1192 +0x44c

goroutine 1 [semacquire, 1 minutes]:
sync.runtime_Semacquire(0x10d670bc)
        /usr/local/go/src/runtime/sema.go:56 +0x2c
sync.(*WaitGroup).Wait(0x10d670b0)
        /usr/local/go/src/sync/waitgroup.go:131 +0x88
cmd/go/internal/work.(*Builder).Do(0x10954240, 0x10a76310)
        /usr/local/go/src/cmd/go/internal/work/build.go:1214 +0x46c
cmd/go/internal/work.runBuild(0x5b2da0, 0x10974068, 0x0, 0x1)
        /usr/local/go/src/cmd/go/internal/work/build.go:455 +0x4ec
main.main()
        /usr/local/go/src/cmd/go/main.go:133 +0x670

goroutine 19 [syscall, 1 minutes]:
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:131 +0x134
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x14
created by os/signal.init.0
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x30

goroutine 6 [syscall]:
syscall.Syscall6(0x118, 0x1, 0x629, 0x10e07674, 0x1000004, 0x0, 0x0, 0x379230, 0x99d04, 0x0)
        /usr/local/go/src/syscall/asm_linux_arm.s:48 +0x8
os.(*Process).blockUntilWaitable(0x10c83920, 0x0, 0x17, 0x18)
        /usr/local/go/src/os/wait_waitid.go:31 +0x64
os.(*Process).wait(0x10c83920, 0x10bd8edc, 0x1cab84, 0x10bd8edc)
        /usr/local/go/src/os/exec_unix.go:22 +0x2c
os.(*Process).Wait(0x10c83920, 0x0, 0x0, 0x3ecbb0)
        /usr/local/go/src/os/exec.go:115 +0x1c
os/exec.(*Cmd).Wait(0x10bd8e70, 0x0, 0x0)
        /usr/local/go/src/os/exec/exec.go:446 +0x40
os/exec.(*Cmd).Run(0x10bd8e70, 0x0, 0x0)
        /usr/local/go/src/os/exec/exec.go:289 +0x44
cmd/go/internal/work.(*Builder).runOut(0x10954240, 0x10bbef90, 0x2f, 0x10b5b8c0, 0x1d, 0x0, 0x0, 0x0, 0x10bd8dc0, 0x10, ...)
        /usr/local/go/src/cmd/go/internal/work/build.go:1983 +0x378
cmd/go/internal/work.gcToolchain.gc(0x10954240, 0x10c466c0, 0x10b90100, 0x36, 0x10b900c0, 0x3a, 0x10914b00, 0x10c55340, 0x2, 0x2, ...)
        /usr/local/go/src/cmd/go/internal/work/build.go:2272 +0xc14
cmd/go/internal/work.(*gcToolchain).gc(0x5c7fa8, 0x10954240, 0x10c466c0, 0x10b90100, 0x36, 0x10b900c0, 0x3a, 0x0, 0x10c55340, 0x2, ...)
        <autogenerated>:1 +0x94
cmd/go/internal/work.(*Builder).build(0x10954240, 0x10910850, 0x0, 0x0)
        /usr/local/go/src/cmd/go/internal/work/build.go:1393 +0x1048
cmd/go/internal/work.(*Builder).Do.func1(0x10910850)
        /usr/local/go/src/cmd/go/internal/work/build.go:1148 +0x58
cmd/go/internal/work.(*Builder).Do.func2(0x10d670b0, 0x10954240, 0x10ce2270)
        /usr/local/go/src/cmd/go/internal/work/build.go:1205 +0x84
created by cmd/go/internal/work.(*Builder).Do
        /usr/local/go/src/cmd/go/internal/work/build.go:1192 +0x44c

Answer 1 · 2018-04-03T01:28:09.000Z

Another crash in the same environment:

fatal error: exitsyscall: syscall frame is no longer valid
fatal error: stopm holding locks
panic during panic
stack trace unavailable

runtime stack:

Answer 2 · 2018-04-03T07:04:49.000Z

Hello there @mappu, thank you for filing this issue!

Could you please provide a minimal reproduction that's runnable for anyone? That helps a lot in prognosis/diagnosis of the issue affecting you, and makes bug fixing faster. Thank you!

Answer 3 · 2018-04-18T00:08:02.000Z

Is the stopm holding locks case also from running the go tool?

Answer 4 · 2018-04-18T07:52:33.000Z

@ianlancetaylor yes, it was.

I tried to reproduce this today on a different machine. I got some different failures instead.

The basic steps to reproduce:

Set up qemu-user-static and binfmt-support to allow running armhf binaries from x86_64 hardware (my qemu is 2.8+dfsg-6+deb9u3)
docker run --rm -it arm32v7/golang:1.9.2-stretch /bin/bash
(inside container) Clone go.git, then run make.bash

First attempt:

root@container:~/go/src# ./make.bash
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
qemu-arm-static: /build/qemu-DqynNa/qemu-2.8+dfsg/translate-all.c:175: tb_lock: Assertion `!have_tb_lock' failed.
qemu-arm-static: /build/qemu-DqynNa/qemu-2.8+dfsg/translate-all.c:175: tb_lock: Assertion `!have_tb_lock' failed.
./make.bash: line 196:   237 Segmentation fault      (core dumped) ./cmd/dist/dist bootstrap $buildall $vflag $GO_DISTFLAGS "$@"

Go binary segfaulted, but with qemu displaying some warnings.

Second attempt:

root@container:~/go/src# ./make.bash
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.

^C

the process hung and must be stopped.

Third attempt:

root@container:~/go/src# ./make.bash
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
fatal error: sigaction failed

runtime stack:
runtime.throw(0x295c16, 0x10)
        /root/go/src/runtime/panic.go:589 +0x54
runtime.sysSigaction.func1()
        /root/go/src/runtime/os_linux.go:419 +0x24
runtime.sysSigaction(0x40, 0xf6fffba4, 0x0)
        /root/go/src/runtime/os_linux.go:418 +0x44
runtime.sigaction(0x40, 0xf6fffba4, 0x0)
        /root/go/src/runtime/sigaction.go:15 +0x20
runtime.setsig(0x40, 0x4dbac)
        /root/go/src/runtime/os_linux.go:381 +0xbc
runtime.initsig(0x1200)
        /root/go/src/runtime/signal_unix.go:111 +0x10c
runtime.mstartm0()
        /root/go/src/runtime/proc.go:1254 +0x3c
runtime.mstart1(0x0)
        /root/go/src/runtime/proc.go:1225 +0xc8
runtime.mstart()
        /root/go/src/runtime/proc.go:1195 +0x60

goroutine 1 [runnable]:
runtime.main()
        /root/go/src/runtime/proc.go:109
runtime.goexit()
        /root/go/src/runtime/asm_arm.s:840 +0x4
go tool dist: FAILED: /root/go/pkg/tool/linux_arm/go_bootstrap install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 2
root@34061c589a23:~/go/src#

This time a crash in Go runtime package. But it's a different panic compared to the earlier ones.

Fourth attempt:

root@container:~/go/src# ./make.bash
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
qemu-arm-static: /build/qemu-DqynNa/qemu-2.8+dfsg/translate-all.c:175: tb_lock: Assertion `!have_tb_lock' failed.
qemu-arm-static: /build/qemu-DqynNa/qemu-2.8+dfsg/translate-all.c:175: tb_lock: Assertion `!have_tb_lock' failed.
./make.bash: line 196:  5352 Segmentation fault      (core dumped) ./cmd/dist/dist bootstrap $buildall $vflag $GO_DISTFLAGS "$@"

Go binary segfaulted, again with the qemu failures.

Fives attempt:

root@container:~/go/src# ./make.bash
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
./make.bash: line 196:  6624 Segmentation fault      (core dumped) ./cmd/dist/dist bootstrap $buildall $vflag $GO_DISTFLAGS "$@"

Go binary segfaulted, but no qemu failure messages.

Go works reliably when running on a real machine. I think these issues in runtime are probably something about this weird qemu&docker environment, not really Go's fault. So if that's out-of-scope, then please close this ticket.

Answer 5 · 2018-04-18T12:51:42.000Z

Thanks for the info. It's not out of scope but I don't know who is going to be able to fix it.

Answer 6 · 2018-05-02T09:34:26.000Z

FWIW I can reproduce this without docker. It appears that go1.11 arm and arm64 binaries cannot be run under qemu user emulation (this is a "regression", since 1.10 binaries work).

On a linux/amd64 system with qemu-user installed:

$ go version
go version go1.10 linux/amd64
$ GOARCH=arm64 go build test.go; ./test
Hello, world!

$ gotip version
go version devel +500d79c410 Wed May 2 04:20:25 2018 +0000 linux/amd64
$ GOARCH=arm64 gotip build test.go; ./test
fatal error: sigaction failed

runtime stack:
runtime.throw(0xc0e2f, 0x10)
	/home/adonizetti/go/src/runtime/panic.go:589 +0x48
runtime.sysSigaction.func1()
	/home/adonizetti/go/src/runtime/os_linux.go:420 +0x2c
runtime.sysSigaction(0x40, 0x4000800250, 0x0)
	/home/adonizetti/go/src/runtime/os_linux.go:419 +0x48
runtime.sigaction(0x40, 0x4000800250, 0x0)
	/home/adonizetti/go/src/runtime/sigaction.go:15 +0x20
runtime.setsig(0x40, 0x446b0)
	/home/adonizetti/go/src/runtime/os_linux.go:382 +0xb8
runtime.initsig(0x4000003800)
	/home/adonizetti/go/src/runtime/signal_unix.go:113 +0x178
runtime.mstartm0()
	/home/adonizetti/go/src/runtime/proc.go:1254 +0x3c
runtime.mstart1()
	/home/adonizetti/go/src/runtime/proc.go:1225 +0xcc
runtime.mstart()
	/home/adonizetti/go/src/runtime/proc.go:1195 +0x50

goroutine 1 [runnable]:
runtime.main()
	/home/adonizetti/go/src/runtime/proc.go:109
runtime.goexit()
	/home/adonizetti/go/src/runtime/asm_arm64.s:1084 +0x4

I know we don't guarantee that Go binaries work under qemu user emulation mode, but it was nice to be able to use it...

Answer 7 · 2018-05-02T17:35:13.000Z

Can you find out why it is failing? The stack trace indicates that the sigaction system call failed, but why?

Answer 8 · 2018-05-02T18:00:33.000Z

Tracing says 29923 rt_sigaction(64,0xfffef304,NULL) = -1 errno=22 (Invalid argument).

Command and full trace:

$ GOARCH=arm gotip build test.go; qemu-arm -strace ./test
29923 getpid() = 29923
29923 sched_getaffinity(0,8192,-76952,8964,0,31) = 64
29923 mmap2(NULL,262144,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xff7af000
29923 mmap2(NULL,270532608,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xef5af000
29923 brk(NULL) = 0x00132000
29923 mmap2(0x00400000,541065216,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00400000
29923 clock_gettime(1,-68856,1597334677,0,-68840,1177936) = 0
29923 mmap2(0x00400000,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0x00400000
29923 mmap2(0xef5af000,266240,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0xef5af000
29923 mmap2(NULL,65536,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xef59f000
29923 clock_gettime(1,-69384,1,1194560,-69364,1177936) = 0
29923 mmap2(NULL,65536,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xef58f000
29923 rt_sigprocmask(SIG_SETMASK,NULL,0x0011f994) = 0
29923 clock_gettime(1,-68824,1194560,0,-68804,1177936) = 0
29923 clock_gettime(1,-68920,0,44,-68900,1177936) = 0
29923 sigaltstack((nil),0xfffef338) = 0
29923 sigaltstack(0xfffef31c,(nil)) = 0
29923 rt_sigprocmask(SIG_SETMASK,0xfffef33c,NULL) = 0
29923 gettid() = 29923
29923 rt_sigaction(SIGHUP,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGHUP,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGINT,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGINT,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGQUIT,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGQUIT,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGILL,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGILL,0xfffef304,NULL) = 0
29923 rt_sigaction(5,NULL,0xfffef31c) = 0
29923 rt_sigaction(5,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGABRT,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGABRT,0xfffef304,NULL) = 0
29923 rt_sigaction(7,NULL,0xfffef31c) = 0
29923 rt_sigaction(7,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGFPE,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGFPE,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGUSR1,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGUSR1,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGSEGV,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGSEGV,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGUSR2,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGUSR2,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGPIPE,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGPIPE,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGALRM,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGALRM,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGTERM,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGTERM,0xfffef304,NULL) = 0
29923 rt_sigaction(16,NULL,0xfffef31c) = 0
29923 rt_sigaction(16,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGCHLD,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGCHLD,0xfffef304,NULL) = 0
29923 rt_sigaction(23,NULL,0xfffef31c) = 0
29923 rt_sigaction(23,0xfffef304,NULL) = 0
29923 rt_sigaction(24,NULL,0xfffef31c) = 0
29923 rt_sigaction(24,0xfffef304,NULL) = 0
29923 rt_sigaction(25,NULL,0xfffef31c) = 0
29923 rt_sigaction(25,0xfffef304,NULL) = 0
29923 rt_sigaction(26,NULL,0xfffef31c) = 0
29923 rt_sigaction(26,0xfffef304,NULL) = 0
29923 rt_sigaction(27,NULL,0xfffef31c) = 0
29923 rt_sigaction(27,0xfffef304,NULL) = 0
29923 rt_sigaction(28,NULL,0xfffef31c) = 0
29923 rt_sigaction(28,0xfffef304,NULL) = 0
29923 rt_sigaction(29,NULL,0xfffef31c) = 0
29923 rt_sigaction(29,0xfffef304,NULL) = 0
29923 rt_sigaction(30,NULL,0xfffef31c) = 0
29923 rt_sigaction(30,0xfffef304,NULL) = 0
29923 rt_sigaction(31,NULL,0xfffef31c) = 0
29923 rt_sigaction(31,0xfffef304,NULL) = 0
29923 rt_sigaction(32,NULL,0xfffef31c) = 0
29923 rt_sigaction(33,NULL,0xfffef31c) = 0
29923 rt_sigaction(34,NULL,0xfffef31c) = 0
29923 rt_sigaction(34,0xfffef304,NULL) = 0
29923 rt_sigaction(35,NULL,0xfffef31c) = 0
29923 rt_sigaction(35,0xfffef304,NULL) = 0
29923 rt_sigaction(36,NULL,0xfffef31c) = 0
29923 rt_sigaction(36,0xfffef304,NULL) = 0
29923 rt_sigaction(37,NULL,0xfffef31c) = 0
29923 rt_sigaction(37,0xfffef304,NULL) = 0
29923 rt_sigaction(38,NULL,0xfffef31c) = 0
29923 rt_sigaction(38,0xfffef304,NULL) = 0
29923 rt_sigaction(39,NULL,0xfffef31c) = 0
29923 rt_sigaction(39,0xfffef304,NULL) = 0
29923 rt_sigaction(40,NULL,0xfffef31c) = 0
29923 rt_sigaction(40,0xfffef304,NULL) = 0
29923 rt_sigaction(41,NULL,0xfffef31c) = 0
29923 rt_sigaction(41,0xfffef304,NULL) = 0
29923 rt_sigaction(42,NULL,0xfffef31c) = 0
29923 rt_sigaction(42,0xfffef304,NULL) = 0
29923 rt_sigaction(43,NULL,0xfffef31c) = 0
29923 rt_sigaction(43,0xfffef304,NULL) = 0
29923 rt_sigaction(44,NULL,0xfffef31c) = 0
29923 rt_sigaction(44,0xfffef304,NULL) = 0
29923 rt_sigaction(45,NULL,0xfffef31c) = 0
29923 rt_sigaction(45,0xfffef304,NULL) = 0
29923 rt_sigaction(46,NULL,0xfffef31c) = 0
29923 rt_sigaction(46,0xfffef304,NULL) = 0
29923 rt_sigaction(47,NULL,0xfffef31c) = 0
29923 rt_sigaction(47,0xfffef304,NULL) = 0
29923 rt_sigaction(48,NULL,0xfffef31c) = 0
29923 rt_sigaction(48,0xfffef304,NULL) = 0
29923 rt_sigaction(49,NULL,0xfffef31c) = 0
29923 rt_sigaction(49,0xfffef304,NULL) = 0
29923 rt_sigaction(50,NULL,0xfffef31c) = 0
29923 rt_sigaction(50,0xfffef304,NULL) = 0
29923 rt_sigaction(51,NULL,0xfffef31c) = 0
29923 rt_sigaction(51,0xfffef304,NULL) = 0
29923 rt_sigaction(52,NULL,0xfffef31c) = 0
29923 rt_sigaction(52,0xfffef304,NULL) = 0
29923 rt_sigaction(53,NULL,0xfffef31c) = 0
29923 rt_sigaction(53,0xfffef304,NULL) = 0
29923 rt_sigaction(54,NULL,0xfffef31c) = 0
29923 rt_sigaction(54,0xfffef304,NULL) = 0
29923 rt_sigaction(55,NULL,0xfffef31c) = 0
29923 rt_sigaction(55,0xfffef304,NULL) = 0
29923 rt_sigaction(56,NULL,0xfffef31c) = 0
29923 rt_sigaction(56,0xfffef304,NULL) = 0
29923 rt_sigaction(57,NULL,0xfffef31c) = 0
29923 rt_sigaction(57,0xfffef304,NULL) = 0
29923 rt_sigaction(58,NULL,0xfffef31c) = 0
29923 rt_sigaction(58,0xfffef304,NULL) = 0
29923 rt_sigaction(59,NULL,0xfffef31c) = 0
29923 rt_sigaction(59,0xfffef304,NULL) = 0
29923 rt_sigaction(60,NULL,0xfffef31c) = 0
29923 rt_sigaction(60,0xfffef304,NULL) = 0
29923 rt_sigaction(61,NULL,0xfffef31c) = 0
29923 rt_sigaction(61,0xfffef304,NULL) = 0
29923 rt_sigaction(62,NULL,0xfffef31c) = 0
29923 rt_sigaction(62,0xfffef304,NULL) = 0
29923 rt_sigaction(63,NULL,0xfffef31c) = 0
29923 rt_sigaction(63,0xfffef304,NULL) = 0
29923 rt_sigaction(64,NULL,0xfffef31c) = 0
29923 rt_sigaction(64,0xfffef304,NULL) = -1 errno=22 (Invalid argument)
29923 write(2,0xb134a,13)fatal error:  = 13
29923 write(2,0xb1a98,16)sigaction failed = 16
29923 write(2,0xb017b,1)
 = 1

[other runtime/panic calls elided]

Answer 9 · 2018-05-02T21:01:48.000Z

Thanks. So the problem seems to be that QEMU does not permit us to set the signal handler for signal 64. That seems like a bug in QEMU. And what has changed is probably that we used to simply ignore the result of sigaction, but now, as of https://golang.org/cl/99077, we check it.

Answer 10 · 2018-05-02T21:22:56.000Z

I think we re-discovered this issue from 2013 (or it went away and now it's back):

https://groups.google.com/d/topic/golang-nuts/MqKTX_XIOKE/discussion

Answer 11 · 2018-05-02T21:40:18.000Z

I think you're right. Perhaps someone could actually report it to QEMU.

Answer 12 · 2018-05-03T08:00:26.000Z

I think they're aware. From here: https://git.qemu.org/?p=qemu.git;a=blob;f=linux-user/signal.c;h=a3022c2f04e83ddf370d30905e51b6fa62c1f23d;hb=HEAD#l77

/* Nasty hack: Reverse SIGRTMIN and SIGRTMAX to avoid overlap with
   host libpthread signals.  This assumes no one actually uses SIGRTMAX :-/
   To fix this properly we need to do manual signal delivery multiplexed
   over a single host signal.  */
[__SIGRTMIN] = __SIGRTMAX,
[__SIGRTMAX] = __SIGRTMIN,

I couldn't find a bug in their tracker, but I found a thread where someone was complaining about not being able to run Go binaries: https://lists.gnu.org/archive/html/qemu-discuss/2016-03/msg00002.html

Also it appears that we had workarounds in place for this issue in the past:

https://codereview.appspot.com/124900043/
https://go-review.googlesource.com/c/go/+/16853

Either we add the signal 64 workaround back or we declare once and for all that running Go binaries under qemu-user is not supported and it's not supposed to work (until they change the way they handle signals).

Answer 13 · 2018-05-03T08:20:39.000Z

Change https://golang.org/cl/111176 mentions this issue: runtime: ignore sigaction error if it is for SIGRTMAX

Answer 14 · 2018-05-03T08:22:30.000Z

I've verified that, with the change linked above applied on tip, Go binaries no longer crash at start-up under qemu-user.

Answer 15 · 2018-05-04T00:08:03.000Z

Thanks, let's do that, I guess.

Answer 16 · 2018-07-10T00:33:33.000Z

As far as I can tell, this is fixed, so closing.

What version of Go are you using (go version)?