runtime: unexpected return pc for runtime.systemstack_switch

Question

runtime: unexpected return pc for runtime.systemstack_switch

myitcv opened this issue 6 years ago · 3 comments

What version of Go are you using (`go version`)?

$ go version
go version devel +3f21c2381d Thu Nov 14 01:57:00 2019 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes (although I cannot trigger this, it just happened randomly)

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myitcv/.cache/go-build"
GOENV="/home/myitcv/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/myitcv/gostuff"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/myitcv/gos"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/myitcv/gostuff/src/github.com/myitcv/govim/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build266412837=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Ran go test -short -count=1 ./... in a project of mine

What did you expect to see?

Regular test output

What did you see instead?

# golang.org/x/tools/go/packages/packagestest
runtime: unexpected return pc for runtime.systemstack_switch called from 0x478882bf1e330a7e
stack: frame={sp:0xc0001c2cf8, fp:0xc0001c2d00} stack=[0xc0001c0000,0xc0001c4000)
000000c0001c2bf8:  0000000000000000  0000000000000000
000000c0001c2c08:  0000000000000000  0000000000000000
000000c0001c2c18:  0000000000000000  0000000000000000
000000c0001c2c28:  0000000000000000  0000000000000000
000000c0001c2c38:  0000000000000000  0000000000000000
000000c0001c2c48:  0000000000000000  0000000000000000
000000c0001c2c58:  0000000000000000  0000000000000000
000000c0001c2c68:  0000000000000000  0000000000000000
000000c0001c2c78:  0000000000000000  0000000000000000
000000c0001c2c88:  0000000000000000  0000000000000000
000000c0001c2c98:  0000000000000000  0000000000000000
000000c0001c2ca8:  0000000000000000  0000000000000000
000000c0001c2cb8:  0000000000000000  0000000000000000
000000c0001c2cc8:  0000000000000000  0000000000000000
000000c0001c2cd8:  0000000000000000  0000000000000000
000000c0001c2ce8:  0000000000000000  0000000000000000
000000c0001c2cf8: <478882bf1e330a7e >d32669b5054179d9
000000c0001c2d08:  0000000000000000  0000000000000000
000000c0001c2d18:  0000000000000000  41cab01ac6de082f
000000c0001c2d28:  b892a3226ac6c35d  0000000000000000
000000c0001c2d38:  0000000000000000  88d9935870d34805
000000c0001c2d48:  1b2f424819f18fa4  0000000000000000
000000c0001c2d58:  0000000000000000  549bb81db167bef5
000000c0001c2d68:  3e1298d7ab4da028  0000000000000000
000000c0001c2d78:  0000000000000000  000000c000452df0
000000c0001c2d88:  00000000004bc4d5 <internal/poll.(*FD).Read+485>  000000c0004459e0
000000c0001c2d98:  000000c0004dfe00  0000000000020000
000000c0001c2da8:  0000000000020000  0000000000020000
000000c0001c2db8:  0000000000000000  0000000000000000
000000c0001c2dc8:  000000c000000180  0000000000020000
000000c0001c2dd8:  000000c0004459f8  000000c0004459e0
000000c0001c2de8:  0000000000776340  000000c000452e20
000000c0001c2df8:  00000000004477cc <runtime.makeslice+108>
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x764701, 0x11)
        /home/myitcv/dev/go/src/runtime/panic.go:1106 +0x72
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000000180, 0x0, 0x0, 0x7fffffff, 0x7764d8, 0xc000057b08, 0x0, ...)
        /home/myitcv/dev/go/src/runtime/traceback.go:273 +0x19fb
runtime.copystack(0xc000000180, 0x4000)
        /home/myitcv/dev/go/src/runtime/stack.go:888 +0x291
runtime.shrinkstack(0xc000000180)
        /home/myitcv/dev/go/src/runtime/stack.go:1140 +0x128
runtime.scanstack(0xc000000180, 0xc00002de90)
        /home/myitcv/dev/go/src/runtime/mgcmark.go:712 +0x538
runtime.markroot.func1()
        /home/myitcv/dev/go/src/runtime/mgcmark.go:228 +0xbf
runtime.markroot(0xc00002de90, 0x5)
        /home/myitcv/dev/go/src/runtime/mgcmark.go:201 +0x2f3
runtime.gcDrain(0xc00002de90, 0x7)
        /home/myitcv/dev/go/src/runtime/mgcmark.go:1001 +0x107
runtime.gcBgMarkWorker.func2()
        /home/myitcv/dev/go/src/runtime/mgc.go:1964 +0x169
runtime.systemstack(0x0)
        /home/myitcv/dev/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
        /home/myitcv/dev/go/src/runtime/proc.go:1077

goroutine 33 [GC worker (idle)]:
runtime.systemstack_switch()
        /home/myitcv/dev/go/src/runtime/asm_amd64.s:330 fp=0xc000098760 sp=0xc000098758 pc=0x45d5a0
runtime.gcBgMarkWorker(0xc00002c800)
        /home/myitcv/dev/go/src/runtime/mgc.go:1928 +0x1be fp=0xc0000987d8 sp=0xc000098760 pc=0x41b65e
runtime.goexit()
        /home/myitcv/dev/go/src/runtime/asm_amd64.s:1375 +0x1 fp=0xc0000987e0 sp=0xc0000987d8 pc=0x45f531
created by runtime.gcBgMarkStartWorkers
        /home/myitcv/dev/go/src/runtime/mgc.go:1822 +0x77

goroutine 1 [GC assist marking (scan)]:
runtime: unexpected return pc for runtime.systemstack_switch called from 0x478882bf1e330a7e
stack: frame={sp:0xc0001c2cf8, fp:0xc0001c2d00} stack=[0xc0001c0000,0xc0001c4000)
000000c0001c2bf8:  0000000000000000  0000000000000000
000000c0001c2c08:  0000000000000000  0000000000000000
000000c0001c2c18:  0000000000000000  0000000000000000
000000c0001c2c28:  0000000000000000  0000000000000000
000000c0001c2c38:  0000000000000000  0000000000000000
000000c0001c2c48:  0000000000000000  0000000000000000
000000c0001c2c58:  0000000000000000  0000000000000000
000000c0001c2c68:  0000000000000000  0000000000000000
000000c0001c2c78:  0000000000000000  0000000000000000
000000c0001c2c88:  0000000000000000  0000000000000000
000000c0001c2c98:  0000000000000000  0000000000000000
000000c0001c2ca8:  0000000000000000  0000000000000000
000000c0001c2cb8:  0000000000000000  0000000000000000
000000c0001c2cc8:  0000000000000000  0000000000000000
000000c0001c2cd8:  0000000000000000  0000000000000000
000000c0001c2ce8:  0000000000000000  0000000000000000
000000c0001c2cf8: <478882bf1e330a7e >d32669b5054179d9
000000c0001c2d08:  0000000000000000  0000000000000000
000000c0001c2d18:  0000000000000000  41cab01ac6de082f
000000c0001c2d28:  b892a3226ac6c35d  0000000000000000
000000c0001c2d38:  0000000000000000  88d9935870d34805
000000c0001c2d48:  1b2f424819f18fa4  0000000000000000
000000c0001c2d58:  0000000000000000  549bb81db167bef5
000000c0001c2d68:  3e1298d7ab4da028  0000000000000000
000000c0001c2d78:  0000000000000000  000000c000452df0
000000c0001c2d88:  00000000004bc4d5 <internal/poll.(*FD).Read+485>  000000c0004459e0
000000c0001c2d98:  000000c0004dfe00  0000000000020000
000000c0001c2da8:  0000000000020000  0000000000020000
000000c0001c2db8:  0000000000000000  0000000000000000
000000c0001c2dc8:  000000c000000180  0000000000020000
000000c0001c2dd8:  000000c0004459f8  000000c0004459e0
000000c0001c2de8:  0000000000776340  000000c000452e20
000000c0001c2df8:  00000000004477cc <runtime.makeslice+108>
runtime: unexpected return pc for runtime.systemstack_switch called from 0x478882bf1e330a7e
stack: frame={sp:0xc0001c2cf8, fp:0xc0001c2d00} stack=[0xc0001c0000,0xc0001c4000)
000000c0001c2bf8:  0000000000000000  0000000000000000
000000c0001c2c08:  0000000000000000  0000000000000000
000000c0001c2c18:  0000000000000000  0000000000000000
000000c0001c2c28:  0000000000000000  0000000000000000
000000c0001c2c38:  0000000000000000  0000000000000000
000000c0001c2c48:  0000000000000000  0000000000000000
000000c0001c2c58:  0000000000000000  0000000000000000
000000c0001c2c68:  0000000000000000  0000000000000000
000000c0001c2c78:  0000000000000000  0000000000000000
000000c0001c2c88:  0000000000000000  0000000000000000
000000c0001c2c98:  0000000000000000  0000000000000000
000000c0001c2ca8:  0000000000000000  0000000000000000
000000c0001c2cb8:  0000000000000000  0000000000000000
000000c0001c2cc8:  0000000000000000  0000000000000000
000000c0001c2cd8:  0000000000000000  0000000000000000
000000c0001c2ce8:  0000000000000000  0000000000000000
000000c0001c2cf8: <478882bf1e330a7e >d32669b5054179d9
000000c0001c2d08:  0000000000000000  0000000000000000
000000c0001c2d18:  0000000000000000  41cab01ac6de082f
000000c0001c2d28:  b892a3226ac6c35d  0000000000000000
000000c0001c2d38:  0000000000000000  88d9935870d34805
000000c0001c2d48:  1b2f424819f18fa4  0000000000000000
000000c0001c2d58:  0000000000000000  549bb81db167bef5
000000c0001c2d68:  3e1298d7ab4da028  0000000000000000
000000c0001c2d78:  0000000000000000  000000c000452df0
000000c0001c2d88:  00000000004bc4d5 <internal/poll.(*FD).Read+485>  000000c0004459e0
000000c0001c2d98:  000000c0004dfe00  0000000000020000
000000c0001c2da8:  0000000000020000  0000000000020000
000000c0001c2db8:  0000000000000000  0000000000000000
000000c0001c2dc8:  000000c000000180  0000000000020000
000000c0001c2dd8:  000000c0004459f8  000000c0004459e0
000000c0001c2de8:  0000000000776340  000000c000452e20
000000c0001c2df8:  00000000004477cc <runtime.makeslice+108>
runtime.systemstack_switch()
        /home/myitcv/dev/go/src/runtime/asm_amd64.s:330

I've tentatively marked as a release blocked because it looks similar to #35328 and #35326

cc @randall77 @aclements

Answer 1 · 2019-11-20T18:54:39.000Z

Another occurrence: https://storage.googleapis.com/go-build-log/90c91afa/linux-amd64-race_b23ed843.log

CC @cherrymui @ianlancetaylor

Answer 2 · 2019-11-22T15:11:07.000Z

This is almost certainly the same issue as #35689. Note the similarity between the bad words at the top of the stack (478882bf1e330a7e, d32669b5054179d9, etc) and the bad pointers in the stack trace from #35689 (637469796d2f656d).

My guess is that this is related to Linux 5.3.x, but waiting for confirmation in #35689.

Answer 3 · 2019-11-22T23:11:26.000Z

Kernel version confirmed. Folding into the super-bug (#35777).

What version of Go are you using (go version)?