crypto/internal/fips140/sha256: sporadic panics when calling "(crypto/x509).SystemCertPool()"
Closed this issue · 8 comments
leonklingele commented
Go version
go1.25.3
Output of go env in your module/workspace:
This was observed on a prod system. `go version -m $BINARY_PATH` yields the following:
[BINARY_NAME]: go1.25.3
[.. REDACTED ..]
build -buildmode=exe
build -compiler=gc
build -ldflags="-w -s -X [REDACTED]"
build CGO_ENABLED=0
build GOARCH=arm64
build GOOS=linux
build GOARM64=v8.0
[.. REDACTED ..]What did you do?
We observed the following panic on one of our prod systems:
panic: runtime error: slice bounds out of range [3616445671586817383:64]
goroutine 1 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:26 +0x5e
main.main.func1()
/go/src/[REDACTED]/main.go:306 +0x39
panic({0x1027e80?, 0xc0003817e8?})
/usr/local/go/src/runtime/panic.go:783 +0x132
crypto/internal/fips140/sha256.(*Digest).Write(0xc0005667f0, {0xc000229500?, 0x33c, 0x870?})
/usr/local/go/src/crypto/internal/fips140/sha256/sha256.go:176 +0x274
crypto/sha256.Sum224({0xc000229500, 0x33c, 0x348})
/usr/local/go/src/crypto/sha256/sha256.go:70 +0xa7
crypto/x509.(*CertPool).AppendCertsFromPEM(0xc0004ec360, {0xc000600000?, 0x0?, 0x10ac6c8?})
/usr/local/go/src/crypto/x509/cert_pool.go:239 +0x179
crypto/x509.loadSystemRoots()
/usr/local/go/src/crypto/x509/root_unix.go:72 +0x3a5
crypto/x509.initSystemRoots()
/usr/local/go/src/crypto/x509/root.go:40 +0x56
sync.(*Once).doSlow(0xc00012c1a8?, 0xc000022b10?)
/usr/local/go/src/sync/once.go:78 +0xac
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:69
crypto/x509.systemRootsPool()
/usr/local/go/src/crypto/x509/root.go:31 +0x45
crypto/x509.SystemCertPool()
/usr/local/go/src/crypto/x509/cert_pool.go:118 +0x25
[REDACTED]/pkg/auth.TLSConfigFromFile({0x10ba74e?, 0x11}, {0x0, 0x0})
/go/src/[REDACTED]/pkg/auth/token.go:380 +0x66
main.main()
/go/src/[REDACTED]/main.go:532 +0x386c
Where [REDACTED]/pkg/auth/token.go includes the following:
import (
"crypto/x509"
)
cp, err := x509.SystemCertPool() // Line 380
We did not investigate this yet but the lower bound in slice bounds out of range [3616445671586817383:64] seems very incorrect.
The panic itself occurs here: https://github.com/golang/go/blob/go1.25.3/src/crypto/internal/fips140/sha256/sha256.go#L176
Find attached the system root store in use on that system (/etc/ssl/certs/ca-certificates.crt which was taken from arm64v8/debian:bookworm-slim):
ca-certificates.crt
What did you see happen?
See above.
What did you expect to see?
Panic should not occur.
leonklingele commented
I was able to reproduce this (or a possibly unrelated issue) using the following simple Go program:
package main
import (
"crypto/x509"
"log"
"runtime/debug"
)
func main() {
defer func() {
if err := recover(); err != nil {
log.Fatalf("panic: %v\n%s", err, debug.Stack())
}
}()
cp, err := x509.SystemCertPool()
if err != nil {
panic(err)
}
println(cp)
}which I called repeatedly with until ! ./test ; do echo '' > /dev/null ; done until a panic would occur.
It took around an hour to run and finally ended up producing the following panic:
SIGSEGV: segmentation violation
PC=0x42c189 m=3 sigcode=128 addr=0x0
goroutine 0 gp=0xc000003340 m=3 mp=0xc000041008 [idle]:
runtime.(*sweepLocked).sweep(0x0?, 0x0)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgcsweep.go:556 +0x1c9 fp=0xc000055e38 sp=0xc000055d10 pc=0x42c189
runtime.(*mcentral).uncacheSpan(0x0?, 0xc000055e90?)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mcentral.go:237 +0x98 fp=0xc000055e60 sp=0xc000055e38 pc=0x41a9d8
runtime.(*mcache).releaseAll(0x7f7652c6ff30)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mcache.go:301 +0x13e fp=0xc000055ec8 sp=0xc000055e60 pc=0x41a31e
runtime.(*mcache).prepareForSweep(0x7f7652c6ff30)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mcache.go:338 +0x35 fp=0xc000055ef0 sp=0xc000055ec8 pc=0x41a415
runtime.gcMarkTermination.func4(0xc000031208)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1228 +0x25 fp=0xc000055f18 sp=0xc000055ef0 pc=0x46dac5
runtime.forEachPInternal(0x5898d0)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:2161 +0x12b fp=0xc000055f98 sp=0xc000055f18 pc=0x44490b
runtime.gcMarkTermination.forEachP.func6()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:2120 +0x3f fp=0xc000055fc8 sp=0xc000055f98 pc=0x421d7f
runtime.systemstack(0x0)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/asm_amd64.s:513 +0x4a fp=0xc000055fd8 sp=0xc000055fc8 pc=0x47628a
goroutine 11 gp=0xc0003afdc0 m=3 mp=0xc000041008 [flushing proc caches]:
runtime.systemstack_switch()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/asm_amd64.s:478 +0x8 fp=0xc00006fa28 sp=0xc00006fa18 pc=0x476228
runtime.forEachP(...)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:2101
runtime.gcMarkTermination({0x1?, 0x4494f728808ba?, 0x4494f728955c6?, 0x36384?})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1227 +0x605 fp=0xc00006fe98 sp=0xc00006fa28 pc=0x421285
runtime.gcMarkDone()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1032 +0x369 fp=0xc00006ff38 sp=0xc00006fe98 pc=0x420a29
runtime.gcBgMarkWorker(0xc000484070)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1600 +0x345 fp=0xc00006ffc8 sp=0xc00006ff38 pc=0x422405
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1373 +0x25 fp=0xc00006ffe0 sp=0xc00006ffc8 pc=0x422085
runtime.goexit({})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x4780c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1373 +0x105
goroutine 1 gp=0xc000002380 m=nil [runnable]:
vendor/golang.org/x/crypto/cryptobyte.(*String).ReadASN1(0xc0001136b8?, 0xc000113608?, 0x31)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/vendor/golang.org/x/crypto/cryptobyte/asn1.go:599 +0x5c fp=0xc000113590 sp=0xc000113588 pc=0x4dd57c
crypto/x509.parseName({0xc000490531, 0xbd, 0x4a0})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/crypto/x509/parser.go:153 +0xd7 fp=0xc0001136b8 sp=0xc000113590 pc=0x4ee0f7
crypto/x509.parseCertificate({0xc000490500, 0x4bd, 0x4d1})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/crypto/x509/parser.go:994 +0x50d fp=0xc000113b20 sp=0xc0001136b8 pc=0x4f418d
crypto/x509.ParseCertificate({0xc000490500?, 0x4bd, 0xc8e?})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/crypto/x509/parser.go:1108 +0x1d fp=0xc000113b48 sp=0xc000113b20 pc=0x4f501d
crypto/x509.(*CertPool).AppendCertsFromPEM(0xc0000786f0, {0xc000400000?, 0x67?, 0x57b3d9?})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/crypto/x509/cert_pool.go:231 +0x11e fp=0xc000113c48 sp=0xc000113b48 pc=0x4ecdbe
crypto/x509.loadSystemRoots()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/crypto/x509/root_unix.go:72 +0x3a5 fp=0xc000113d28 sp=0xc000113c48 pc=0x4f56e5
crypto/x509.initSystemRoots()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/crypto/x509/root.go:40 +0x56 fp=0xc000113d60 sp=0xc000113d28 pc=0x4f5236
sync.(*Once).doSlow(0x0?, 0xc000070de8?)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/sync/once.go:78 +0xac fp=0xc000113db8 sp=0xc000113d60 pc=0x47c8ac
sync.(*Once).Do(...)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/sync/once.go:69
crypto/x509.systemRootsPool()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/crypto/x509/root.go:31 +0x45 fp=0xc000113e08 sp=0xc000113db8 pc=0x4f50e5
crypto/x509.SystemCertPool()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/crypto/x509/cert_pool.go:118 +0x25 fp=0xc000113f18 sp=0xc000113e08 pc=0x4ec985
main.main()
/private/var/folders/g6/klk00pdn7xj6rbjxmhm34nb00000gn/T/tmp.ZfWdyMoXFa/main.go:16 +0x30 fp=0xc000113f50 sp=0xc000113f18 pc=0x4f8f10
runtime.main()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:285 +0x29d fp=0xc000113fe0 sp=0xc000113f50 pc=0x440e3d
runtime.goexit({})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000113fe8 sp=0xc000113fe0 pc=0x4780c1
goroutine 2 gp=0xc0000028c0 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:460 +0xce fp=0xc00003afa8 sp=0xc00003af88 pc=0x47158e
runtime.goparkunlock(...)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:466
runtime.forcegchelper()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:373 +0xb3 fp=0xc00003afe0 sp=0xc00003afa8 pc=0x441173
runtime.goexit({})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00003afe8 sp=0xc00003afe0 pc=0x4780c1
created by runtime.init.7 in goroutine 1
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:361 +0x1a
goroutine 3 gp=0xc000002e00 m=nil [runnable]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:460 +0xce fp=0xc00003b780 sp=0xc00003b760 pc=0x47158e
runtime.goparkunlock(...)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:466
runtime.bgsweep(0xc00005a000)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgcsweep.go:279 +0x94 fp=0xc00003b7c8 sp=0xc00003b780 pc=0x42ba74
runtime.gcenable.gowrap1()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:212 +0x25 fp=0xc00003b7e0 sp=0xc00003b7c8 pc=0x41fcc5
runtime.goexit({})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00003b7e8 sp=0xc00003b7e0 pc=0x4780c1
created by runtime.gcenable in goroutine 1
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:212 +0x66
goroutine 4 gp=0xc000002fc0 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x5ba010?, 0x0?, 0x0?, 0x0?)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:460 +0xce fp=0xc00003bf78 sp=0xc00003bf58 pc=0x47158e
runtime.goparkunlock(...)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:466
runtime.(*scavengerState).park(0x6cbbc0)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00003bfa8 sp=0xc00003bf78 pc=0x429549
runtime.bgscavenge(0xc00005a000)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00003bfc8 sp=0xc00003bfa8 pc=0x429af9
runtime.gcenable.gowrap2()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:213 +0x25 fp=0xc00003bfe0 sp=0xc00003bfc8 pc=0x41fc65
runtime.goexit({})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00003bfe8 sp=0xc00003bfe0 pc=0x4780c1
created by runtime.gcenable in goroutine 1
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:213 +0xa5
goroutine 5 gp=0xc000003500 m=nil [GOMAXPROCS updater (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:460 +0xce fp=0xc00003c788 sp=0xc00003c768 pc=0x47158e
runtime.goparkunlock(...)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:466
runtime.updateMaxProcsGoroutine()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:6706 +0xe7 fp=0xc00003c7e0 sp=0xc00003c788 pc=0x44ef67
runtime.goexit({})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00003c7e8 sp=0xc00003c7e0 pc=0x4780c1
created by runtime.defaultGOMAXPROCSUpdateEnable in goroutine 1
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:6694 +0x37
goroutine 6 gp=0xc0000036c0 m=nil [finalizer wait]:
runtime.gopark(0x44ff35?, 0x428a3c?, 0x0?, 0xd2?, 0x490013?)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:460 +0xce fp=0xc00003a620 sp=0xc00003a600 pc=0x47158e
runtime.runFinalizers()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mfinal.go:210 +0x107 fp=0xc00003a7e0 sp=0xc00003a620 pc=0x41ebc7
runtime.goexit({})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00003a7e8 sp=0xc00003a7e0 pc=0x4780c1
created by runtime.createfing in goroutine 1
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mfinal.go:172 +0x3d
goroutine 7 gp=0xc000003880 m=nil [cleanup wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:460 +0xce fp=0xc00003cf68 sp=0xc00003cf48 pc=0x47158e
runtime.goparkunlock(...)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:466
runtime.(*cleanupQueue).dequeue(0x6cbd00)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mcleanup.go:439 +0xc5 fp=0xc00003cfa0 sp=0xc00003cf68 pc=0x41bde5
runtime.runCleanups()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mcleanup.go:635 +0x45 fp=0xc00003cfe0 sp=0xc00003cfa0 pc=0x41c4a5
runtime.goexit({})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00003cfe8 sp=0xc00003cfe0 pc=0x4780c1
created by runtime.(*cleanupQueue).createGs in goroutine 1
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mcleanup.go:589 +0xa5
goroutine 8 gp=0xc0003af880 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:460 +0xce fp=0xc00003d738 sp=0xc00003d718 pc=0x47158e
runtime.gcBgMarkWorker(0xc000484070)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1463 +0xeb fp=0xc00003d7c8 sp=0xc00003d738 pc=0x4221ab
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1373 +0x25 fp=0xc00003d7e0 sp=0xc00003d7c8 pc=0x422085
runtime.goexit({})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00003d7e8 sp=0xc00003d7e0 pc=0x4780c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1373 +0x105
goroutine 9 gp=0xc0003afa40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:460 +0xce fp=0xc00003df38 sp=0xc00003df18 pc=0x47158e
runtime.gcBgMarkWorker(0xc000484070)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1463 +0xeb fp=0xc00003dfc8 sp=0xc00003df38 pc=0x4221ab
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1373 +0x25 fp=0xc00003dfe0 sp=0xc00003dfc8 pc=0x422085
runtime.goexit({})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00003dfe8 sp=0xc00003dfe0 pc=0x4780c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1373 +0x105
goroutine 10 gp=0xc0003afc00 m=nil [runnable]:
runtime.gopark(0x40ea7e?, 0x476272?, 0x0?, 0xc0?, 0x40ebf3?)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:460 +0xce fp=0xc000036630 sp=0xc000036610 pc=0x47158e
runtime.goparkunlock(...)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/proc.go:466
runtime.semacquire1(0x6cc2fc, 0x0, 0x0, 0x0, 0x13)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/sema.go:192 +0x21d fp=0xc000036698 sp=0xc000036630 pc=0x452ffd
runtime.semacquire(...)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/sema.go:143
runtime.gcMarkDone()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:890 +0x32 fp=0xc000036738 sp=0xc000036698 pc=0x4206f2
runtime.gcBgMarkWorker(0xc000484070)
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1600 +0x345 fp=0xc0000367c8 sp=0xc000036738 pc=0x422405
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1373 +0x25 fp=0xc0000367e0 sp=0xc0000367c8 pc=0x422085
runtime.goexit({})
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000367e8 sp=0xc0000367e0 pc=0x4780c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/nix/store/mkdfnr1nkfj2kznxyag9pypbxp3wqqdv-go-1.25.1/share/go/src/runtime/mgc.go:1373 +0x105
rax 0x0
rbx 0x0
rcx 0x0
rdx 0x7f760c1780a0
rdi 0x6231653433383937
rsi 0x2
rbp 0xc000055e28
rsp 0xc000055d10
r8 0x1
r9 0x6231653433383937
r10 0xf
r11 0x50
r12 0xc000055d90
r13 0x0
r14 0xc000003340
r15 0x3
rip 0x42c189
rflags 0x10206
cs 0x33
fs 0x0
gs 0x0
seankhliao commented
I think we'll want to see the full go env.
And does this happen using a non nix packaged go?
leonklingele commented
If you're referring to the segmentation violation panic reported in #76169 (comment) (which might not be the same as the one in the issue report) — Yes! It just took a bit longer to reproduce on a non-Nix-machine (Debian).
$ go env
AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE='auto'
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build110928171=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/root/go/go.mod'
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='go1.25.0+auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.3'
GOWORK=''
PKG_CONFIG='pkg-config'prattmic commented
cc @golang/security @golang/runtime
prattmic commented
Yes! It just took a bit longer to reproduce on a non-Nix-machine (Debian).
To clarify, you got the same panic #76169 (comment) (same exact PC, sigcode, and addr) on a different physical machine?
Edit: looks like the original was linux-arm64 and the new machine is linux-amd64? So the PC would be different even if the crash is identical.
FiloSottile commented
3616445671586817383 is 0011001000110000001100100010110101110100011011100110010101100111, and nx is only ever assigned to 0, assigned to the return value of copy, or incremented by the return value of copy, so this is definitely memory corruption of some sort, there is no way for the program to reach that value.
On Linux, I don't think x509.SystemCertPool does anything special or spicy (unlike on macOS and Windows), just reading files, parsing them, hashing them.
leonklingele commented
We're now properly investigating this. It looks like some kind of hardware defect as we weren't able to reproduce this on similar hardware so far.
ftr: This night another panic was found (on the "borked" device)
SIGSEGV: segmentation violation
PC=0x4a80e0 m=4 sigcode=1 addr=0x210b6
#
goroutine 0 gp=0xc0000821c0 m=4 mp=0xc000080008 [idle]:
runtime.gcResetMarkState.func1(0x400000002?)
/usr/local/go/src/runtime/mgc.go:1784 fp=0x7f114989cdf0 sp=0x7f114989cde8 pc=0x4a80e0
runtime.forEachG(0x650868)
/usr/local/go/src/runtime/proc.go:732 +0x55 fp=0x7f114989ce20 sp=0x7f114989cdf0 pc=0x47aa55
runtime.gcResetMarkState()
/usr/local/go/src/runtime/mgc.go:1783 +0x1e fp=0x7f114989ce50 sp=0x7f114989ce20 pc=0x45ba7e
runtime.systemstack(0x800000)
/usr/local/go/src/runtime/asm_amd64.s:513 +0x4a fp=0x7f114989ce60 sp=0x7f114989ce50 pc=0x4b104a
#
goroutine 1 gp=0xc000002380 m=4 mp=0xc000080008 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:478 +0x8 fp=0xc000149240 sp=0xc000149230 pc=0x4b0fe8
runtime.gcStart({0x7f1192536a78?, 0x441f48?, 0x4b46d9?})
/usr/local/go/src/runtime/mgc.go:728 +0x32f fp=0xc000149338 sp=0xc000149240 pc=0x458e8f
runtime.mallocgcSmallScanHeader(0x4b5ee9?, 0x63ff40)
/usr/local/go/src/runtime/malloc.go:1530 +0x2fe fp=0xc000149390 sp=0xc000149338 pc=0x44fb1e
runtime.mallocgc(0x590, 0x63ff40, 0x1)
/usr/local/go/src/runtime/malloc.go:1068 +0xac fp=0xc0001493e0 sp=0xc000149390 pc=0x4aa4ec
runtime.newobject(0xc000253d77?)
/usr/local/go/src/runtime/malloc.go:1747 +0x25 fp=0xc000149408 sp=0xc0001493e0 pc=0x450125
crypto/x509.parseCertificate({0xc000482d80, 0x437, 0x447})
/usr/local/go/src/crypto/x509/parser.go:895 +0x51 fp=0xc000149950 sp=0xc000149408 pc=0x57cc31
crypto/x509.ParseCertificate({0xc000482d80, 0x437, 0x447})
/usr/local/go/src/crypto/x509/parser.go:1085 +0x3f fp=0xc000149998 sp=0xc000149950 pc=0x57ec7f
crypto/x509.(*CertPool).AppendCertsFromPEM(0xc0001246f0, {0xc000580000, 0x34311, 0x34312})
/usr/local/go/src/crypto/x509/cert_pool.go:231 +0x4b5 fp=0xc000149af8 sp=0xc000149998 pc=0x571ab5
crypto/x509.loadSystemRoots()
/usr/local/go/src/crypto/x509/root_unix.go:72 +0x55c fp=0xc000149be8 sp=0xc000149af8 pc=0x57f6dc
crypto/x509.initSystemRoots()
/usr/local/go/src/crypto/x509/root.go:40 +0x73 fp=0xc000149c58 sp=0xc000149be8 pc=0x57f013
sync.(*Once).doSlow(0x7e4b10, 0x6505c8)
/usr/local/go/src/sync/once.go:78 +0xd2 fp=0xc000149cf8 sp=0xc000149c58 pc=0x4b9312
sync.(*Once).Do(0x7e4b10, 0x6505c8)
/usr/local/go/src/sync/once.go:69 +0x45 fp=0xc000149d18 sp=0xc000149cf8 pc=0x4b9205
crypto/x509.systemRootsPool()
/usr/local/go/src/crypto/x509/root.go:31 +0x38 fp=0xc000149d88 sp=0xc000149d18 pc=0x57ee78
crypto/x509.SystemCertPool()
/usr/local/go/src/crypto/x509/cert_pool.go:118 +0x2f fp=0xc000149ee8 sp=0xc000149d88 pc=0x57108f
main.main()
/root/go/main.go:16 +0x37 fp=0xc000149f50 sp=0xc000149ee8 pc=0x5853d7
runtime.main()
/usr/local/go/src/runtime/proc.go:285 +0x29d fp=0xc000149fe0 sp=0xc000149f50 pc=0x479bbd
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000149fe8 sp=0xc000149fe0 pc=0x4b3001
#
goroutine 0 gp=0x7f113c000870 m=139713448369456 mp=0x7f119275dc60 [idle]:
runtime: g 0 gp=0x7f113c000870: unknown pc 0x7f119275dfd0
stack: frame={sp:0x7f119275dfd0, fp:0x0} stack=[0x7f113c000860,0x7f113c000860)
#
runtime: g 0 gp=0x7f113c000870: unknown pc 0x7f119275dfd0
stack: frame={sp:0x7f119275dfd0, fp:0x0} stack=[0x7f113c000860,0x7f113c000860)
#
#
goroutine 0 gp=0x7f113c000870 m=139713448369456 mp=0x7f119275dc60 [idle]:
runtime: g 0 gp=0x7f113c000870: unknown pc 0x7f119275dfd0
stack: frame={sp:0x7f119275dfd0, fp:0x0} stack=[0x7f113c000860,0x7f113c000860)
#
runtime: g 0 gp=0x7f113c000870: unknown pc 0x7f119275dfd0
stack: frame={sp:0x7f119275dfd0, fp:0x0} stack=[0x7f113c000860,0x7f113c000860)
#
/bin/bash: line 1: 25492 Segmentation fault (core dumped) /root/go/testleonklingele commented
We boiled this down to a Kernel issue. Sorry for the disruption.
Possibly related: