panic in http2

Question

panic in http2

BastienDurel opened this issue a year ago · 7 comments

Hello,
a newly installed meerkat on a newly-installed debian 12 panics after a few requests

When I'm starting the meerkat server, I can refresh my dashboard once, and after that http2 crashes and it's broken

Logs

Nov 03 11:32:43 SRV-DSI systemd[1]: Started meerkat.service - meerkat.
Nov 03 11:32:44 SRV-DSI meerkat[572008]: 2023/11/03 11:32:44 Subscribing to event streams
Nov 03 11:32:44 SRV-DSI meerkat[572008]: 2023/11/03 11:32:44 Starting https web server on https://0.0.0.0:8080
Nov 03 11:32:50 SRV-DSI meerkat[572008]: 2023/11/03 11:32:50 http2: panic serving 192.168.198.22:39036: runtime error: invalid memory address or nil pointer dereference
Nov 03 11:32:50 SRV-DSI meerkat[572008]: goroutine 247 [running]:
Nov 03 11:32:50 SRV-DSI meerkat[572008]: net/http.(*http2serverConn).runHandler.func1()
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/h2_bundle.go:5903 +0x125
Nov 03 11:32:50 SRV-DSI meerkat[572008]: panic({0x7f5e60, 0xc8d350})
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/runtime/panic.go:884 +0x212
Nov 03 11:32:50 SRV-DSI meerkat[572008]: log.(*Logger).Output(0xc9eca0, 0x4?, {0xc00002cc30, 0x48})
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/log/log.go:198 +0x356
Nov 03 11:32:50 SRV-DSI meerkat[572008]: log.(*Logger).Println(0xc9eca0, {0xc0000c1400?, 0x1?, 0xc003328980?})
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/log/log.go:226 +0x5a
Nov 03 11:32:50 SRV-DSI meerkat[572008]: main.getObjectHandler({0xa331f0, 0xc0000fe1a8}, 0xc003430d00)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/local/meerkat/meerkat/meerkat/cmd/meerkat/dashboard.go:443 +0x9f8
Nov 03 11:32:50 SRV-DSI meerkat[572008]: net/http.HandlerFunc.ServeHTTP(0x7f3ba0?, {0xa331f0?, 0xc0000fe1a8?}, 0xc0000ef0e0?)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/server.go:2109 +0x2f
Nov 03 11:32:50 SRV-DSI meerkat[572008]: github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc000070540, {0xa331f0, 0xc0000fe1a8}, 0xc003430d00)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:442 +0x216
Nov 03 11:32:50 SRV-DSI meerkat[572008]: net/http.HandlerFunc.ServeHTTP(0xa33778?, {0xa331f0?, 0xc0000fe1a8?}, 0xc8d130?)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/server.go:2109 +0x2f
Nov 03 11:32:50 SRV-DSI meerkat[572008]: github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc000070540, {0xa331f0, 0xc0000fe1a8}, 0xc003430c00)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:88 +0x310
Nov 03 11:32:50 SRV-DSI meerkat[572008]: net/http.serverHandler.ServeHTTP({0xc000043d01?}, {0xa331f0, 0xc0000fe1a8}, 0xc003430c00)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/server.go:2947 +0x30c
Nov 03 11:32:50 SRV-DSI meerkat[572008]: net/http.initALPNRequest.ServeHTTP({{0xa33820?, 0xc0032cc0f0?}, 0xc000153500?, {0xc0031dc000?}}, {0xa331f0, 0xc0000fe1a8}, 0xc003430c00)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/server.go:3556 +0x245
Nov 03 11:32:50 SRV-DSI meerkat[572008]: net/http.(*http2serverConn).runHandler(0xc000043e60?, 0xc000043fd0?, 0x709946?, 0xc0033b2d80?)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/h2_bundle.go:5910 +0x78
Nov 03 11:32:50 SRV-DSI meerkat[572008]: created by net/http.(*http2serverConn).processHeaders
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/h2_bundle.go:5645 +0x5b9
Nov 03 11:32:50 SRV-DSI meerkat[572008]: 2023/11/03 11:32:50 http2: panic serving 192.168.198.22:39036: runtime error: invalid memory address or nil pointer dereference
Nov 03 11:32:50 SRV-DSI meerkat[572008]: goroutine 246 [running]:
Nov 03 11:32:50 SRV-DSI meerkat[572008]: net/http.(*http2serverConn).runHandler.func1()
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/h2_bundle.go:5903 +0x125
Nov 03 11:32:50 SRV-DSI meerkat[572008]: panic({0x7f5e60, 0xc8d350})
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/runtime/panic.go:884 +0x212
Nov 03 11:32:50 SRV-DSI meerkat[572008]: log.(*Logger).Output(0xc9eca0, 0x4?, {0xc0031fd1d0, 0x48})
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/log/log.go:198 +0x356
Nov 03 11:32:50 SRV-DSI meerkat[572008]: log.(*Logger).Println(0xc9eca0, {0xc0000c5400?, 0x1?, 0xc003328980?})
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/log/log.go:226 +0x5a
Nov 03 11:32:50 SRV-DSI meerkat[572008]: main.getObjectHandler({0xa331f0, 0xc0000fe1a0}, 0xc0034b8c00)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/local/meerkat/meerkat/meerkat/cmd/meerkat/dashboard.go:443 +0x9f8
Nov 03 11:32:50 SRV-DSI meerkat[572008]: net/http.HandlerFunc.ServeHTTP(0x7f3ba0?, {0xa331f0?, 0xc0000fe1a0?}, 0xc0000ef020?)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/server.go:2109 +0x2f
Nov 03 11:32:50 SRV-DSI meerkat[572008]: github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc000070540, {0xa331f0, 0xc0000fe1a0}, 0xc0034b8c00)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:442 +0x216
Nov 03 11:32:50 SRV-DSI meerkat[572008]: net/http.HandlerFunc.ServeHTTP(0xa33778?, {0xa331f0?, 0xc0000fe1a0?}, 0xc8d130?)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/server.go:2109 +0x2f
Nov 03 11:32:50 SRV-DSI meerkat[572008]: github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc000070540, {0xa331f0, 0xc0000fe1a0}, 0xc003430a00)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:88 +0x310
Nov 03 11:32:50 SRV-DSI meerkat[572008]: net/http.serverHandler.ServeHTTP({0x62c5c0?}, {0xa331f0, 0xc0000fe1a0}, 0xc003430a00)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/server.go:2947 +0x30c
Nov 03 11:32:50 SRV-DSI meerkat[572008]: net/http.initALPNRequest.ServeHTTP({{0xa33820?, 0xc0032cc0f0?}, 0xc000153500?, {0xc0031dc000?}}, {0xa331f0, 0xc0000fe1a0}, 0xc003430a00)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/server.go:3556 +0x245
Nov 03 11:32:50 SRV-DSI meerkat[572008]: net/http.(*http2serverConn).runHandler(0xa32938?, 0xccf158?, 0x0?, 0x0?)
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/h2_bundle.go:5910 +0x78
Nov 03 11:32:50 SRV-DSI meerkat[572008]: created by net/http.(*http2serverConn).processHeaders
Nov 03 11:32:50 SRV-DSI meerkat[572008]:         /usr/lib/go-1.19/src/net/http/h2_bundle.go:5645 +0x5b9

Browser console :

Desktop

OS: ArchLinux
Browser chromium
Version 119.0.6045.59

Additional context
As long as I don't reload the dashboard page, updates flows correctly, but the first reload causes the panic, and every subsequent load shows the broken page & panics in logs. (even with another browser)
The crash happens too if I open the dashboard in another browser

Answer 1 · 2023-11-03T11:02:09.000Z

(I removed TLS errors from logs : it turns ou it was from a co-worker browser using the IP of the server instead of its FQDN, therefore the certificate was invalid)

Answer 2 · 2023-11-07T15:33:03.000Z

hey @BastienDurel we have manged to reproduce this I believe. Working on a fix now

Answer 3 · 2023-11-07T16:19:24.000Z

Fixed in latest commit just need to create a new release

Answer 4 · 2023-11-07T20:26:22.000Z

Thanks ! :)

Answer 5 · 2023-11-08T03:45:54.000Z

The workaround to this bug is to ensure you have configured the logging options in the meerkat config. The icinga logging check has faulty logic.

Answer 6 · 2023-11-08T05:06:25.000Z

Released a new version with this fix v3.1.5

Answer 7 · 2023-11-08T08:49:06.000Z

Thanks ! :)