redpanda-data/kminion

Kminion 2.2.1 panics time after time

mikekamornikov opened this issue · 2 comments

I have several kminion v2.2.1 deployments and time after time some of them panic:

panic: interface conversion: kmsg.Request is *kmsg.OffsetFetchRequest, not *kgo.pinReq

goroutine 21872 [running]:
github.com/twmb/franz-go/pkg/kgo.(*Client).handleShardedReq.func2({0x0, {0xcf55e8, 0xc00069a0c0}, {0x0, 0x0}})
	/go/pkg/mod/github.com/twmb/franz-go@v1.10.0/pkg/kgo/client.go:1766 +0x974
github.com/twmb/franz-go/pkg/kgo.(*Client).handleShardedReq(0xc0000ce000, {0xcf0750?, 0xc00054ca80}, {0xcf55e8?, 0xc00069a0c0})
	/go/pkg/mod/github.com/twmb/franz-go@v1.10.0/pkg/kgo/client.go:1821 +0x9f9
github.com/twmb/franz-go/pkg/kgo.(*Client).shardedRequest(0xc0000ce000, {0xcf07f8?, 0xc000a9d8c0?}, {0xcf55e8?, 0xc00069a0c0})
	/go/pkg/mod/github.com/twmb/franz-go@v1.10.0/pkg/kgo/client.go:978 +0x691
github.com/twmb/franz-go/pkg/kgo.(*Client).Request(0xc00011f8a0?, {0xcf07f8?, 0xc000a9d8c0?}, {0xcf55e8?, 0xc00069a0c0?})
	/go/pkg/mod/github.com/twmb/franz-go@v1.10.0/pkg/kgo/client.go:778 +0x2d
github.com/twmb/franz-go/pkg/kmsg.(*OffsetFetchRequest).RequestWith(...)
	/go/pkg/mod/github.com/twmb/franz-go/pkg/kmsg@v1.2.0/generated.go:10894
github.com/cloudhut/kminion/v2/minion.(*Service).listConsumerGroupOffsets(0xc0004b6000, {0xcf07f8, 0xc000a9d8c0}, {0xc000041c00, 0x20})
	/app/minion/consumer_group_offsets.go:73 +0xf0
github.com/cloudhut/kminion/v2/minion.(*Service).listConsumerGroupOffsetsBulk.func1.1()
	/app/minion/consumer_group_offsets.go:42 +0x65
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa

I use the following settings on top of kminion defaults:

configMapGenerator:
- name: kafka-minion
  literals:
  - EXPORTER_NAMESPACE=minion
  - LOGGER_LEVEL=error
  - KAFKA_TLS_ENABLED=true
  - KAFKA_TLS_INSECURE_SKIP_TLS_VERIFY=true
    # See https://github.com/redpanda-data/kminion/blob/v2.2.1/docs/reference-config.yaml#L66-L74
  - MINION_CONSUMERGROUPS_SCRAPEMODE=adminApi

I use AWS MSK with Kafka 3.3.1. Any ideas what could it be?

weeco commented

Updated the deps and released v2.2.3