Consistent RedisTimeoutException after Servers topology change
VasilyIlin opened this issue · 1 comments
We have redis cluster in Kubernetes with 6 instances.
At some point of time the list of endpoints changed from [10.240.2.111:6379, 10.240.3.77:6379, 10.240.1.112:6379, 10.240.2.5:6379, 10.240.1.149:6379, 10.240.3.149:6379] to [10.240.3.77:6379, 10.240.2.5:6379, 10.240.1.112:6379, 10.240.1.149:6379, 10.240.2.165:6379, 10.240.3.149:6379], so the are no 10.240.2.111 anymore.
We recreated ConnectionMultiplexer on each instance of out service and It returned correct list of endpoints.
But one of our service pod kept throw following exception of every Redis call:
StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 30420ms elapsed, timeout is 30000ms), command=XINFO, next: XREADGROUP platform:*, inst: 0, qu: 0, qs: 6, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 0, async-ops: 62871, serverEndpoint: 10.240.2.111:6379, conn-sec: 1058.03, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: *(SE.Redis-v2.6.122.38350), PerfCounterHelperkeyHashSlot: 5669, IOCP: (Busy=0,Free=1000,Min=50,Max=1000), WORKER: (Busy=1,Free=32766,Min=100,Max=32767), POOL: (Threads=6,QueuedItems=0,CompletedItems=293278,Timers=50), v: 2.6.122.38350 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
the value in the exceptions are always the same(same conn-sec, same qs)
After we restarted the pod, the error was gone.
Client spec: Asp.Net Core, Net8, Ubuntu
Sorry, found the problem in our code