chrnola/redis-streams-exporter

Crash loop timeout

Opened this issue · 2 comments

Just to inform that I did a rollback from latest to 0.6 due this intermittent error causing crash loopback error.

{
    "@t": "2022-12-27T18:08:32.2127523Z",
    "@mt": "Unhandled exception, application terminating",
    "@l": "Fatal",
    "@x": "System.AggregateException: One or more errors occurred. (Timeout awaiting response (outbound=0KiB, inbound=95549KiB, 5768ms elapsed, timeout is 5000ms), command=XINFO, next: XINFO, inst: 0, qu: 0, qs: 2, aw: False, rs: TryParseResult, ws: Idle, in: 0, in-pipe: 17484279, out-pipe: 0, serverEndpoint: redacted:6379, mc: 1/1/0, mgr: 9 of 10 available, clientName: redis-stream-exporter-845d4749bc-l42dn, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=1,Free=32766,Min=8,Max=32767), v: 2.2.88.56325 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts))\n ---> StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=95549KiB, 5768ms elapsed, timeout is 5000ms), command=XINFO, next: XINFO, inst: 0, qu: 0, qs: 2, aw: False, rs: TryParseResult, ws: Idle, in: 0, in-pipe: 17484279, out-pipe: 0, serverEndpoint: redacted:6379, mc: 1/1/0, mgr: 9 of 10 available, clientName: redis-stream-exporter-845d4749bc-l42dn, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=1,Free=32766,Min=8,Max=32767), v: 2.2.88.56325 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)\n   --- End of inner exception stack trace ---\n   at Microsoft.FSharp.Control.AsyncResult`1.Commit() in D:\\a\\_work\\1\\s\\src\\fsharp\\FSharp.Core\\async.fs:line 391\n   at Microsoft.FSharp.Control.AsyncPrimitives.QueueAsyncAndWaitForResultSynchronously[a](CancellationToken token, FSharpAsync`1 computation, FSharpOption`1 timeout) in D:\\a\\_work\\1\\s\\src\\fsharp\\FSharp.Core\\async.fs:line 1044\n   at Microsoft.FSharp.Control.AsyncPrimitives.RunSynchronously[T](CancellationToken cancellationToken, FSharpAsync`1 computation, FSharpOption`1 timeout) in D:\\a\\_work\\1\\s\\src\\fsharp\\FSharp.Core\\async.fs:line 1070\n   at Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T](FSharpAsync`1 computation, FSharpOption`1 timeout, FSharpOption`1 cancellationToken) in D:\\a\\_work\\1\\s\\src\\fsharp\\FSharp.Core\\async.fs:line 1365\n   at RedisStreamsMonitor.Main.main(String[] _argv) in /App/Program.fs:line 87"
}

Originally posted by @danielrosa-zh in #7 (comment)

@danielrosa-zh I might be interpreting this error message incorrectly, but do you expect the XINFO command to return 95549KiB worth of the data for the stream that you're monitoring?

Also what version are you referring to by "latest"? 0.6 is the latest published release.

Sorry for my late response, my bad about latest, we are using 0.6 as a latest. Looks like now we are trying to retrieve almost 100 MB. not sure why, will ask developers, thanks.
Is there a parameter to increase timeout ? because the pod is in crash loop.

Timeout awaiting response (outbound=0KiB, inbound=106481KiB, 5060ms elapsed, timeout is 5000ms)