Intermittent CrashLoopBackOffs
Closed this issue · 8 comments
Every 2 hours or so, the webhook container becomes unhealthy and crashes with the following logs.
{"error":"json: cannot unmarshal object into Go value of type []unifi.DNSRecord","level":"error","msg":"error getting records","requestMethod":"GET","requestPath":"/records","time":"2024-05-25T04:35:18Z"}
After some time in the CrashLoopBackoff state, it fixes itself and redeploys in a healthy state, until the same crash happens 2 hours later.
@kashalls has brought up that cookie expires every 2 hours which definitely could be causing this, but it seems somewhat intermittent. My latest deployment was at 1:33PM and I didn't get my first alert until 6:38PM. After that first alert, it has happened every 2 hours
I believe that the token refresh is not happening in the background, we would probably need to call the Login method after the cookie expires (which lasts for 2 hours from the time of iat).
I can confirm that I'm getting this after roughly two hours, then it clears up and happens again some time later. I cleared the alert notifications, I'll try to note the times next time it happens.
I added more debugging lines in #21
We will get some more info if it happens again.
Can confirm that the webhook starts erroring out at 2 hours, then the external-dns container starts getting 500s from the web hook and crashloops.
This can be closed