updater leads to high CPU usage
Closed this issue · 1 comments
Short description
We use OPA release v0.60.0 as a library in https://github.com/zalando/skipper. For a bit of context, we start OPA plugin manager in https://github.com/zalando/skipper/blob/master/filters/openpolicyagent/openpolicyagent.go#L392.
Over time CPU usage look like this:
I guess that 10m is the download interval of policies from s3. I don't really know right now where the time.Duration is set.
This download should have ETag and If-modified-since behavior to not recompile the rule set. The last update of the file was end of August 2023, so there is really nothing to do.
This leads to hpa scale out to maxReplicas and the cluster has only <200 RPS, so basically it should be fine to run minReplicas.
I was able to get this CPU profile from an instance that spiked in CPU usage:
The function activateBundles() shows up in the callstack.
I am looking into the code, but I thought it might makes sense to ask if you understand from the CPU profile what happens.
Steps To Reproduce
I don't know for sure.
Expected behavior
no high CPU spike
As far as I understand we found the issue on our side.