Logs are missing during high loads
beth2016 opened this issue · 2 comments
beth2016 commented
What happened?
Description
When we run performance tests (40k requests/min) , we see logs are intermittently dropped for 30-90 seconds. This issue started manifesting after we upgrade otel from 0.43 to 0.81.
Steps to Reproduce
Expected Result
Actual Result
Chart version
0.81
Environment information
Environment
Cloud: Openshift
k8s version: 1.23
Chart configuration
No response
Log output
No response
Additional context
No response
matthewmodestino commented
Hey Beth! Hope all is well!
This could be caused by file rotation under high load on the node, or other scenarios.
Any logs from the agents affected that may help?
atoulme commented
You might be best served by opening a support case to follow up. Please post the support case ID in this issue so we can work with you directly.