robusta-dev/robusta

Node memory pressure - pod eviction

Opened this issue · 2 comments

prom3theu5 commented

Hi

Currently evaluating some workloads running on a low tier AKS cluster of 3 b2 machines
I've setup robusta minimal without prometheus purely because I need to capture issues with evictions while I evaluate which tier of machine we need for our workloads

Currently a couple of deployments get evicted and rescheduled on another node because of node: low memory
However, these aren't pod oom killed issues, its the node memory threshold I guess which is the reason for the eviction

I expected robusta to alera me when this happened, like it does for pod OOM issues, however it doesn't appear to - i had 50 instances of a pod evicted, and no alerts

Is there not a default playbook setup for this?
do I have to setup a custom playbook, or is what I want not possible with robusta?

thanks

github-actions commented

Hi 👋, thanks for opening an issue! Please note, it may take some time for us to respond, but we'll get back to you as soon as we can!

  • 💬 Slack Community: Join Robusta team and other contributors on Slack here.
  • 📖 Docs: Find our documentation here.
  • 🎥 YouTube Channel: Watch our videos here.
aantn commented

Hey, you'll have to setup a custom playbook for this. It's not captured by default due to the potential for a lot of noise on auto-scaled clusters.

I believe there is a Kubernetes event (kubectl get events) that you can use for the custom playbook trigger - can you confirm?