mattjmcnaughton/personal-k8s

Create single reserved instance node for critical pods

Closed this issue · 2 comments

mattjmcnaughton commented

Issue
To save money, all of my k8s nodes are spot instances. However, even if I set my maxPrice equal to the on-demand price, there are still windows in which I don't get spot instances. I have critical pods (i.e. online services like my blog) running on these spot instances. In addition, my monitoring/alerting nodes (i.e. prometheus and alertmanager) als run on these spot instances. So I don't even receive alerts when the spot instances are not available.

Impl

  • Purchase a single "reserved instance" node.
  • Create a new kops instance group which will not use spot instances and will use the reserved instance.
  • Use affinities/tolerations to schedule critical pods to this node.

Definition of Done

  • Critical pods are running on a non-spot instance.
  • File ticket to consider if should use spot instance => schedule for 1 month from now.
  • Blog post.
RmMsr commented

Would you mind, posting a link to the blog post? Your issue ranks interestingly high on searches for dedicating k8s nodes to critical workloads. Thanks