Setting WARM_IP_TARGET=0 or WARM_ENI_TARGET=0 breaks the addon
CowKeyMan opened this issue · 5 comments
What happened:
I am on EKS.
When I set WARM_IP_TARGET=0 and WARM_ENI_TARGET=0, VPC-CNI addon fails to add more of these 2 items when more pods are added.
Attach logs
I will update this issue when i have the logs again.
What you expected to happen:
As I add more pods, I would like that VPC-CNI adds more secondary IP addresses dynamically so that my pods may get IP addresses attached to them. However, I will have many (many) instances, and the amount of pods is roughly equal to the number of IP addresses I can allocate. Therefore, I would like to NOT have any warm instances, which is why I want to set WARM_IP_TARGET and WARM_ENI_TARGET to 0, so I do not waste any IP addresses. However, as said before, when WARM_IP_TARGET and WARM_ENI_TARGET are 0, the addon fails to add any more IP addresses, as well as ENI targets (the ENI targets are important because I then also want to use security groups for pods).
How to reproduce it (as minimally and precisely as possible):
- Create an empty cluster with the VPC-CNI.
- Set the WARM_IP_TARGET to 0.
- Add a single node to the cluster
- Put a couple of different pods on it
- See how they fail to be added
If you want to see the ENI targets failing to be added as well, set WARM_ENI_TARGET to 0, then add some pods with different security groups, and see how they fail to be added as well.
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version): 1.27 - CNI Version: v1.13.2-eksbuild.1
- OS (e.g:
cat /etc/os-release): N/A - Kernel (e.g.
uname -a): N/A
@CowKeyMan when WARM_IP_TARGET is set, WARM_ENI_TARGET is ignored. WARM_IP_TARGET=0 is a special mode where no new ENIs will ever be allocated. It is meant to be used in conjunction with MINIMUM_IP_TARGET to allocate all of the IPs up front, and then nothing more.
That is one option, but the more common option that I see customers using is setting WARM_IP_TARGET=1 in conjunction with MINIMUM_IP_TARGET and MAX_ENI.
When only WARM_ENI_TARGET or WARM_IP_TARGET is set to 0, the goal is to allocate IPs "on-demand", i.e. as CNI ADD requests come in, but that is not something that IPAMD is currently capable of. It is on our road-map, though.
Thanks for the answer. I feel you just put A LOT of useful information here which does not seem to be (at least to me) well documented. Maybe the documentation on AWS could be changed to include this information, because this tripped me up for a couple of days.
Continuing on this thread to make sure I understand: am I correct in saying then that when WARM_IP_TARGET=1 (or more than 1), then VPC-CNI will keep allocating IPs as the pods keep coming into the node, always having WARM_IP_TARGET extra IP addresses allocated?
WARM IP settings overriding WARM ENI settings is documented in https://github.com/aws/amazon-vpc-cni-k8s#warm_ip_target , but https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/eni-and-ip-target.md definitely needs some TLC.
And yes, your understanding is correct, though the reason you want to set MINIMUM_IP_TARGET in addition to a low WARM_IP_TARGET value is to reduce the number of EC2 API calls, which that 2nd link discusses a bit
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days
Issue closed due to inactivity.