After Upgrade to 1.12.6 to 1.13.4 eks cluster node failed to join nodegroup
junnhong opened this issue · 2 comments
What happened:
After upgrading to v1.13.4 from v1.12.6, node is failing to join nodegroup without ENI Config. Our pipeline worked fine on 1.12 while we create ENI Config after eks cluster provision is done, but after upgrading it to v1.13, it block us from creating eks cluster. I think v1.13 requires ENI Config to be created when node is joining. Is this expected behavior of v1.13?
{"level":"error","ts":"2023-11-27T22:54:08.373Z","caller":"ipamd/ipamd.go:889","msg":"error while retrieving eniconfig: ENIConfig.crd.k8s.amazonaws.com \"us-east-1c\" not found"}, {"level":"error","ts":"2023-11-27T22:54:08.373Z","caller":"aws-k8s-agent/main.go:28","msg":"Initialization failure: Failed to attach any ENIs for custom networking"}
Attach logs
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version): 1.24 - CNI Version v.1.13.4
- OS (e.g:
cat /etc/os-release):
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30" - Kernel (e.g.
uname -a): Linux ip-10-0-34-73.ec2.internal 5.10.192-183.736.amzn2.x86_64 #1 SMP Wed Sep 6 21:15:41 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
@junnhong yes, this is expected behavior. In v1.13.0+, the node will not be marked "Ready" until pod IPs are available. So when Custom Networking is configured, a valid ENIConfig must be present, otherwise the node will remain "Not Ready" until an ENI is attached (added in #2354). This is done to prevent the scheduler from scheduling pods on this node until it has IPs.
I am not sure how your pipeline is configured, but you will want to create the ENIConfig resources first, yes
⚠️ COMMENT VISIBILITY WARNING⚠️
Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.