Using vpc-cni version v1.15.1-eksbuild.1 with cluster version 1.28 removes irsa annotation from service account aws-node
samispurs opened this issue · 8 comments
Environment:
- Kubernetes version (use
kubectl version): v1.29.0 - CNI Version v1.15.1-eksbuild.1
- OS (e.g:
cat /etc/os-release): amazon linux - Kernel (e.g.
uname -a):
@samispurs upgrading should not affect the service account. How did you upgrade?
Here is a doc on using IRSA with the VPC CNI: https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html. Please provide more information to keep this issue open
Hi, We are using terraform to deploy clusters. We had been trying to build a new cluster with 1.28. The same TF code works for deploying a 1.26/1.27 cluster, but as soon as our client tried to deploy 1.28, we noticed that the service account annotation was not being updated like it was for the other two versions.
As a workaround we had to create a null_resource and local-exec that executes the command kubectl annotate serviceaccount -n kube-system aws-node eks.amazonaws.com/role-arn= right after the cluster is created. The vpc-cni driver then uses the correct irsa role with the correct permissions to enable the nodes to join the cluster.
The setup for the vpc-cni driver is supposed to use the IRSA role, and we don't have the vpc-cni iam policy attached to the node role, therefore nodes are unable to join the cluster when not using IRSA. These are the errors we were seeing.
"Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
- Unauthorized operation: failed to call ec2:DescribeNetworkInterfaces due to missing permissions.
Hope this provides a bit more clarity. Thanks!
PS. If we specify a addon version, it seems that v1.15.1-eksbuild.1 is used regardless. Only when the TF code executes for managing the addon does the version get applied.
@samispurs this problem feels unrelated to the VPC CNI repository. Annotating the aws-node Service Account with the role ARN that you create is the responsibility of the cluster manager, which is Terraform in your case. Do you know why Terraform is not annotating the Service Account properly?
Note that the Service Account should be updated before the VPC CNI pods are deployed, otherwise the pods would have to be restarted to pick up the credentials.
@jdn5126 Thanks! I really don't know why the aws-node SA is not being annotated with a specific version of the cluster being deployed. I'll reach out to the module maintainers for the cluster manager. Cheers!
I have been looking through https://github.com/terraform-aws-modules/terraform-aws-eks/issues and I haven't found any similar issues reported yet
Hi @jdn5126
Is this EKS platform eks.6 or eks.7?
@TechnoTaff Are you asking the customer what EKS control plane platform version they are using? Is there some link there that you are aware of?
This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.