Failed to assign IP to the Container.
riteshsonawane1372 opened this issue · 9 comments
What happened:
Failed to assign IP to the Container. Something similar to #1791
Attach logs
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version): - CNI Version
- OS (e.g:
cat /etc/os-release): - Kernel (e.g.
uname -a):
@riteshsonawane1372 there is not enough information here to do any sort of debugging. Did you check the node logs to see why IP allocation failed?
Currently I don't have the logs.
@riteshsonawane1372 there is nothing for us to debug here then. If you reproduce this and can provide the logs and more information, then we have something to work from.
⚠️ COMMENT VISIBILITY WARNING⚠️
Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.
We have this errors aswell
its happens when we scaling deployments from 0 replicas to higher
logs:
[/var/log/messages] E0211 07:30:45.142216 4510 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"env-mgmt-b5c5fbdb5-kfdr5_dev01(d944aa2c-25b3-4a76-802d-bc8423355184)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"env-mgmt-b5c5fbdb5-kfdr5_dev01(d944aa2c-25b3-4a76-802d-bc8423355184)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"d571d03cae4532a4ca5a7f8c01f10da4b9b589c38db841ec56de809fc408745d\\\": plugin type=\\\"aws-cni\\\" name=\\\"aws-cni\\\" failed (add): add cmd: failed to assign an IP address to container\"" pod="dev01/env-mgmt-b5c5fbdb5-kfdr5" podUID="d944aa2c-25b3-4a76-802d-bc8423355184"
[/var/log/messages] E0211 07:30:45.142154 4510 kuberuntime_manager.go:1177] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"d571d03cae4532a4ca5a7f8c01f10da4b9b589c38db841ec56de809fc408745d\": plugin type=\"aws-cni\" name=\"aws-cni\" failed (add): add cmd: failed to assign an IP address to container" pod="dev01/env-mgmt-b5c5fbdb5-kfdr5"
[event-exporter] Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f37fe951acc2cd4e9fff69880941b0b187aef64593ff8fd93d138efb3b81a1c7": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
K8s EKS version: v1.29.0-eks-5e0fdde
AWS VPC CNI version: v1.14.1-eksbuild.1
@kfirfer Simply add a Secondary CIDR range. After that, the problem will be solved. You also need to create a new ENI-Config for ec2 to receive Ip's from Secondary CIDR range.
@riteshsonawane1372 theres documentation about it ?
we using terraform for lifting the EKS and some node pools, and karpenter for more EC2 nodes
@riteshsonawane1372 you meant here?
@kfirfer Yes, Add a new CIDR range. After that Create a new Subnet with that same CIDR range, also note that the A.Z of the new subnet should be the same as the previous subnet. Create these resources
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
name: $az_1
spec:
securityGroups:
- $cluster_security_group_id
subnet: $new_subnet_id_1
After that Run Below cmds
kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true
kubectl set env daemonset aws-node -n kube-system ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone
Now you need to restart the Node so that It can chose the new CIDR range IP's.