aws/amazon-vpc-cni-k8s

ImageInspectError when eks try to pull the image

itai-spiritt opened this issue · 10 comments

itai-spiritt commented

What happened:

Running in AWS EKS (version 1.21) we have started to receive the following error:

Failed to inspect image "602401143452.dkr.ecr.eu-west-1.amazonaws.com/amazon-k8s-cni-init:v1.7.5-eksbuild.1": Id or size of image "602401143452.dkr.ecr.eu-west-1.amazonaws.com/amazon-k8s-cni-init:v1.7.5-eksbuild.1" is not set

It happened suddenly, for some of our nodes (also for new nodes).

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14", GitCommit:"0f77da5bd4809927e15d1658fb4aa8f13ad890a5", GitTreeState:"clean", BuildDate:"2022-06-15T14:17:29Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.14-eks-3af4770", GitCommit:"3c86b95399a496380e90327b114db108546c6345", GitTreeState:"clean", BuildDate:"2024-04-30T03:32:40Z", GoVersion:"go1.21.8", Compiler:"gc", Platform:"linux/amd64"}
n2taylor commented

Check that your node AMI was not deprecated

  • 👍2
  • 🎉3
  • ❤️1
orsenthil commented

And v1.7.5-eksbuild.1" is very old, it is time to upgrade to latest supported version of CNI.

itai-spiritt commented

when using eks, is it sufficient to just update the cluster?

itai-spiritt commented

Check that your node AMI was not deprecated

ami-0c71ddf0753e561aa amazon-eks-node-1.18-v20201126

On 2 different nodes, the ready one is docker 20 and on the not ready the docker is 25
both have the same AMI
What causes the docker version diff? And how do I revert it back to 20?

n2taylor commented

Check that your node AMI was not deprecated

ami-0c71ddf0753e561aa amazon-eks-node-1.18-v20201126

I'm not able to find that AMI in the list of publicly searchable images in the AWS console. I would recommend getting on the latest 1.21 compatible AMI which is ami-0c9aa39ce6bc59190 amazon-eks-node-1.21-v20240615

On 2 different nodes, the ready one is docker 20 and on the not ready the docker is 25 both have the same AMI What causes the docker version diff? And how do I revert it back to 20?

Is it possible you have multiple node AMI versions running in your cluster?

itai-spiritt commented

Check that your node AMI was not deprecated

ami-0c71ddf0753e561aa amazon-eks-node-1.18-v20201126

I'm not able to find that AMI in the list of publicly searchable images in the AWS console. I would recommend getting on the latest 1.21 compatible AMI which is ami-0c9aa39ce6bc59190 amazon-eks-node-1.21-v20240615

On 2 different nodes, the ready one is docker 20 and on the not ready the docker is 25 both have the same AMI What causes the docker version diff? And how do I revert it back to 20?

Is it possible you have multiple node AMI versions running in your cluster?

no, it's the same AMI version...
I'll try update it to version 1.21, but where should I change it to be applied on the whole cluster?

n2taylor commented

I'll try update it to version 1.21, but where should I change it to be applied on the whole cluster?

You'll have to update the node group either via eksctl or modifying your launch template. See here.

orsenthil commented

This is not a VPC CNI related bug. Closing this issue.

github-actions commented

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.

frankietoledo commented

Check that your node AMI was not deprecated

+1 here, we got problems with cni and kube proxy, solved upgrading the ami.