amazon-ebs: fatal error: An error occurred (404) when calling the HeadObject operation: Key "1.30/2022-06-06/bin/linux/amd64/kubelet" does not exist
eravindar12 opened this issue · 5 comments
What happened:
I am attempting to use the AL2023 base AMI for EKS optimized with the AMI Release v20240531. However, I am encountering a "fatal error: An error occurred (404) when calling the HeadObject operation: Key "1.30/2022-06-06/bin/linux/amd64/kubelet" does not exist" This appears to be a bug.
Do you have any suggestions for a workaround to fix this issue?
Error details:
2024-06-06T19:54:42Z: amazon-ebs: Downloading binaries from: s3://amazon-eks
2024-06-06T19:54:43Z: amazon-ebs: AWS cli present - using it to copy binaries from s3.
2024-06-06T19:54:44Z: amazon-ebs: fatal error: An error occurred (404) when calling the HeadObject operation: Key "1.30/2022-06-06/bin/linux/amd64/kubelet" does not exist
2024-06-06T19:54:44Z: ==> amazon-ebs: Provisioning step had errors: Running the cleanup provisioner, if present...
2024-06-06T19:54:44Z: ==> amazon-ebs: Terminating the source AWS instance...
2024-06-06T19:55:44Z: ==> amazon-ebs: Cleaning up any extra volumes...
2024-06-06T19:55:45Z: ==> amazon-ebs: No volumes to clean up, skipping
2024-06-06T19:55:45Z: ==> amazon-ebs: Deleting temporary security group...
2024-06-06T19:55:45Z: ==> amazon-ebs: Deleting temporary keypair...
2024-06-06T19:55:45Z: Build 'amazon-ebs' errored after 2 minutes 2 seconds: Script exited with non-zero exit status: 1. Allowed exit codes are: [0]
==> Wait completed after 2 minutes 2 seconds
==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Script exited with non-zero exit status: 1. Allowed exit codes are: [0]
==> Builds finished but no artifacts were created.
make[1]: *** [Makefile:95: k8s] Error 1
templates/install-worker.sh
################################################################################
### Kubernetes #################################################################
################################################################################
sudo mkdir -p /etc/kubernetes/manifests
sudo mkdir -p /var/lib/kubernetes
sudo mkdir -p /var/lib/kubelet
sudo mkdir -p /opt/cni/bin
echo "Downloading binaries from: s3://$BINARY_BUCKET_NAME"
S3_DOMAIN="amazonaws.com"
if [ "$BINARY_BUCKET_REGION" = "cn-north-1" ] || [ "$BINARY_BUCKET_REGION" = "cn-northwest-1" ]; then
S3_DOMAIN="amazonaws.com.cn"
elif [ "$BINARY_BUCKET_REGION" = "us-iso-east-1" ] || [ "$BINARY_BUCKET_REGION" = "us-iso-west-1" ]; then
S3_DOMAIN="c2s.ic.gov"
elif [ "$BINARY_BUCKET_REGION" = "us-isob-east-1" ]; then
S3_DOMAIN="sc2s.sgov.gov"
fi
S3_URL_BASE="https://$BINARY_BUCKET_NAME.s3.$BINARY_BUCKET_REGION.$S3_DOMAIN/$KUBERNETES_VERSION/$KUBERNETES_BUILD_DATE/bin/linux/$ARCH"
S3_PATH="s3://$BINARY_BUCKET_NAME/$KUBERNETES_VERSION/$KUBERNETES_BUILD_DATE/bin/linux/$ARCH"
BINARIES=(
kubelet
)
for binary in ${BINARIES[*]}; do
if [[ -n "$AWS_ACCESS_KEY_ID" ]]; then
echo "AWS cli present - using it to copy binaries from s3."
aws s3 cp --region $BINARY_BUCKET_REGION $S3_PATH/$binary .
aws s3 cp --region $BINARY_BUCKET_REGION $S3_PATH/$binary.sha256 .
else
echo "AWS cli missing - using wget to fetch binaries from s3. Note: This won't work for private bucket."
sudo wget $S3_URL_BASE/$binary
sudo wget $S3_URL_BASE/$binary.sha256
fi
sudo sha256sum -c $binary.sha256
sudo chmod +x $binary
sudo mv $binary /usr/bin/
done
sudo rm ./*.sha256
template.json
{
"type": "shell",
"remote_folder": "{{ user `remote_folder`}}",
"script": "{{template_dir}}/provisioners/install-worker.sh",
"environment_vars": [
"AWS_ACCESS_KEY_ID={{user `aws_access_key_id`}}",
"AWS_SECRET_ACCESS_KEY={{user `aws_secret_access_key`}}",
"AWS_SESSION_TOKEN={{user `aws_session_token`}}",
"BINARY_BUCKET_NAME={{user `binary_bucket_name`}}",
"BINARY_BUCKET_REGION={{user `binary_bucket_region`}}",
"CONTAINERD_VERSION={{user `containerd_version`}}",
"KUBERNETES_BUILD_DATE={{user `kubernetes_build_date`}}",
"KUBERNETES_VERSION={{user `kubernetes_version`}}",
"RUNC_VERSION={{user `runc_version`}}",
"SSM_AGENT_VERSION={{user `ssm_agent_version`}}",
"WORKING_DIR={{user `working_dir`}}"
],"execute_command": "{{.Vars}} bash '{{.Path}}'"
},
Environment: EKS
- AWS Region: us-east-1
- Instance Type(s): EC2
- EKS Platform version: EKS 1.30
- Kubernetes version : 1.30
- AMI Version: CIS Amazon Linux 2023 Benchmark - Level 1
What make command are you running?
@cartermckinnon JFYR, I am running the make command as shown below.
make k8s=1.30 os_distro=al2023 aws_region="us-east-1" source_ami_id="ami-xxxx" source_ami_owners=xxxx source_ami_filter_name="CIS Amazon Linux 2023 Benchmark - Level 1 - v05 -prod-xxxxx" kubernetes_build_date="2022-06-06" kubernetes_version="1.30" subnet_id="subnet-xxxx" vpc_id="vpc-xxxx"
This is the problem:
kubernetes_build_date="2022-06-06"
There are no 1.30 binaries from 2022 😄 you can find the latest ones for 1.30 using this script: https://github.com/awslabs/amazon-eks-ami/blob/main/hack/latest-binaries.sh
@cartermckinnon Thank you for correcting the build date; now it's able to download. However, I'm encountering a "permission denied" error.
Do you have any suggestions on why I'm getting this permission denied?
amazon-ebs: /tmp/script_5266.sh: line 225: /tmp/worker/shared/bin/imds: Permission denied
2024-06-06T23:13:27Z: amazon-ebs: Downloading binaries from: s3://amazon-eks
2024-06-06T23:13:27Z: amazon-ebs: AWS cli present - using it to copy binaries from s3.
2024-06-06T23:13:30Z: amazon-ebs: download: s3://amazon-eks/1.30.0/2024-05-12/bin/linux/amd64/kubelet to ./kubelet
2024-06-06T23:13:31Z: amazon-ebs: download: s3://amazon-eks/1.30.0/2024-05-12/bin/linux/amd64/kubelet.sha256 to ./kubelet.sha256
2024-06-06T23:13:31Z: amazon-ebs: kubelet: OK
2024-06-06T23:13:32Z: amazon-ebs: AWS cli present - using it to copy ecr-credential-provider from s3.
2024-06-06T23:13:33Z: amazon-ebs: download: s3://amazon-eks/1.30.0/2024-05-12/bin/linux/amd64/ecr-credential-provider to ./ecr-credential-provider
2024-06-06T23:13:34Z: amazon-ebs: amazon-ssm-agent.x86_64 3.3.380.0-1.amzn2023 @System
2024-06-06T23:13:34Z: amazon-ebs: amazon-ssm-agent already present - skipping install
2024-06-06T23:13:34Z: amazon-ebs: /tmp/script_5266.sh: line 225: /tmp/worker/shared/bin/imds: Permission denied
2024-06-06T23:13:34Z: ==> amazon-ebs: Provisioning step had errors: Running the cleanup provisioner, if present...
2024-06-06T23:13:34Z: ==> amazon-ebs: Terminating the source AWS instance...
2024-06-06T23:15:05Z: ==> amazon-ebs: Cleaning up any extra volumes...
2024-06-06T23:15:05Z: ==> amazon-ebs: No volumes to clean up, skipping
2024-06-06T23:15:05Z: ==> amazon-ebs: Deleting temporary security group...
2024-06-06T23:15:05Z: ==> amazon-ebs: Deleting temporary keypair...
2024-06-06T23:15:06Z: Build 'amazon-ebs' errored after 2 minutes 43 seconds: Script exited with non-zero exit status: 126. Allowed exit codes are: [0]
It's because you're using a CIS base image, same problem as here: #1838 (comment)