Failed to retrieve instance data from ec2 metadata

[brizaldi]

What happened:
We're currently using CIS Benchmark with Amazon Linux 2 for our EKS cluster. But when we installed the ebs-csi-driver addon we got this error on the ebs-csi-node pod:

I0513 08:41:02.467945       1 ec2.go:40] "Retrieving EC2 instance identity metadata" regionFromSession=""
I0513 08:41:02.468167       1 metadata.go:52] "failed to retrieve instance data from ec2 metadata; retrieving instance data from kubernetes api" err="could not get EC2 instance identity metadata: operation error ec2imds: GetInstanceIdentityDocument, request canceled, context deadline exceeded"
E0513 08:41:32.470244       1 main.go:154] "Could not determine region from any metadata service. The region can be manually supplied via the AWS_REGION environment variable." err="error getting instance data from ec2 metadata or kubernetes api"
panic: error getting instance data from ec2 metadata or kubernetes api

What you expected to happen:
We suspect that it might come from the custom AMI, since there's no error when were using the default AMI.

How to reproduce it (as minimally and precisely as possible):

1. Create a custom AMI using CIS Benchmark Level 1;Kernel 5.10 with Amazon Linux 2.
2. Create a nodegroup using that custom AMI, since we're using terraform here's what our config look like:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "20.8.5"

  eks_managed_node_groups = {
    cis = {
      cluster_version = "1.29"
      desired_size    = 1
      max_size        = 10
      min_size        = 1

      ami_id                     = "ami-xxx" # Use custom CIS AMI
      enable_bootstrap_user_data = true
      iam_role_additional_policies = {
        AmazonEBSCSIDriverPolicy = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
        AmazonEFSCSIDriverPolicy = "arn:aws:iam::aws:policy/service-role/AmazonEFSCSIDriverPolicy"
      }
      pre_bootstrap_user_data    = <<EOT
        # Allow inbound traffic for kubelet (so kubectl logs/exec works)
        iptables -I INPUT -p tcp -m tcp --dport 10250 -j ACCEPT

        # Ensure IPv4 loopback traffic is configured (Automated)
        iptables -A INPUT -i lo -j ACCEPT
        iptables -A OUTPUT -o lo -j ACCEPT
        iptables -A INPUT -s 127.0.0.0/8 -j DROP

        # Ensure IPv4 outbound and established connections are configured (Manual)
        iptables -A OUTPUT -p tcp -m state --state NEW,ESTABLISHED -j ACCEPT
        iptables -A OUTPUT -p udp -m state --state NEW,ESTABLISHED -j ACCEPT
        iptables -A OUTPUT -p icmp -m state --state NEW,ESTABLISHED -j ACCEPT
        iptables -A INPUT -p tcp -m state --state ESTABLISHED -j ACCEPT
        iptables -A INPUT -p udp -m state --state ESTABLISHED -j ACCEPT
        iptables -A INPUT -p icmp -m state --state ESTABLISHED -j ACCEPT

        # Allow inbound traffic for kubelet on ipv6 if needed (so kubectl logs/exec works)
        ip6tables -A INPUT -p tcp --destination-port 10250 -j ACCEPT

        # Ensure IPv6 loopback traffic is configured (Automated)
        ip6tables -A INPUT -i lo -j ACCEPT
        ip6tables -A OUTPUT -o lo -j ACCEPT
        ip6tables -A INPUT -s ::1 -j DROP

        # Ensure IPv6 outbound and established connections are configured (Manual)
        ip6tables -A OUTPUT -p tcp -m state --state NEW,ESTABLISHED -j ACCEPT
        ip6tables -A OUTPUT -p udp -m state --state NEW,ESTABLISHED -j ACCEPT
        ip6tables -A OUTPUT -p icmp -m state --state NEW,ESTABLISHED -j ACCEPT
        ip6tables -A INPUT -p tcp -m state --state ESTABLISHED -j ACCEPT
        ip6tables -A INPUT -p udp -m state --state ESTABLISHED -j ACCEPT
        ip6tables -A INPUT -p icmp -m state --state ESTABLISHED -j ACCEPT
      EOT
    }
  }
}

3. Install ebs-csi-driver addon.
4. Check the status/logs of ebs-csi-node pod.

Anything else we need to know?:

Environment:

• AWS Region: ap-southeast-1
• Instance Type(s): Amazon Linux 2
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.7
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.29
• AMI Version:
• Kernel (e.g. uname -a):
• Release information (run cat /etc/eks/release on a node):

[cartermckinnon]

This definitely sounds related to your custom AMI and/or those iptables rules; can you open a ticket with AWS support? We try to keep this repo focused on the AMI templates and the official EKS AMI builds.