Unmatched iptables rule

[bwells]

I've been unable to get kiam working on a new kops managed Kubernetes 1.19 cluster. As best as I've been able to track things down so far the iptable tables rules being generated aren't matching traffic AWS credentials traffic.

The generated iptables rules:

> iptables -L -v -t nat 

Chain PREROUTING (policy ACCEPT 16 packets, 1178 bytes)
 pkts bytes target     prot opt in     out     source               destination
  183 13680 KUBE-SERVICES  all  --  any    any     anywhere             anywhere             /* kubernetes service portals */
    0     0 DNAT       tcp  --  ens5   any     anywhere             169.254.169.254      tcp dpt:http to:10.100.13.131:8181
<snip>

I see zero pkts and bytes values despite AWS calls such as aws sts get-caller-identity or curl http://169.254.169.254/latest/meta-data/iam/info from within a test pod or directly on a worker VM. I've seen no activity in the agent logs other than error free bootup messages and pings. All auth error messages refer to the role assigned to the node instance.

The host IP and interface values look correct for the VM.

Am I heading down the right path? What would normal log activity be for successfully proxied traffic?

Thank you in advance for any help you're able to provide.

Agent Spec:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-system
  name: kiam-agent
spec:
  selector:
    matchLabels:
      app: kiam
      role: agent
  updateStrategy:
    type: OnDelete
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9620"
      labels:
        app: kiam
        role: agent
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      nodeSelector:
        kubernetes.io/role: node
      volumes:
        - name: ssl-certs
          hostPath:
            # for AWS linux or RHEL distros
            # path: /etc/pki/ca-trust/extracted/pem/
            # debian or ubuntu distros
            path: /etc/ssl/certs
            # path: /usr/share/ca-certificates
        - name: tls
          secret:
            secretName: kiam-agent-tls
        - name: xtables
          hostPath:
            path: /run/xtables.lock
            type: FileOrCreate
      containers:
        - name: kiam
          securityContext:
            capabilities:
              add: ["NET_ADMIN"]
          image: quay.io/uswitch/kiam:master # USE A TAGGED RELEASE IN PRODUCTION
          imagePullPolicy: Always
          command:
            - /kiam
          args:
            - agent
            - --iptables
            - --host-interface=ens5
            - --json-log
            - --level=debug
            - --port=8181
            - --cert=/etc/kiam/tls/agent.pem
            - --key=/etc/kiam/tls/agent-key.pem
            - --ca=/etc/kiam/tls/ca.pem
            - --server-address=kiam-server:443
            - --prometheus-listen-addr=0.0.0.0:9620
            - --prometheus-sync-interval=5s
            - --gateway-timeout-creation=1s
          env:
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          volumeMounts:
            - mountPath: /etc/ssl/certs
              name: ssl-certs
            - mountPath: /etc/kiam/tls
              name: tls
            - mountPath: /var/run/xtables.lock
              name: xtables
          livenessProbe:
            httpGet:
              path: /ping
              port: 8181
            initialDelaySeconds: 3
            periodSeconds: 3

Annotated debugging pod

apiVersion: v1
kind: Pod
metadata:
  annotations:
    iam.amazonaws.com/role: external-dns
  name: awscli
  labels:
    app: awscli
spec:
  containers:
  - image: amazon/aws-cli
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: awscli
  restartPolicy: Always

default namespace is annotated as such:

metadata:
  annotations:
    iam.amazonaws.com/permitted: .*

[droctothorpe]

@bwells Have you made any progress on this? Running into a similar issue.