1.24 breaking change - No Secret by default for Service Account token

[connordwoods]

Hello contributors,

I have used this Patcher in my environment to automate the propagation of Docker credentials across the cluster for existing & new namespaces. However, in Kubernetes v1.24, a change to the Token APIs may break some functionality of this module. I have tried to describe the scenario below. Related issue

Description of Issue

Kubernetes is moving to short-lived Tokens as a security improvement. A change to the TokenRequest API in Kubernetes v1.24 requires new Service Accounts to manually create a Secret which contains its API access token. Similarly, when a new Pod is created, kubelet will use the TokenRequest API to generate a token that is specific to that Pod (and Service Account/Namespace?). The token is mounted to the Pod by a projected volume & updated regularly before expiration.

Replicate the Issue

1. Create a cluster running Kubernetes version 1.24+
2. Verify the default service account of the default namespace has no associated Tokens or Secrets.
   $ kubectl describe sa default
3. Deploy the imagepullsecrets-patcher module to the default namespace.
4. Create a Pod in the default namespace which will attempt to pull a remote image.
5. The Pod fails to pull the image because it cannot access the credentials Secret.

Proposed Change

The Patcher should also create a "Service Account Token Secret" for each Service Account included in its scope, if one does not already exist. This means that new Service Accounts do not need to have this Token Secret manually created. The Secret containing registry credentials is then accessible by the Service Account and its Pods. Relevant documentation

Disclaimer: I am relatively new to Kubernetes and am not a Go developer. I hope I have presented the issue clearly. Please let me know if more information is needed or an alternative solution exists. Thank you!

[jselleck-sans]

I've tested on 1.25 and this does not seem to be an issue. Granted there are no tokens, the patcher is still able to sync the secret and patch the sa accounts with

  imagePullSecrets:
  - name: image-pull-secret  

I was able to pull an image from a private registry using the appropriate secret and no token in the sa account.

I do not believe patcher should be updated to create the tokens. This seems out of scope. Perhaps I'm missing something.

[keithharvey]

We started our deployment on a k3s cluster hosted by Civo, where the manifest initially worked as expected. At some point, though, the deployment failed with the following error: MountVolume.SetUp failed for volume "cdk8s-plugin-cm" : configmap "cdk8s-plugin-cm" not found.

After some digging, we discovered that explicitly specifying the service account token resolved the issue, contrary to the current behavior that relies on Kubernetes' auto-mounting of service account tokens. This seems to align with the original poster's experience on Kubernetes 1.25. Here is the modified YAML for the Deployment spec:

# ... (other parts of the Deployment)
spec:
  # ...
  template:
    spec:
      automount_service_account_token: false
      volumes:
      - name: explicit-token
        projected:
          sources:
          - secret:
              name: imagepullsecret-patcher-token
              items:
              - key: token
                path: token
      containers:
      - name: imagepullsecret-patcher
        # ...
        volumeMounts:
        - name: explicit-token
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount/
          readOnly: true

This approach avoids the issue for all providers.

To see the full changes, you can view the pull request here: [PR]

Related stack overflow: https://stackoverflow.com/questions/69038012/mountvolume-setup-failed-for-volume-kube-api-access-fcz9j-object-default