External-secrets pod assumes node's role instead of custom

[CodingJonas]

The external-secrets pod assumes the node's role instead of the custom role to access secrets:

{"level":30,"message_time":"2021-11-10T14:13:17.100Z","pid":18,"hostname":"external-secrets-helm-kubernetes-external-secrets-86f4c56c55n54","msg":"fetching secret property test with role: pods role in region: eu-central-1"}
{"level":50,"message_time":"2021-11-10T14:13:23.132Z","pid":18,"hostname":"external-secrets-helm-kubernetes-external-secrets-86f4c56c55n54","payload":{"message":"User: arn:aws:sts::111111111111:assumed-role/site-k20210915... is not authorized to perform: secretsmanager:GetSecretValue on resource: site-k/test because no identity-based policy allows the secretsmanager:GetSecretValue action","code":"AccessDeniedException","time":"2021-11-10T14:13:23.131Z","requestId":"16ff2638-59bc-4fe0-8e73-82324f60be4b","statusCode":400,"retryable":false,"retryDelay":41.93099749592386},"msg":"failure while polling the secret external-secrets/hello-service"}
{"level":20,"message_time":"2021-11-10T14:13:23.132Z","pid":18,"hostname":"external-secrets-helm-kubernetes-external-secrets-86f4c56c55n54","msg":"updating status for external-secrets/hello-service to: ERROR, User: arn:aws:sts::111111111111:assumed-role/site-k20210915... is not authorized to perform: secretsmanager:GetSecretValue on resource: test because no identity-based policy allows the secretsmanager:GetSecretValue action"}

I looked through old issues and found that fs-system needs to be set to 65534. I install the external-secrets setup via helm and include this configuration. Here is my value.yaml:

env:
  AWS_REGION: eu-central-1
  LOG_LEVEL: debug
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/SecretsManagerAccess
securityContext:
  fsGroup: 65534

I think I set the role correctly up, following these instructions. It seems to work as EKS injects the token, with these env variables:

AWS_ROLE_ARN: arn:aws:iam::111111111111:role/SecretsManagerAccess
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

What am I missing? I created a simple external-secret:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: hello-service
  namespace: default
spec:
  backendType: secretsManager
  region: eu-central-1
  data:
    - key: test
      name: password

I receive the error message as stated above.

To mention some context, I run this on an EKS cluster with Kubernetes version 1.21 and use the helm chart version 8.3.2. Also, when I add the secretmanager access policy to the node's role, the secret creation works. But I don't want to use that IAM setup.

Am I missing something? Or is this a bug?

[spirosoik]

@CodingJonas could you with us the policy you have created for your role?

[CodingJonas]

Thank you for taking some time for my issue! My policy is this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": [
                "arn:aws:secretsmanager:eu-central-1:111111111111:secret:*"
            ]
        }
    ]
}

(The account id is the correct one everywhere, I just anonymized it)

[ewestover-aumni]

i had this problem and it was because the trust relationship associated with the IAM role was not referencing the correct serviceaccount name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:<CORRECT_NAMESPACE_HERE>:<CORRECT_SA_NAME_HERE>"
        }
      }
    }
  ]
}

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity.