doesn't work with service account role in EKS in AWS China

Question

doesn't work with service account role in EKS in AWS China

yongzhang opened this issue 4 years ago · 14 comments

Hi,

I'm having issues to make it work with service account role in EKS in AWS China, I checked a few similar issues but none of them work for me, the pod is still trying to use node iam role to assumerole, please help, thanks.

my eks version: 1.18.8, in AWS China cn-northwest-1
my deployment:

(${account_id} below is just used to hide my actual account id):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-external-secrets
  namespace: "default"
  labels:
    app.kubernetes.io/name: kubernetes-external-secrets
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kubernetes-external-secrets
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kubernetes-external-secrets
    spec:
      serviceAccountName: kubernetes-external-secrets
      containers:
        - name: kubernetes-external-secrets
          image: godaddy/kubernetes-external-secrets:6.0.0
          ports:
          - name: prometheus
            containerPort: 3001
          imagePullPolicy: IfNotPresent
          resources:
            {}
          env:
          - name: "AWS_DEFAULT_REGION"
            value: "cn-northwest-1"
          - name: "AWS_REGION"
            value: "cn-northwest-1"
          - name: "LOG_LEVEL"
            value: "info"
          - name: "LOG_MESSAGE_KEY"
            value: "msg"
          - name: "METRICS_PORT"
            value: "3001"
          - name: "POLLER_INTERVAL_MILLISECONDS"
            value: "10000"
          - name: "VAULT_ADDR"
            value: "http://127.0.0.1:8200"
          - name: "AWS_ROLE_ARN"
            value: "arn:aws-cn:iam::${account_id}:role/kubernetes-external-secrets"
          - name: "AWS_WEB_IDENTITY_TOKEN_FILE"
            value: "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
          # Params for env vars populated from k8s secrets
      securityContext:
        runAsNonRoot: true
        fsGroup: 65534

my service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubernetes-external-secrets
  namespace: "default"
  labels:
    app.kubernetes.io/name: kubernetes-external-secrets
  annotations:
    eks.amazonaws.com/role-arn: arn:aws-cn:iam::${account_id}:role/kubernetes-external-secrets

my iam role Trust Relationship:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws-cn:iam::${account_id}:oidc-provider/oidc.eks.cn-northwest-1.amazonaws.com.cn/id/xxx"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.cn-northwest-1.amazonaws.com.cn/id/xxxx:sub": "system:serviceaccount:default:kubernetes-external-secrets"
        }
      }
    }
  ]
}

my iam role policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:ListSecretVersionIds",
                "secretsmanager:GetSecretValue",
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "*"
        }
    ]
}

logs:

npm info it worked if it ends with ok
npm info using npm@6.14.6
npm info using node@v12.18.4
npm info lifecycle kubernetes-external-secrets@6.0.0~prestart: kubernetes-external-secrets@6.0.0
npm info lifecycle kubernetes-external-secrets@6.0.0~start: kubernetes-external-secrets@6.0.0

> kubernetes-external-secrets@6.0.0 start /app
> ./bin/daemon.js

{"level":30,"time":1608083428550,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"loading kube specs"}
{"level":30,"time":1608083428632,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"successfully loaded kube specs"}
{"level":30,"time":1608083428633,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"updating CRD"}
{"level":30,"time":1608083428633,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"Upserting custom resource externalsecrets.kubernetes-client.io"}
{"level":30,"time":1608083428695,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"successfully updated CRD"}
{"level":30,"time":1608083428698,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"starting app"}
{"level":20,"time":1608083428698,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"Starting watch stream"}
Wed, 16 Dec 2020 01:50:28 GMT kubernetes-client deprecated .getStream use .getObjectStream, see https://github.com/godaddy/kubernetes-client/blob/master/merging-with-kubernetes.md at lib/external-secret.js:40:10
{"level":30,"time":1608083428701,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"successfully started app"}
{"level":30,"time":1608083428702,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"MetricsServer listening on port 3001"}
{"level":20,"time":1608083428709,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"spinning up poller for default/hello-service"}
{"level":30,"time":1608083428709,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"starting poller for default/hello-service"}
{"level":20,"time":1608083428720,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"next poll for default/hello-service in 0 ms"}
{"level":30,"time":1608083428721,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"running poll on the secret default/hello-service"}
{"level":30,"time":1608083428732,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"fetching secret property hello-service/password with role: pods role in region undefined"}
{"level":50,"time":1608083433272,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","message":"User: arn:aws-cn:sts::my_account_id:assumed-role/eks-main-base-worker/i-03210c3c1fd0b6930 is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws-cn:secretsmanager:cn-northwest-1:my_account_id:secret:hello-service/password-yQqi19","code":"AccessDeniedException","time":"2020-12-16T01:50:33.271Z","requestId":"69161890-77c9-46b0-9e5f-f06c12ef4449","statusCode":400,"retryable":false,"retryDelay":2.8864295764524206,"stack":"AccessDeniedException: User: arn:aws-cn:sts::my_account_id:assumed-role/eks-main-base-worker/i-03210c3c1fd0b6930 is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws-cn:secretsmanager:cn-northwest-1:my_account_id:secret:hello-service/password-yQqi19\n    at Request.extractError (/app/node_modules/aws-sdk/lib/protocol/json.js:51:27)\n    at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n    at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n    at Request.emit (/app/node_modules/aws-sdk/lib/request.js:683:14)\n    at Request.transition (/app/node_modules/aws-sdk/lib/request.js:22:10)\n    at AcceptorStateMachine.runTo (/app/node_modules/aws-sdk/lib/state_machine.js:14:12)\n    at /app/node_modules/aws-sdk/lib/state_machine.js:26:10\n    at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:38:9)\n    at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:685:12)\n    at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:116:18)","type":"Error","msg":"failure while polling the secret default/hello-service"}
{"level":20,"time":1608083433273,"pid":18,"hostname":"kubernetes-external-secrets-5dc6d76dc8-rz85z","msg":"updating status for default/hello-service to: ERROR, User: arn:aws-cn:sts::my_account_id:assumed-role/eks-main-base-worker/i-03210c3c1fd0b6930 is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws-cn:secretsmanager:cn-northwest-1:my_account_id:secret:hello-service/password-yQqi19"}

envs in pod:

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
AWS_ROLE_ARN=arn:aws-cn:iam::my_account_id:role/kubernetes-external-secrets
AWS_REGION=cn-northwest-1
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_DEFAULT_REGION=cn-northwest-1

Answer 1 · 2020-12-23T20:28:57.000Z

any update on this, i am having the same issue it doesnt work with eks OIDC. it works with kube2iam no issues.

Answer 2 · 2020-12-24T03:23:44.000Z

@renperez china specifically or just setup issues using IAM roles for service accounts? If the latter please explore various previous issues to try and identify any configuration issues.

@yongzhang Thanks for all the details, I can't see anything that looks out of place. I'm not sure if this is specifically a CN region issue or something else. The only thing that came to mind was #523 but your logs doesn't seem to match.
The logs you provide make it look like an IAM permission issue, eg the IAM role is assumed but the roles permissions are invalid? Unless its not assuming IAM roles for service account at all (judging by name of arn:aws-cn:sts::my_account_id:assumed-role/eks-main-base-worker , is this the node/instance role?)

so updating your service account annotations to:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubernetes-external-secrets
  namespace: "default"
  labels:
    app.kubernetes.io/name: kubernetes-external-secrets
  annotations:
    eks.amazonaws.com/role-arn: arn:aws-cn:iam::${account_id}:role/kubernetes-external-secrets
    eks.amazonaws.com/audience: sts.cn-northwest-1.amazonaws.com.cn

Might help, but I'm just guessing here :)

Answer 3 · 2020-12-24T03:39:42.000Z

@renperez china specifically or just setup issues using IAM roles for service accounts? If the latter please explore various previous issues to try and identify any configuration issues.

@yongzhang Thanks for all the details, I can't see anything that looks out of place. I'm not sure if this is specifically a CN region issue or something else. The only thing that came to mind was #523 but your logs doesn't seem to match.
The logs you provide make it look like an IAM permission issue, eg the IAM role is assumed but the roles permissions are invalid? Unless its not assuming IAM roles for service account at all (judging by name of arn:aws-cn:sts::my_account_id:assumed-role/eks-main-base-worker , is this the node/instance role?)

so updating your service account annotations to:
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubernetes-external-secrets
  namespace: "default"
  labels:
    app.kubernetes.io/name: kubernetes-external-secrets
  annotations:
    eks.amazonaws.com/role-arn: arn:aws-cn:iam::${account_id}:role/kubernetes-external-secrets
    eks.amazonaws.com/audience: sts.cn-northwest-1.amazonaws.com.cn
Might help, but I'm just guessing here :)

Well, this looks relevant, I'll try.

Answer 4 · 2020-12-24T04:52:09.000Z

it still doesn't work by adding eks.amazonaws.com/audience: sts.cn-northwest-1.amazonaws.com.cn, I have to use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY instead.

Answer 5 · 2020-12-27T17:50:35.000Z

@Flydiverny this is a set up using service account with IAM role. The role that I am using is configured the way @yongzhang configured his with trust relationship. is external secret app using aws sdk? like i mentioned in my first comment, this works with using kube2iam, but we are trying to move away from kube2iam. we are now using service account with iam role for authenticating to aws resources.

Answer 6 · 2020-12-27T17:51:15.000Z

@yongzhang how did you deploy using aws access key and id?

Answer 7 · 2020-12-28T04:43:17.000Z

@yongzhang how did you deploy using aws access key and id?

I use terraform, so simply to create an IAM user with policies to access ssm, and create a kubernetes secret for aws access key id and secret, and finally let external secret pod retrieve envs from secret, this is a bit complex, I think if you can use kube2iam then use it.

Answer 8 · 2020-12-28T17:47:57.000Z

@yongzhang yeah im better off using kube2iam for the mean time.

Answer 9 · 2021-01-06T13:52:41.000Z

I have the same issue and I got this error:

{"message":"Missing credentials in config","code":"CredentialsError

I want to use credentials, please, any workaround on this

Answer 10 · 2021-01-11T16:39:11.000Z

I fixed with:
securityContext:
  fsGroup: 1000
the to the fact the node process is running with uid and gid 1000 inside the container

Originally posted by @c4m4 in #597 (comment)

Please try the same here if this resolves the issue!

Answer 11 · 2021-01-15T00:36:50.000Z

@Flydiverny that worked flawlessly! thank you!!!

Answer 12 · 2021-02-25T04:55:33.000Z

@Flydiverny that worked flawlessly! thank you!!!

Are you using China accounts? Did you add eks.amazonaws.com/audience: sts.cn-northwest-1.amazonaws.com.cn to service account?

Answer 13 · 2021-05-27T03:28:43.000Z

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

Answer 14 · 2021-06-27T02:15:55.000Z

This issue was closed because it has been stalled for 30 days with no activity.