Always tries to use instance profile instead of service account

Question

Always tries to use instance profile instead of service account

hongkongkiwi opened this issue 4 years ago · 2 comments

Hi everyone,

I've restricted access to the instance profile as detailed by amazon here. For reference I'm using the region "us-gov-west-1", that's the only difference, but everything else looks (from my eyes) correct.

But in my logs, I can see that it always tries to look for the instance role and does not try my service role.

{"message":"Missing credentials in config","errno":"ETIMEDOUT","code":"CredentialsError","syscall":"connect","address":"169.254.169.254","port":80,"time":"2020-10-22T16:31:31.521Z","originalError":{"message":"Could not load credentials from ChainableTemporaryCredentials","errno":"ETIMEDOUT","code":"CredentialsError","syscall":"connect","address":"169.254.169.254","port":80,"time":"2020-10-22T16:31:31.521Z","originalError":{"message":"Missing credentials in config","errno":"ETIMEDOUT","code":"CredentialsError","syscall":"connect","address":"169.254.169.254","port":80,"time":"2020-10-22T16:31:31.521Z","originalError":{"message":"Could not load credentials from any providers","errno":"ETIMEDOUT","code":"CredentialsError","syscall":"connect","address":"169.254.169.254","port":80,"time":"2020-10-22T16:31:31.521Z","originalError":{"message":"EC2 Metadata roleName request returned error","errno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect","address":"169.254.169.254","port":80,"time":"2020-10-22T16:31:31.521Z","originalError":{"errno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect","address":"169.254.169.254","port":80,"message":"connect ETIMEDOUT 169.254.169.254:80"}}}}},"stack":"Error: connect ETIMEDOUT 169.254.169.254:80\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1141:16)","type":"Error","msg":"failure while polling the secret myapp/myapp-db-pass"}
{"msg":"updating status for myapp/myapp-db-pass to: ERROR, Missing credentials in config"}

Here are the values I'm setting up with:

env:
  AWS_REGION: us-gov-west-1
  AWS_DEFAULT_REGION: us-gov-west-1
  DISABLE_POLLING: false
  LOG_LEVEL: debug
  ROLE_PERMITTED_ANNOTATION: iam.amazonaws.com/permitted

customResourceManagerDisabled: false

crds:
  create: false

rbac:
  create: true

serviceAccount:
  name: external-secrets
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws-us-gov:iam::<my_account_id>:role/EKSExternalSecretsManager

securityContext:
  fsGroup: 65534
  runAsNonRoot: false

image:
  repository: godaddy/kubernetes-external-secrets
  tag: 6.0.0

Here's my ExternalSecret:

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: my-db-pass
  namespace: mynamespace
spec:
  backendType: secretsManager
  region: us-gov-west-1
  roleArn: "arn:aws-us-gov:iam::<my_account_id>:role/EKSApp"
  data:
    - key: /dev/db
      name: password

Here's the EKSExternalSecretsManager role trust policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws-us-gov:iam::<my_acc_id>:oidc-provider/oidc.eks.us-gov-west-1.amazonaws.com/id/<MY_ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-gov-west-1.amazonaws.com/id/<MY_ID>:sub": "system:serviceaccount:external-secrets:*"
        }
      }
    }
  ]
}

This role has the permissions to assume the EKSApp role, which in turn trusts this role. So that seems setup properly.

Looking inside the external-secrets pod these values are set:

AWS_ROLE_ARN=arn:aws-us-gov:iam::<my_acc_id>:role/EKSExternalSecretsManager
AWS_DEFAULT_REGION=us-gov-west-1
AWS_REGION=us-gov-west-1
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

Here is the decoded token above

{
 "aud": [
  "sts.amazonaws.com"
 ],
 "exp": 1603419185,
 "iat": 1603415434,
 "iss": "https://oidc.eks.us-gov-west-1.amazonaws.com/id/<my_id>",
 "kubernetes.io": {
  "namespace": "external-secrets",
  "pod": {
   "name": "external-secrets-kubernetes-external-secrets-dbc9f877b-zhb6b",
   "uid": "bb5efd79-c066-401f-8b19-7c39390e46f7"
  },
  "serviceaccount": {
   "name": "external-secrets",
   "uid": "f5b8cebe-38b6-4529-8086-c494751a9592"
  }
 },
 "nbf": 1603415434,
 "sub": "system:serviceaccount:external-secrets:external-secrets",
 "jti": "cda27c2d-07fd-468e-80bb-fe4ff797fdfc"
}

When I use the node.js commands from #290 to manually authenticate inside the pod this is the error that seems to show

Error [CredentialsError]: Missing credentials in config
    at Request.extractError (/app/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/app/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/app/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/app/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /app/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'CredentialsError',
  time: 2020-10-23T01:17:13.398Z,
  requestId: '9cce30e2-36bc-4d16-8e09-a39ebe5a8af2',
  statusCode: 400,
  retryable: true,
  originalError: {
    message: 'Could not load credentials from TokenFileWebIdentityCredentials',
    code: 'CredentialsError',
    time: 2020-10-23T01:17:13.398Z,
    requestId: '9cce30e2-36bc-4d16-8e09-a39ebe5a8af2',
    statusCode: 400,
    retryable: true,
    originalError: {
      message: 'Incorrect token audience',
      code: 'InvalidIdentityToken',
      time: 2020-10-23T01:17:13.397Z,
      requestId: '9cce30e2-36bc-4d16-8e09-a39ebe5a8af2',
      statusCode: 400,
      retryable: true
    }
  }
} null

This error seems misleading as I saw in #290 he saw the same thing, but turns out it was something else.

I'm really lost! What else can I do to debug?

I saw a few other posts about this issue saying there was some bugs in the external-secrets code, but since I'm using the latest 6.0.0 release, I would have to assume these are solved. Should I be using a fork?

Answer 1 · 2020-10-27T14:32:50.000Z

I have resolved this. For future referenece of others some Amazon accounts need to use a different STS url. By default the mutating webhook inserts sts.amazonaws.com , in my case for my account I had to use the us-gov sts endpoint. This can be changed with the following annotation in the service account. Here's an example:

serviceAccount:
  name: external-secrets
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws-us-gov:iam::<MY_ACCOUNT_ID>:role/<MY_ROLE_NAME>
    eks.amazonaws.com/audience: sts.us-gov-west-1.amazonaws.com

Answer 2 · 2020-10-27T14:53:18.000Z

Thanks for reporting back! Everything in the initial post really did look correct to me so nice to know the reason :)