The request signature we calculated does not match the signature you provided

[taragurung]

Describe the solution you'd like
ExternalSecret log shows the following error message:

ERROR, The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details

Steps?

1. installed External secret with helm
2. Access to AWS secrets backends given by Setting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars in the kubernetes-external-secrets session/pod. You can use envVarsFromSecret in the helm chart
3. Created a secret manager in AWS secret manager with name stage/email-service

ExternalSecret:

kind: ExternalSecret
metadata:
  name: email-env-external
  namespace: email-service-stage
spec:
  backendType: secretsManager
  region: us-west-2
  dataFrom:
    - stage/email-service

Secret in Secret Manager in AWS

Created a secret plaintext values in AWS SecretManager

{
"APP_DEBUG": "False",
"APP_ENV": "local",
"APP_LOG": "errorlog",
"BROADCAST_DRIVER": "log",
"CACHE_DRIVER": "redis",
"EMAIL_APP_KEY": "base64=4rtfd",
}

output

kubectl get ExternalSecret -n email-service-stage
ERROR, The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details

[moolen]

Hey @taragurung i think you're refering to kubernetes-external-secrets, i moved the issue to the correct repository.

[moolen]

This is usually an issue with defect credentials, please double-check if they are correct.

[taragurung]

@moolen Thanks for the reply, I am using the AWS Secret manager to store my data and using default-encrypion to encrypt it. And using kubernetes-external-secrets to pull the secret. And using IAM role to verify AWS-Backend.

[moolen]

One common problem is, that the aws credentials AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID are mounted via secrets. The secrets hold are base64 encoded values. Base64 encoding the credentials is where most issues appear (echo appends a newline unless -n is given):

$ echo "1234" | base64       
MTIzNAo=
$ echo -n "1234" | base64
MTIzNA==

Please double-check that the credentials you've mounted are correct

[taragurung]

Thanks! @moolen it was a credential mismatching.
How do I pass the custom Role to which has the required access.

When I pass the role as following it does not accept the credentials, passed via ENV.
Where can I pass the roles that will have the access required.

apiVersion: "kubernetes-client.io/v1"
kind: ExternalSecret
metadata:
  name: secret-testing
  namespace: email-service-stage
spec:
  backendType: secretsManager
  roleArn: arn:aws:iam::XXXXXX56:role/testing-remove-it
  region: us-west-2
  data:
    - key: test/secret
      name: newpass

ERROR, User: arn:aws:sts::5xxxxx:assumed-role/AmazonEKSNodes/i-0ff5720e2b41 is not authorized to perform: secretsmanager:GetSecretValue on resource: test/secret

Looks like it takes some default IAM role

[moolen]

It's really hard to tell where exactly the issue is as i don't know the specifics of your setup.
I'd suggest to execute a shell in the controller container and check your environment with the aws cli:

# prerequisite: install aws cli

# check who you are based on the default credential loading chain
$ aws sts get-caller-identity
{
    "UserId": "XYZ:botocore-session-123",
    "Account": "123412341234",
    "Arn": "arn:aws:sts::123412341234:assumed-role/some-role-i-created/botocore-session-123"
}

# check if you are able to assume the specified role
$ aws sts assume-role --role-arn arn:aws:iam::XXXXXX56:role/testing-remove-it --role-session-name example-sess
{
   # [...]
    "Credentials": {
        "SecretAccessKey": "...",
        "SessionToken": "...",
        "AccessKeyId": "..."
    }
}

# export aws environment variables you got from the above call
# now you are role "testing-remove-it" 
$ export AWS_ACCESS_KEY_ID=...
$ export AWS_SECRET_ACCESS_KEY=...
$ export AWS_SESSION_TOKEN=...

# double-check identity
$ aws sts get-caller-identity loader chain
{
    "UserId": "XYZ:example-sess",
    "Account": "XXXXXX56",
    "Arn": "arn:aws:sts::XXXXXX56:assumed-role/testing-remove-it/example-sess"
}
# now make a call to aws secrets manager
$ aws secretsmanager list-secrets
{
    "SecretList": [
        {
            "ARN": "arn:aws:secretsmanager:eu-central-1:XXXXXX56:secret:example-foo",
            "Name": "example-foo",
            # [...]
        }
]

further docs:
https://docs.aws.amazon.com/cli/latest/reference/sts/assume-role.html#examples
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html

Also, i recommend using IRSA with EKS so you don't need to manage/inject keys:
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

[taragurung]

Thanks a lot, @moolen. The ExternalSecret is successful and also created a secret.
A final question to clear my doubts.

1. It has created the secret, I edit the secret and get the values and the data value is still base64 encoded only. Which anyone can decode easily.
2. As It is pulling from the secret manager where I implemented default encryption. Isn't it supposed to be encrypted. Or is it., It was not encrypted on secret-manager.

[moolen]

(1) this is by design! They are supposed to be base64 encoded, not encrypted (docs). I recommend reading this blog article from aquasec for the reasoning behind that. You can implement encryption at rest with etcd (example) but this just changes the way how they are stored on disk, not the Kind=Secret.
(2) SecretsManager does encryption at rest - It gives out the secret in plain text, not ciphertext

I'm gonna close this then, glad i could help 😃