okigan/awscurl

Doesn't recognize AWS_WEB_IDENTITY_TOKEN_FILE, botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

rfvermut opened this issue · 15 comments

rfvermut commented

In AWS EKS environment with IRSA. In version 0.22.

AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

awscurl -X PUT https://example.com
Traceback (most recent call last):
  File "/usr/local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 504, in main
    inner_main(sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 469, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 393, in load_aws_config
    cred = session.get_credentials()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 441, in get_credentials
    self._credentials = self._components.get_component(
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 937, in get_component
    self._components[name] = factory()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 151, in _create_credential_resolver
    return botocore.credentials.create_credential_resolver(
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 64, in create_credential_resolver
    metadata_timeout = session.get_config_variable('metadata_service_timeout')
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 251, in get_config_variable
    return self.get_component('config_store').get_config_variable(
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 313, in get_config_variable
    return provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 410, in provide
    value = provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 471, in provide
    scoped_config = self._session.get_scoped_config()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 351, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

Works fine in 0.21

okigan commented

Thanks for reporting - I’ll have a look

iainelder commented

@rfvermut How are your other CLI environment variables set?

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html

What sections do you have in your CLI config file and credentials file?

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

rfvermut commented
iainelder commented

I'm unfamiliar with AWS EKS. Does IRSA refer to the IAM roles for service accounts feature?

I don't think the behavior is affecterd by AWS_WEB_IDENTITY_TOKEN_FILE so I'll ignore it for now.

The server at example.com doesn't appear to respond to PUT requests so lets change that to a GET request.

awscurl -X GET https://example.com

So here's what I think is happening.

The behavior occurs only when botocore is available to awscurl.

Without knowing more about your environment, I will be using pipx to install awscurl in virtualenv and use the awslibs optional dependency to ensure that botocore is available.

pipx install awscurl[awslibs]==0.22.0

With the example command, awscurl via botocore loads the "default" profile (because that's the default value set by the CLI parser The default value is not None as I had previously thought!).

Since you have no configuration, botocore fails to find that profile and so fails.

But what happens in the previous version?

pipx uninstall awscurl
pipx install awscurl[awslibs]==0.21.0

With the example command, on my local machine, I get a different error, but it's for the same reason; there are no credentials to be found.

$ awscurl -X PUT https://example.com
Traceback (most recent call last):
  File "/home/norm/.local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 500, in main
    inner_main(sys.argv[1:])
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 466, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 392, in load_aws_config
    access_key, secret_key, security_token = cred.access_key, cred.secret_key, cred.token
AttributeError: 'NoneType' object has no attribute 'access_key'

If I execute it on an EC2 instance with an attached IAM role, then awscurl via botocore loads the instance credentials from the instance metadata service.

@okigan I looks like this behavior was broken by my PR #116 for AWS SSO profiles.

I had believed that the default value for profile was None, but actually it's hard-coded to "default". That changes how botocore's default credential resolution works. It stops it falling back to the instance metadata servce.

I think there is a way to support all the desired credentials sources, but it will take time to find a correct solution.

For now I think the right thing to do is to undo my changes and not include them until we find a way to test all the use cases.

iainelder commented

@rfvermut I'm not aware of a way to workaround this with version 0.22.0. For now I suggest you specify 0.21.0 when you install the command.

okigan commented

I'm unfamiliar with AWS EKS. Does IRSA refer to the IAM roles for service accounts feature?

I don't think the behavior is affecterd by AWS_WEB_IDENTITY_TOKEN_FILE so I'll ignore it for now.

The server at example.com doesn't appear to respond to PUT requests so lets change that to a GET request.

awscurl -X GET https://example.com

So here's what I think is happening.

The behavior occurs only when botocore is available to awscurl.

Without knowing more about your environment, I will be using pipx to install awscurl in virtualenv and use the awslibs optional dependency to ensure that botocore is available.

pipx install awscurl[awslibs]==0.22.0

With the example command, awscurl via botocore loads the "default" profile (because that's the default value set by the CLI parser The default value is not None as I had previously thought!).

Since you have no configuration, botocore fails to find that profile and so fails.

But what happens in the previous version?

pipx uninstall awscurl
pipx install awscurl[awslibs]==0.21.0

With the example command, on my local machine, I get a different error, but it's for the same reason; there are no credentials to be found.

$ awscurl -X PUT https://example.com
Traceback (most recent call last):
  File "/home/norm/.local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 500, in main
    inner_main(sys.argv[1:])
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 466, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 392, in load_aws_config
    access_key, secret_key, security_token = cred.access_key, cred.secret_key, cred.token
AttributeError: 'NoneType' object has no attribute 'access_key'

If I execute it on an EC2 instance with an attached IAM role, then awscurl via botocore loads the instance credentials from the instance metadata service.

@okigan I looks like this behavior was broken by my PR #116 for AWS SSO profiles.

I had believed that the default value for profile was None, but actually it's hard-coded to "default". That changes how botocore's default credential resolution works. It stops it falling back to the instance metadata servce.

I think there is a way to support all the desired credentials sources, but it will take time to find a correct solution.

For now I think the right thing to do is to undo my changes and not include them until we find a way to test all the use cases.

@iainelder i will be backing out the SSO feature/commit and releasing bug fix build - that should buy us time to figure out what’s going on here.

okigan commented

Change reverted: #124, new release will follow shortly

okigan commented

@rfvermut I am looking for a way to setup a test environment for this, is the following script sufficient to repo? and in which environment does that need to run?

docker run  -it python:3-buster /bin/bash -c "pip3 install awscli awscurl && awscurl -X GET https://example.com"
rfvermut commented

First, let me thank you reverting that change so fast. I have a gazillion non-centralized Jenkins scripts that refer to non-pinned install of awscurl and it affected half of our organization. My mistake, always pin your versions.

Second, the "just enough" setup for tests would look like this:

Get inside a container

docker run -ti --rm python:3-buster /bin/bash

Prepare environment

export AWS_ROLE_ARN="arn:aws:iam::123456789012:role/marketingadminrole"
export AWS_WEB_IDENTITY_TOKEN_FILE="/tmp/test"
echo "garbage" > $AWS_WEB_IDENTITY_TOKEN_FILE
pip3 install awscli awscurl==0.23 (or 22)

0.23/0.21 will fail trying to decode garbage while doing AssumeRoleWithWebIdentity which means it understands identity tokens

root@369514c4e8e8:/# awscurl example.com
Traceback (most recent call last):
  File "/usr/local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 504, in main
    inner_main(sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 469, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 394, in load_aws_config
    access_key, secret_key, security_token = cred.access_key, cred.secret_key, cred.token
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 421, in access_key
    self._refresh()
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 513, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 529, in _protected_refresh
    metadata = self._refresh_using()
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 670, in fetch_credentials
    return self._get_cached_credentials()
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 680, in _get_cached_credentials
    response = self._get_credentials()
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 890, in _get_credentials
    return client.assume_role_with_web_identity(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidIdentityTokenException: An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: The ID Token provided is not a valid JWT. (You may see this error if you sent an Access Token)

0.22 will die as usual.

root@369514c4e8e8:/# awscurl example.com
Traceback (most recent call last):
  File "/usr/local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 504, in main
    inner_main(sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 469, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 393, in load_aws_config
    cred = session.get_credentials()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 441, in get_credentials
    self._credentials = self._components.get_component(
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 937, in get_component
    self._components[name] = factory()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 151, in _create_credential_resolver
    return botocore.credentials.create_credential_resolver(
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 64, in create_credential_resolver
    metadata_timeout = session.get_config_variable('metadata_service_timeout')
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 251, in get_config_variable
    return self.get_component('config_store').get_config_variable(
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 313, in get_config_variable
    return provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 410, in provide
    value = provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 471, in provide
    scoped_config = self._session.get_scoped_config()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 351, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found
okigan commented

@rfvermut thanks for repro script!
@iainelder I have an idea, why profile is not working: in the command line parser it is the only one (among AWS_* related variables) that has a default value (of 'default'), so I think removing that and adding (back) the Session(profile=profile) could make it work. Here is related change: a501a43

iainelder commented

Commit a501a43 looks to me like it could work.

(It's what I was alluding to in my ending comments in #123 (comment))

How would you test it?

okigan commented

@iainelder i've tested it by checking out and running a501a43.
Getting the InvalidIdentityTokenException error as shown in #122 (comment) using following script:

export AWS_ROLE_ARN="arn:aws:iam::123456789012:role/marketingadminrole"
export AWS_WEB_IDENTITY_TOKEN_FILE="/tmp/test"
echo "garbage" > $AWS_WEB_IDENTITY_TOKEN_FILE
python -m awscurl

i am still looking for a way to test load_aws_config, but do not see a way to mock the relevant component.

iainelder commented

@okigan have you looked at moto? it has mocks for many AWS services including STS.

We might be able to use it to mock responses from the STS service that we are authenticating against.

https://github.com/spulec/moto

I have some ideas here, but no time to implement them for a while yet :-)

NeckBeardPrince commented

This is still an issue.

okigan commented