[v2] credentials supplied by aws sso login do not conform to AWS standards
dayer4b opened this issue · 33 comments
AWS documents that credentials generated by aws configure are stored in the standard path ~/.aws/credentials:
The AWS CLI stores the credentials that you specify with aws configure in a local file named credentials, in a folder named .aws in your home directory.
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html
This path is not utilized by credentials stored by aws configure sso or aws sso login.
Other tools (like Terraform and Boto3) that rely on this standard do not function because they do not see the credentials stored in the ~/.aws/cli/cache/ JSON files.
Shared credential file (~/.aws/credentials)
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
For my own purposes, I am working around this issue by extracting the data from the JSON object and dumping it into the ~/.aws/credentials file as such:
#!/bin/bash
JSON_BASEPATH="${HOME}/.aws/cli/cache"
AWS_CREDENTIALS_PATH="${HOME}/.aws/credentials"
if [ -f ${AWS_CREDENTIALS_PATH} ]; then
echo "backing up existing credentials"
cp -rf ${AWS_CREDENTIALS_PATH} "${AWS_CREDENTIALS_PATH}-"$(date +"%s")
fi
# find the latest CLI JSON file
json_file=$(ls -tr "${JSON_BASEPATH}" | tail -n1)
# use jq to dump stuff in the right place
aws_access_key_id=$(cat ${JSON_BASEPATH}/${json_file} | jq -r '.Credentials.AccessKeyId')
aws_secret_access_key=$(cat ${JSON_BASEPATH}/${json_file} | jq -r '.Credentials.SecretAccessKey')
aws_session_token=$(cat ${JSON_BASEPATH}/${json_file} | jq -r '.Credentials.SessionToken')
echo "[default]" > ${AWS_CREDENTIALS_PATH}
echo "aws_access_key_id = ${aws_access_key_id}" >> ${AWS_CREDENTIALS_PATH}
echo "aws_secret_access_key = ${aws_secret_access_key}" >> ${AWS_CREDENTIALS_PATH}
echo "aws_session_token = ${aws_session_token}" >> ${AWS_CREDENTIALS_PATH}
I hope this helps others until a fix is ready!
Hi @dayer4b ,
To make sure I understand the problem here,
you've ran the configure sso command and logged in and are able to use the various services with the generated temporary credentials right?
aws sso login --profile my-sso-profile
aws s3 ls --profile my-sso-profile
this kind of call works for you, but when running through another sdk, it is not recognizing the credentials. Am I understanding correctly?
@KaibaLopez that is the current best solutions I think we agree. Those STS creds are valid only for an hour so you have to generate those quite often.
This is not a good solutions. SSO does not always create new cache json files so you actually need to clear cache folder beforehand to be sure that the newest file is created by AWS cli.
I think I'm missing something here, but this is not really a problem on the CLI, unless it is not working for you for cli actions either it'd be a problem with how you're getting credentials on the other sdks. You're right that you shouldn't need to move them manually, but each sdk has it's own way of retrieving credentials when it comes to sso logins, make sure that you're doing it correctly, it might need extra configuration.
Also, not sure if this was part of the problem but sso is meant to be temporary so not sure I can offer you a solution there.
Again let me know if I missed something here or you don't agree with my response, I do feel like I'm not getting the full picture...
I think you are correct in away. I think that it would be nice that cli would provide supported way for SSO STS creds like you can get those from the portal. Now this "hook" is way too complex and error brone.
Support could be very similar to STS lookup with MFA code.
I think the problem OP is describing is really a feature request for the aws terraform provider. I found a feature request over on the provider repo related to supporting SSO credentials: hashicorp/terraform-provider-aws#10851
Doing this on a utility-by-utility or sdk-by-sdk basis will make for an absolutely horrid developer and user experience.
The SDKs all support the AWS Shared Config file in some way. The SSO auth/cred process should also integrate through the AWS Shared Config so all SDKs and all utilities are able to leverage it the same way.
@lorengordon
Yes,and we are trying to unify the sdk's behaviors, but I think you misunderstood what I meant there, it's not as bad as that.
It's more on the default credentials provider chain, I'm not familiar with how terraform does theirs, but they should have a way to read the credentials from the same place as the cli, it just may not be the default, that's what I meant by extra steps, not that you'd need to modify the code but that you'd need a different method call.
All this being said, I'd recommend checking those sdk's documentation or opening an issue on their repos regarding this, it still sounds to me like there is no problem with the CLI's implementation.
@KaibaLopez Ok, gotcha. I use terraform a lot and have also helped review/expand the credential handling to better work with the features of the AWS Shared Config. It will indeed leverage the SDK credential chain properly. The only known real issue currently is that IF you are using the AWS_PROFILE env and that profile uses certain features from the AWS Shared Config file, then you must export AWS_SDK_LOAD_CONFIG=1. That's kind of a thing in the aws-sdk-go for enabling resolution for the AWS Shared Config.
Far as I can tell, I don't think there is truly a terraform issue here.
I would have to say I am experiencing the same "issue" as the OP.
I am using boto3 and it is expecting the AWS CLI V1 shared credentials file in INI format. https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
As for a feature request I think it would be helpful to be able to choose if you also want the temporary credentials stored in a "legacy" format.
I am currently seeing if I can make a wrapper script similar to the OP that exports the access key, secret key, and session token.
I have created and shared a script to simplify updating ~/.aws/credentials for AWS SSO users. It will update the AWS credentials file by adding/updating the specified profile credentials using the AWS CLI v2 cached SSO login.
I pulled the SSO credential support from botocore v2 and the CLI into a credential_process helper, so by adding one line to your SSO-configured profile in .aws/config, it'll work with any SDK that supports credential_process (or any tool that uses one of those SDKs, like Terraform): aws-sso-credential-process.
@flyinprogrammer has written a similar thing in go that's a little less full-featured, but has the advantage of being a single executable: aws-sso-fetcher
Well, come here for some quick ready made solution. Ended up writing one, yawsso -- which I extend @sgtoj work, without depending on anything except CLI v2 itself, to sync up CLI v2 SSO login session to v1 credentials file.
pip install yawsso
aws configure sso
aws sso login --profile=dev
yawsso -p dev
Sorry I didn't reply to these messages sooner, but I think most of the commenters here are understanding that pain that I have described. My concern with the proposed solution is that it effectively breaks the document contract that the AWS CLI has established and fostered for years.
This is particularly problematic with boto3 as this is literally managed by Amazon. I pointed to boto3 and terraform because these are very commonly used and one of them is managed by Amazon.
In the meantime, I have entirely given up any attempt at using aws configure from the command line, instead I have been manually copying credentials from the SSO web page and placing them somewhere that command line tools understand.
@victorskl I look forward to trying out your tool!
Sorry I didn't reply to these messages sooner, but I think most of the commenters here are understanding that pain that I have described. My concern with the proposed solution is that it effectively breaks the document contract that the AWS CLI has established and fostered for years.
This is particularly problematic with
boto3as this is literally managed by Amazon. I pointed toboto3andterraformbecause these are very commonly used and one of them is managed by Amazon.In the meantime, I have entirely given up any attempt at using
aws configurefrom the command line, instead I have been manually copying credentials from the SSO web page and placing them somewhere that command line tools understand.@victorskl I look forward to trying out your tool!
I've moved (back) to aws-vault https://github.com/99designs/aws-vault/blob/master/USAGE.md#aws-single-sign-on-aws-sso
it covers all my needs, and is far more secure than aws cli leaving creds in clear text on disk
Note for aws-vault you need to get a v6 pre-release binary to use the SSO functionality. This is what has allowed us to roll out SSO to our org's backend developers
For anyone looking for a node version of this type of script I've published this one to npm https://github.com/ryansonshine/aws-sso-creds-helper
For folks still needing backward compatibility to ~/.aws/credentials file, I have created a script to automate the web flow of 'aws sso login', so you do not need to switch to the browser for SSO authentication, update the ~/.aws/credentials and inject env params AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN into current terminal.
Please note that @arvindkgs's tool requires you to put your password in an environment variable or in your shell history, both of which are run against recommended practice. The process of using your browser to sign in is a security feature of AWS SSO login, not a bug. It also only appears to work with Okta, and I see no support for MFA.
@benkehoe I built this for developers who don't want to shift focus from their cmd line. While this is a feature of SSO it is distracting for me. Yes, it is only for OKTA now. I added to environment because some tools require creds in the environment. I will add a switch to make this optional. I have explored other tools like https://github.com/Noovolari/leapp and https://github.com/linaro-its/aws2-wrap and found them lacking.
Any traction on this? I would love to be able to give my developers an sso user to authenticate against AWS while using the sdk rather than have to have them manually go and download the credentials from the login screen each time.
@dazhanghalo what's wrong with aws-vault?
aws-vault is a band-aid to the real problem.
The json generated via aws-cli/botocore using a sha1 for its json-names (based on account-id,role/set-permission,start-url) is not great - I'm in favor of calling things for what they are - this currently feels like security by obfuscation and really buys us nothing but a headache. I am hoping this gets addressed soon, as well - the file names should really reflect the profile-name and be ini formatted making them both predictable and usable for other apps that already utilize the ini standard.
Hi all, I appreciate your continued conversation on this, and I'd like to clarify and update.
It seems like the biggest pain point underlying this issue is that all AWS SDKs did not support using the credential provider that is configured via aws configure sso. As of now, all AWS SDKs except C++ support the credentials from SSO login.
@dazhanghalo - what language are your developers using? Hopefully they are covered now by my above statement.
To clarify a bit of @dayer4b's concerns about the contract or standard set by the AWS SDK credential configuration:
The AWS CLI stores the credentials that you specify with
aws configurein a local file named credentials, in a folder named.awsin your home directory.
This is still true - if you supply credentials to aws configure they are stored in the file (directly setting your access id, secret key, and token). However, aws configure sso does not store credentials, only configuration for the SSO portal that you will use. So the statement still holds, although I can agree that the distinction is nuanced. Is there clarification in the documentation that could make this more clear?
There are authentication methods besides SSO that also do not store them in the credentials file, and should not because of their transient nature. Assume role credentials are not written to the credential file, nor are those that are pulled from IMDS, or a custom credential provider. Writing them to the credential file would not work as the file is currently read once when a session is created (for example in Python), so if the credentials were refreshed after expiring they would be broken.
@olenm - I'm not sure I understand the issue here. There is no intention to use SHA1 as a security method, only as a way to ensure that the resulting name of the file is compatible with the storage (e.g., not having a filename that might be too long when combining things like you mentioned).
One thing I hear from this is the need to still get out the current set of credentials that a profile would be using, regardless if they come from SSO, an assume role, or even are configured in the credential file itself. This open issue (#5261) proposes that specifically for SSO in response to the lack of support for SSO, but it seems to have further value to those who need the credentials in a predictable format for other uses
Many of you have written tools to do this (@benkehoe, @flyinprogrammer, @ryansonshine, @victorskl, and probably others). For users who already have the AWS CLI installed to perform the SSO login, it would be a win to also use the same tool to interrogate what credentials are in use and export them if needed. This looks like feature that the AWS CLI team can support and review for implementing.
Thanks @kdaily !
Re the sha1 as a file-name - I am glad to hear it is not an attempt at a security measure; however I think that file name length as a consideration in terms of it potentially reflecting the profile-name should be reconsidered as it is already reaching some 40 characters. Anybody with so many profiles that utilize that many combinations on a single system is in trouble in another way if you ask me. Though personally even with long account+role name combinations in my own profile list, I can easily see 60+ chars - but really the problem comes if people are using paragraphs to identify profiles, if that is the case, I do feel sorry for when they have to use aws --profile myLongProfileNameThatIs200charactersLooooooooooong
The fact that people need to utilize other apps/wrappers/scripts to simply correlate a profile to a credential set is a problem (myself included).
Getting a printout of credentials from the CLI would be very helpful to a degree, but programs like Terraform which already utilize reading the ini file format and utilize profiles could easily benefit from a profile-named-ini - especially since the cache folder path does not change. Even still, I think there are some oddities in the behavior of the creation of the json - as in, it gets created not on login, but after the first call afterward (I personally call aws --profile bla sts get-caller-identity for the file to create, then generate the my ini) - and upon further use, notice that the json gets removed far before it expires, if multiple start-urls are logged into at once. So I can only ask for some predicable file locations based on profile-name alone, and that the files do not get removed pre-expiration.
Please note that @arvindkgs's tool requires you to put your password in an environment variable or in your shell history, both of which are run against recommended practice. The process of using your browser to sign in is a security feature of AWS SSO login, not a bug. It also only appears to work with Okta, and I see no support for MFA.
I have made modifications to my script that no longer needs credentials as environment variables and have been using it for 4 months now without issues.
- I have abandoned trying to automate the web login using selenium, rather now you would need to run
aws sso loginbefore running the script. - The script sets credentials from ~/.aws/cli/cache into ~/.aws/credentials and also sets environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN in your shell
FWIW, I created a small Python 3 wrapper (no external libraries required) around aws sso login that also updates .aws/credentials: https://gist.github.com/robvanderleek/7dd54a53ce72e835ac1f319ef2631c5e
@robvanderleek thanks so much! It is contributions like this that make me think that this feature is still broken. That there are commonly required and readily available solutions to fix the implementation suggests that the implementation is faulty, or perhaps doesn't go far enough in bridging towards backwards compatibility.
Please, please fix this. Makes transitioning to aws sso extremely painful. That command needs to update ~/.aws/credentials.
Thanks again for the input. I'm going to close this issue, as we've outlined above the reasons why temporary credentials (including those from SSO) will not (and should not) be saved in the credentials file.
Nearly all AWS SDKs now automatically support this configuration, so other tools to get this functionality should no longer be required. Downstream tools (like Terraform) also support credential_process, as suggested by @benkehoe's comment. As a last resort, a credential exporter (#5261) can fill the remaining gaps in support.
⚠️ COMMENT VISIBILITY WARNING⚠️
Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
Breaks my process. Thanks a lot.
To echo @kdaily's comment, I've written an article on how the "AWS standard" for credentials is more comprehensive than just ~/.aws/credentials, and why credential_process is better than stuffing temporary credentials into ~/.aws/credentials or environment variables: https://medium.com/@ben11kehoe/never-put-aws-temporary-credentials-in-env-vars-or-credentials-files-theres-a-better-way-25ec45b4d73e
When using the configuration functionality of aws-sso-util (which can automatically create profiles for every account and role you have access to through AWS SSO), it automatically sets credential_process alongside the AWS SSO configuration to backfill for tools that don't yet support AWS SSO configuration. For tools that don't even support credential_process, there's aws-export-credentials as a last resort.