Go Panic during all-checks
Closed this issue · 15 comments
Description of Bug
When running cloudfox aws all-checks the program crashes during the privesc phase with:
[resource-trusts][666412488974-AIDAJBTEDJEDFGHJKLWJU] Supported Services: CodeBuild, ECR, EFS, Glue, Lambda, SecretsManager, S3, SNS, SQS
failed to get shared config profile, 116412344578-AIDAJBTEDJEDFGHJKLWJU
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x101022724]
goroutine 9981 [running]:
github.com/BishopFox/cloudfox/internal.AWSConfigFileLoader({_, _}, {_, _}, {_, _})
github.com/BishopFox/cloudfox/internal/aws.go:67 +0x3e4
github.com/BishopFox/cloudfox/aws.initCloudFoxS3Client({_, _, _, {_}, {}}, {_, _}, {_, _}, {_, ...})
github.com/BishopFox/cloudfox/aws/client-initializers.go:48 +0x80
github.com/BishopFox/cloudfox/aws.(*ResourceTrustsModule).getS3Buckets(0x140011a4580, 0x100157644?, 0x14002196660, 0x140006733f0?)
github.com/BishopFox/cloudfox/aws/resource-trusts.go:371 +0x114
created by github.com/BishopFox/cloudfox/aws.(*ResourceTrustsModule).PrintResources in goroutine 1
github.com/BishopFox/cloudfox/aws/resource-trusts.go:94 +0x8c8
What should the expected behavior be
Not crash
Platform Affected
macOS Darwin 21.6.0 ARM64 (macOS Monterey)
Steps to Reproduce
- Install via
brew - Use
aws-vaultto start an authenticated shell with STS credentials - Run
cloudfox aws all-checks
Hey @johnkeates - Thanks for this bug report! I see you ran all-checks, but it looks like the error only hits around resource-trusts. Was everything working well up until then? Does it always error out at the same spot (resource-trusts)? I'm wondering if its related to the session timing out.
After the cloudfox error, can you do an aws sts get-caller-identity --profile [vault profile] still?
Are you saying you got the sts credentions with something like aws sts assume-role --role-arn and then entered the ID, SECRET, and TOKEN into aws-vault?
Hi @sethsec-bf it does indeed break at the same point every time, the resource-trusts part. Everything else up to that point works correctly.
The aws-vault stuff ends up spawning a shell (or running an individual command) with preset environment variables that contain the usual suspects for STS tokens: AWS_REGION, AWS_DEFAULT_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, AWS_CREDENTIAL_EXPIRATION
Getting a caller-identity does still work after the run finishes, but running cloudfox aws resource-trusts separately still breaks:
% cloudfox aws resource-trusts
[🦊 cloudfox v1.13.0 🦊 ][] AWS Caller Identity: arn:aws:iam::123412341234:user/jkeates
[🦊 cloudfox v1.13.0 🦊 ][] Account is part of an Organization and is a child account. Management Account: 678967893
[resource-trusts][123412341234-AIDAJBTEDJE6HXXXXXXX] Enumerating Resources with resource policies for account 123412341234.
[resource-trusts][123412341234-AIDAJBTEDJE6HXXXXXXX] Supported Services: CodeBuild, ECR, EFS, Glue, Lambda, SecretsManager, S3, SNS, SQS
failed to get shared config profile, 123412341234-AIDAJBTEDJE6HXXXXXXX
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x103146724]
goroutine 102 [running]:
github.com/BishopFox/cloudfox/internal.AWSConfigFileLoader({_, _}, {_, _}, {_, _})
github.com/BishopFox/cloudfox/internal/aws.go:67 +0x3e4
github.com/BishopFox/cloudfox/aws.initCloudFoxS3Client({_, _, _, {_}, {}}, {_, _}, {_, _}, {_, ...})
github.com/BishopFox/cloudfox/aws/client-initializers.go:48 +0x80
github.com/BishopFox/cloudfox/aws.(*ResourceTrustsModule).getS3Buckets(0x140005ce2c0, 0x140003afe60?, 0x1400020c0c0, 0x1400020c240?)
github.com/BishopFox/cloudfox/aws/resource-trusts.go:371 +0x114
created by github.com/BishopFox/cloudfox/aws.(*ResourceTrustsModule).PrintResources in goroutine 1
github.com/BishopFox/cloudfox/aws/resource-trusts.go:94 +0x8c8
The user arn:aws:iam::123412341234:user/jkeates is one set of credentials, same happens with STS credentials for a role.
% aws sts get-caller-identity
{
"UserId": "AIDAJBTEDJE6HXXXXXXX",
"Account": "123412341234",
"Arn": "arn:aws:iam:: 123412341234:user/jkeates"
}
thanks for the extra info, that really helps. i'll look into this now
Considering the logger seems to assume a profile was used, and the last error lines assume that too, I'll also try again but with a 'classic' hardcoded profile (aws-vault uses the system keychain or SSO instead of plaintext long lived keys).
I'll try the ec2 instance role metadata emulation feature too to see if that does anything different. I haven't dug into the code yet, but I can do that later on as well.
yea pretty sure it has to do with an "unnamed" profile. when you don't use a profile and instead use environment variables or ec2 IMDS creds, cloudfox makes a pseudo profile for file storage and stuff. usually things just work out but i was easily able to replicate the issue you have with resource-trusts command. so i must have done something slightly differently there.
Using a classic (and also via a passthrough credential_process) profile everything works as expected, so that's good to know.
During a copy-paste mistake I got this interesting (unrelated) panic:
% cloudfox aws --profile=--profile=bftest all-checks
failed to get shared config profile, --profile=bftest
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x103676724]
goroutine 1 [running]:
github.com/BishopFox/cloudfox/internal.AWSConfigFileLoader({_, _}, {_, _}, {_, _})
github.com/BishopFox/cloudfox/internal/aws.go:67 +0x3e4
github.com/BishopFox/cloudfox/internal.AWSWhoami({0x16d6bf95f, 0x10}, {0x1066a5bc8, 0x6}, {0x0?, 0x102757720?})
github.com/BishopFox/cloudfox/internal/aws.go:91 +0x38
github.com/BishopFox/cloudfox/cli.awsPreRun(0x1400026b900?, {0x1066a368d?, 0x4?, 0x1066a3691?})
github.com/BishopFox/cloudfox/cli/aws.go:513 +0xcc
github.com/spf13/cobra.(*Command).execute(0x109816280, {0x140003432e0, 0x1, 0x1})
github.com/spf13/cobra@v1.8.0/command.go:972 +0x7c0
github.com/spf13/cobra.(*Command).ExecuteC(0x1098100c0)
github.com/spf13/cobra@v1.8.0/command.go:1115 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
github.com/spf13/cobra@v1.8.0/command.go:1039
main.main()
github.com/BishopFox/cloudfox/main.go:19 +0x58
I could create an issue for that, but I suspect most people can figure out why the panic happens in the first place.
Ok. Found and fixed the bug in resource trusts, thanks for reporting this one. Def was a sharp edge in the way I have one command call functionality from another command. This might need a better fix in the long run but i think this shoudl work for now. Can you try checkout the seth-dev branch and build from that let me know if it works?
Also, thanks for documenting the non existing profile bug. That is something that used to work but I broke it at some point, so thanks for the bug report!
I'll check it out!
Works 👍
(edited for brevity)
% ./cloudfox aws role-trusts
[🦊 cloudfox v1.13.1-prerelease 🦊 ][] AWS Caller Identity: arn:aws:sts::335430111111:assumed-role/Administrator/1706021397146943000
[🦊 cloudfox v1.13.1-prerelease 🦊 ][] Account is part of an Organization and is a child account. Management Account: 649301222222
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] Enumerating role trusts for account 335430111111.
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] Output written to /Users/jkeates/.cloudfox/cloudfox-output/aws/335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000-335430111111/table/role-trusts-principals.txt
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] Output written to /Users/jkeates/.cloudfox/cloudfox-output/aws/335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000-335430111111/table/role-trusts-principals-root-trusts-without-external-id.txt
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] Output written to /Users/jkeates/.cloudfox/cloudfox-output/aws/335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000-335430111111/table/role-trusts-services.txt
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] Output written to /Users/jkeates/.cloudfox/cloudfox-output/aws/335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000-335430111111/table/role-trusts-federated.txt
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] Output written to /Users/jkeates/.cloudfox/cloudfox-output/aws/335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000-335430111111/csv/role-trusts-principals.csv
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] Output written to /Users/jkeates/.cloudfox/cloudfox-output/aws/335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000-335430111111/csv/role-trusts-principals-root-trusts-without-external-id.csv
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] Output written to /Users/jkeates/.cloudfox/cloudfox-output/aws/335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000-335430111111/csv/role-trusts-services.csv
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] Output written to /Users/jkeates/.cloudfox/cloudfox-output/aws/335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000-335430111111/csv/role-trusts-federated.csv
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] Output written to /Users/jkeates/.cloudfox/cloudfox-output/aws/335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000-335430111111/json/role-trusts-principals.json
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] Output written to /Users/jkeates/.cloudfox/cloudfox-output/aws/335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000-335430111111/json/role-trusts-principals-root-trusts-without-external-id.json
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] Output written to /Users/jkeates/.cloudfox/cloudfox-output/aws/335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000-335430111111/json/role-trusts-services.json
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] Output written to /Users/jkeates/.cloudfox/cloudfox-output/aws/335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000-335430111111/json/role-trusts-federated.json
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] 11 principal role trusts found.
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] 24 principal role trusts found.
[role-trusts][335430111111-AROAU4GJ3AAAAAAAAAAAA_1706021397146943000] 9 principal role trusts found.
[🦊 cloudfox v1.13.1-prerelease 🦊 ][] Cached AWS data written to /Users/jkeates/.cloudfox/cached-data/aws/335430111111
I found another teeny tiny related issue, pmapper expects the same profile as cloud fox, so when you run the suggested pmapper command it wouldn't be able to find the same profile as the temporarily created profile name because.. well it's temporary isn't it 😁
Looking at pmapper a bit more, it seems to be able to create a graph with non-profile commands, but I don't know if it will store the graph values somewhere in a known location vs. how it stores the data in Cloudfox. Maybe it's not a big deal since the pmapper data is available outside of Cloudfox anyway.
Just commenting here that the patch in seth-dev did not fix the issue for me. I noticed that @johnkeates ran role-trusts command above and not resource-trusts so i'm wondering if the issue is actually resolved on their side as well. Funny thing for me is that manually running resource-trusts worked, but not when it got to the point in all-checks. Sorry, I won't be able to help debug any further, just wanted to include my two cents if it enabled @johnkeates to help further.
Thanks @cyberbutler! Can you pull the latest change from seth-dev and try again? The fix for @johnkeates was kind of a hack that only worked if the AWS ID came back as AROAxxxxxxx. The new fix should be more universal.
@johnkeates - can you also confirm that the latest code works for you with the resource-trusts command as well as the all-checks command?
Thanks!!
@sethsec-bf I am now getting the following error:
panic: runtime error: index out of range [0] with length 0
goroutine 4523 [running]:
github.com/BishopFox/cloudfox/aws/sdk.CachedCodeBuildBatchGetProjects({_, _}, {_, _}, {_, _}, {_, _})
/redshare/Assessments/DAS-2023/AWS/cloudfox/aws/sdk/codebuild.go:90 +0x45a
github.com/BishopFox/cloudfox/aws.(*ResourceTrustsModule).getCodeBuildResourcePoliciesPerRegion(0xc0005922c0, {0xc001d04210, 0xd}, 0xc001d04210?, 0xc0033161e0, 0xc00178dfb8?)
/redshare/Assessments/DAS-2023/AWS/cloudfox/aws/resource-trusts.go:595 +0x37e
github.com/BishopFox/cloudfox/aws.(*ResourceTrustsModule).executeChecks(0xc0005922c0, {0xc001d04210, 0xd}, 0xc001d04510, 0x6f0b25?, 0xc0001fcfc0?)
/redshare/Assessments/DAS-2023/AWS/cloudfox/aws/resource-trusts.go:233 +0x405
created by github.com/BishopFox/cloudfox/aws.(*ResourceTrustsModule).PrintResources in goroutine 1
/redshare/Assessments/DAS-2023/AWS/cloudfox/aws/resource-trusts.go:92 +0x957
I am currently running from an assumed role that i've created through environment variables. I unfortunately have to stop my accesses of this environment, so I may not be able to assist any further than this. My apologies!
thanks for the context!
Hey @cyberbutler - I understand if you don't have access anymore, but if you do, can you check out and try the latest. you actually found a completely unrelated bug lol, but i think i fixed it!
Also, thank you so much for taking the time to chime in and report the other bug!
Fixed the pmapper issue also. I think all three issues are fixed now. I'm going to push to main and make a new release. Feel free to re-open if is not fixed.