CDKToolkit failed to deploy resulting in CREATE_FAILED from permissions error.
jacido opened this issue · 2 comments
Repo Link: https://github.com/aws-samples/amazon-bedrock-audio-summarizer/tree/main?tab=readme-ov-file
Below Quickstart in the Prerequisites section. There is not a mention of additional IAM roles required. Specifically there are 4 IAM permissions mentioned and during deployment of the solution there were several IAM permission errors:
Where we encountered permission errors:
- Roles
- CloudFormationExecutionRole
- ImagePublishingRole
- FilePublishingRole
- Resources
- ssm:DeleteParameter
- ecr:CreateRepository
Expected behavior: CDKtoolkit stack deployment would be successful in Step 4 allowing to proceed to Step 5.
Behavior that occurred during the bootstrap:
Environment aws://8417XXXXXXXX/ap-south-1 failed bootstrapping: Error [ValidationError]: Stack:arn:aws:cloudformation:ap-south-1:8417XXXXXXXX:stack/CDKToolkit/37fcd710-0ef8-11ef-9f07-0244757fd391 is in DELETE_FAILED state and can not be updated.
at Request.extractError (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:46692)
at Request.callListeners (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:91600)
at Request.emit (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:91048)
at Request.emit (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:199651)
at Request.transition (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:193203)
at AcceptorStateMachine.runTo (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:158075)
at /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:158405
at Request. (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:193495)
at Request. (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:199726)
at Request.callListeners (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:390:91768) {
code: 'ValidationError',
time: 2024-05-10T18:10:33.045Z,
requestId: '08285ca9-60bf-4c0e-97bc-9909b1daef3f',
statusCode: 400,
retryable: false,
retryDelay: 378.37039647842306
}
Additional error encountered: Potential Signal Echo after deleting the CREATE_FAILED CDKToolkit stack and deleting the S3 bucket. Upon initating the new bootstrap, encoutered Signal Echo.
(venv) (base) nish@Nishants-MacBook-Air amazon-bedrock-audio-summarizer % cdk bootstrap
⏳ Bootstrapping environment aws://8417XXXXXXXX/ap-south-1...
Trusted accounts for deployment: (none)
Trusted accounts for lookup: (none)
Using default execution policy of 'arn:aws:iam::aws:policy/AdministratorAccess'. Pass '--cloudformation-execution-policies' to customize.
CDKToolkit: creating CloudFormation changeset...
12:48:15 AM | CREATE_FAILED | AWS::S3::Bucket | StagingBucket
Resource handler returned message: "cdk-hnb659fds-assets-8417XXXXXXXX-ap-south-1 already exists (Service: S3, Status Cod
e: 0, Request ID: null)" (RequestToken: e2d46156-eXXX-2326-2a54-3b2aXXXXX, HandlerErrorCode: AlreadyExists)
❌ Environment aws://8417XXXXXXXX/ap-south-1 failed bootstrapping: Error: The stack named CDKToolkit failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "cdk-hnbXXXXX-assets-8417XXXXXXXX-ap-south-1 already exists (Service: S3, Status Code: 0, Request ID: null)" (RequestToken: e2dXXXXX-e598-2326-2a54-3b2a107XXXXX, HandlerErrorCode: AlreadyExists)
at FullCloudFormationDeployment.monitorDeployment (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:440:10568)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:445:XXXX
at async Promise.all (index 0)
at async CdkToolkit.bootstrap (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:445:2174)
at async exec4 (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:498:54396)
Next steps to resolve:
- Add missing IAM permissions:
- ssm:DeleteParameter
- ecr:CreateRepository (Added AmazonEC2ContainerRegistryReadOnly) according to documentation.
- Grant Admin access (In lieu of AWS::IAM::Role's below missing)
- CloudFormationExecutionRole
- ImagePublishingRole
- FilePublishingRole
According to documentation, CDK are assumed by the AWS CDK Toolkit to publish assets into an environment: that is, to write to the S3 bucket and the ECR repository, respectively.
- Delete bootstrap stack and S3 bucket then redeploy CDK stack.
Result: CDK solution deployed successfully upon re-attempt.
Fix request: Include roles and additional permissions needed in the Prerequisites section.
We enabled full admin for the account. The customer is requesting what permissions he'll need for least privilege.
Used github rep example for Code source eventbridge-bedrock-inference. Customer uploaded an audio file manually 30 minutes ago to the S3 bucket from Step 5a of the repo which should have created a transcription folder which should process the transcription to create the summary to place into the process folder. Following the architecture path , after transcribe the from Amazon EventBridge it did not cross to the event-bridge-bedrock-inference. Eventbridge function was not triggered by the update.
Lambda name: eventbridge-bedrock-inference