[BUG] - Codebuild Project/Deployer Hangs
AllOtherUserNamesTaken opened this issue · 7 comments
Describe the bug
CodeBuild project hangs for several hours.
To Reproduce
Steps to reproduce the behavior:
- Clone AWS Labs github repository
- Install the CLI
- Install AWS CodeSeeder
- Generate a new manifest
- Deploy a new foundation (edit: skipped used existing VPC and subnets)
- Deploy a new toolkit
- Deploy credentials
- Deploy docker images
- Deploy environment (edit: last attempted step)
- Deploy teams (edit: never executed)
Expected behavior
Expect CodeBuild project to fail or succeed.
Screenshots
Desktop (please complete the following information):
- EC2 instance
- OS: [ubunut 20.04]
Additional context
While attempting to deploy, I had to make changes to both codeseeder and the cli to ensure that the S3 buckets for the toolkit were encrypted. I also had to update the orbit cli's remote-requirements.txt to use the latest 1.* release of AWS Python SDK because the current default version of node running in the codebuild project's agent does not support older versions of the SDK.
are you able to get the complete logs from the codebuild execution?
did you actually install the foundation or did you use existing vpcs, subnets, etc? I was able to reproduce the cdk hangup and have determined the cause, but the same issue would have affected the foundation deployment as well as the env deployment.
@chamcca you are correct I had to use existing VPCs and subnets, so I didn't run the foundation. There aren't additional logs, it just hangs.
I followed the guideline:
"If the above is defined, then you can populate the manifest with these values, removing the need to deploy a separate foundation."
The generated manifest file has this layout (I can't spot an issue with it):
{
"AccountId":"0000000000",
"CdkToolkit":{
"S3Bucket":"orbit-f-orbit-cdk-toolkit-0000000000-vtbduy",
"StackName":"orbit-f-orbit-cdk-toolkit"
},
"CodeartifactDomain":null,
"CodeartifactRepository":null,
"CognitoExternalProviderDomain":null,
"CognitoExternalProviderRedirect":null,
"CognitoUsersUrl":null,
"EnvTag":"orbit-f-orbit",
"Images":{
"CodeBuild":{
"Path":null,
"Repository":"public.ecr.aws/0000000000/aws-codeseeder/code-build-base",
"Version":"2.0.0"
}
},
"IsCodeartifactExternal":null,
"Name":"orbit",
"Networking":{
"AvailabilityZones":[
"eu-central-1a",
"eu-central-1b"
],
"Data":{
"InternetAccessible":false,
"NodesSubnets":[
"subnet-0000000000",
"subnet-0000000000"
]
},
"Frontend":{
"CustomDomainName":null,
"LoadBalancersSubnets":[
"subnet-111111111",
"subnet-111111111"
],
"SslCertArn":null
},
"IsolatedSubnets":[
],
"MaxAvailabilityZones":null,
"PrivateSubnets":[
{
"AvailabilityZone":"eu-central-1a",
"CidrBlock":"10.10.9.0/21",
"Kind":"private",
"RouteTableId":"rtb-0000000000",
"SubnetId":"subnet-0000000000",
"VpcId":"vpc-0000000000"
},
{
"AvailabilityZone":"eu-central-1b",
"CidrBlock":"10.10.9.0/21",
"Kind":"private",
"RouteTableId":"rtb-0000000000",
"SubnetId":"subnet-0000000000",
"VpcId":"vpc-0000000000"
}
],
"PublicSubnets":[
{
"AvailabilityZone":"eu-central-1b",
"CidrBlock":"10.10.40.20/28",
"Kind":"public",
"RouteTableId":"rtb-111111111",
"SubnetId":"subnet-111111111",
"VpcId":"vpc-0000000000"
},
{
"AvailabilityZone":"eu-central-1a",
"CidrBlock":"00.04.04.24/28",
"Kind":"public",
"RouteTableId":"rtb-111111111",
"SubnetId":"subnet-111111111",
"VpcId":"vpc-0000000000"
}
],
"SecondaryCidr":false,
"VpcCidrBlock":"10.20.11.000/27",
"VpcId":"vpc-0000000000"
},
"OrbitVersion":"1.6.0.dev0",
"Policies":[
],
"Region":"eu-central-1",
"ResourcesSsmParameterName":"/orbit-f/orbit/resources",
"RolePrefix":null,
"ScratchBucketArn":null,
"SsmParameterName":"/orbit-f/orbit/context",
"StackName":"orbit-f-orbit",
"Toolkit":{
"AdminRole":"orbit-f-orbit-eu-central-1-admin",
"AdminRoleArn":"arn:aws:iam::0000000000:role/orbit-f-orbit-eu-central-1-admin",
"CodeartifactDomain":"aws-codeseeder-orbit",
"CodeartifactRepo":"python-repository",
"CodebuildProject":"codeseeder-orbit",
"CodeseederPolicy":"arn:aws:iam::0000000000:policy/codeseeder-orbit-eu-central-1-resources",
"DeployId":"vtbduy",
"KmsAlias":null,
"KmsArn":"arn:aws:kms:eu-central-1:0000000000:key/0000000000-b691-4992-be04-0000000000",
"S3Bucket":"codeseeder-orbit-0000000000-qqqqq",
"StackName":"orbit-f-orbit-toolkit"
},
"UserPoolId":null
}@[chamcca]I have the same issue as well but my deployment get stack at foundation stage. It just stack on below stage (/remote_files/cdk/foundation.py ).
[2022-07-12 16:32:52,007][deploy.py :337] CDK Toolkit Stack deployed
578 | [2022-07-12 16:32:52,007][foundation.py: 34] Deploying self signed cert...
579 | [2022-07-12 16:32:52,007][cert.py : 35] Generating self-signed certificate...
580 | [2022-07-12 16:32:52,007][cert.py : 37] Creating private key...
581 | [2022-07-12 16:32:52,158][cert.py : 41] Creating Certificate signing request...
582 | [2022-07-12 16:32:52,166][cert.py : 49] Creating Public certificate...
583 | [2022-07-12 16:32:52,172][cert.py : 59] Uploading the cert to IAM...
584 | [2022-07-12 16:32:52,698][foundation.py: 37] Deploying orbit-f-dev-env Foundation...
585 | [2022-07-12 16:32:52,698][sh.py : 28] + cdk deploy --require-approval never --progress events --toolkit-stack-name orbit-f-dev-env-cdk-toolkit --app "python /root/.venv/lib/python3.7/site-packages/aws_orbit/remote_files/cdk/foundation.py dev-env arn:aws:iam::903917712828:server-certificate/dev-env-eu-west-1" --output .orbit.out/dev-env/cdk/orbit-f-dev-env
I've tracked this down to an incompatibility between the version of the CDK used for deployment and newer versions of the jsii library. with jsii > 1.60.0 the CDK cli will hang indefinitely. I was able to deploy by adding the following lines to the cli/remote-requirements.txt file:
constructs~=3.3.161
jsii~=1.42.0
I'll be issuing a PR with update later today after some additional testing and discussion with the team on cutting a patch release.
Thank you @chamcca we have also verified that this allowed progress.