aws/amazon-genomics-cli

Error: There are no 'Private' subnet groups in this VPC. Available types: Public

Closed this issue ยท 8 comments

rmontroy commented

If I'm not mistaken, usePublicSubnets should be passed in to the Batch constructor:

amazon-genomics-cli/packages/cdk/lib/stacks/engines/snakemake-engine-construct.ts

Lines 163 to 174 in 086d816

private renderBatch(id: string, vpc: IVpc, appParams: ContextAppParameters, computeType?: ComputeResourceType): Batch {
return new Batch(this, id, {
vpc,
computeType,
instanceTypes: appParams.instanceTypes,
maxVCpus: appParams.maxVCpus,
launchTemplateData: LaunchTemplateData.renderLaunchTemplateData(ENGINE_SNAKEMAKE),
awsPolicyNames: ["AmazonSSMManagedInstanceCore", "CloudWatchAgentServerPolicy"],
resourceTags: Stack.of(this).tags.tagValues(),
workflowOrchestrator: ENGINE_SNAKEMAKE,
});
} 

<user-id>:~/environment/amazon-genomics-cli/examples/demo-snakemake-project $ agc context deploy -c spotContext -v                   
2022-05-13T17:49:00Z โ†“  Checking AGC version...
2022-05-13T17:49:00Z ๐’Š  Deploying context(s)
2022-05-13T17:49:00Z โ†“  executeCDKClearContext(/home/ec2-user/.agc/cdk/apps/context)
2022-05-13T17:49:00Z โ†“  executeCDKCommand(/home/ec2-user/.agc/cdk/apps/context, [context --clear])
2022-05-13T17:49:01Z โ†“  
2022-05-13T17:49:01Z โ†“  > cdk@0.1.0 cdk
2022-05-13T17:49:01Z โ†“  > cd $INIT_CWD && cdk "context" "--clear"
2022-05-13T17:49:01Z โ†“  
2022-05-13T17:49:03Z โ†“  verifying presence of 'aws/snakemake-mirror:internal-fork' in region: 'us-east-2' of registry (account): '680431765560'
2022-05-13T17:49:03Z โ†“  executeCDKCommand(/home/ec2-user/.agc/cdk/apps/context, [deploy --all --profile  --require-approval never --toolkit-stack-name Agc-CDKToolkit --output /home/ec2-user/.agc/cdk/apps/context/cdk-output3517600036 -c USER_EMAIL=rob.montroy@vai.org -c AGC_VERSION=1.4.0 -c ENGINE_DESIGNATION=snakemake -c READ_BUCKET_ARNS= -c MAX_V_CPUS=256 -c PUBLIC_SUBNETS=true -c CONTEXT=spotContext -c OUTPUT_BUCKET=agc-<acct-id>-us-east-2 -c FILESYSTEM_TYPE= -c FS_PROVISIONED_THROUGHPUT=0 -c ENGINE_REPOSITORY= -c ENGINE_HEALTH_CHECK_PATH= -c ADAPTER_DESIGNATION= -c USER_ID=robmontroy2GXvkm -c CUSTOM_TAGS= -c ENGINE_NAME=snakemake -c ADAPTER_NAME= -c READ_WRITE_BUCKET_ARNS= -c BATCH_COMPUTE_INSTANCE_TYPES= -c PROJECT=SnakemakeDemo -c ADAPTER_REPOSITORY= -c ARTIFACT_BUCKET=agc-<acct-id>-us-east-2 -c REQUEST_SPOT_INSTANCES=true -c ECR_WES_ACCOUNT_ID=680431765560 -c ECR_WES_REGION=us-east-2 -c ECR_WES_TAG=0.1.0 -c ECR_WES_REPOSITORY=aws/wes-release -c ECR_CROMWELL_ACCOUNT_ID=680431765560 -c ECR_CROMWELL_REGION=us-east-2 -c ECR_CROMWELL_TAG=64 -c ECR_CROMWELL_REPOSITORY=aws/cromwell-mirror -c ECR_NEXTFLOW_ACCOUNT_ID=680431765560 -c ECR_NEXTFLOW_REGION=us-east-2 -c ECR_NEXTFLOW_TAG=21.04.3 -c ECR_NEXTFLOW_REPOSITORY=aws/nextflow-mirror -c ECR_MINIWDL_ACCOUNT_ID=680431765560 -c ECR_MINIWDL_REGION=us-east-2 -c ECR_MINIWDL_TAG=v0.1.11 -c ECR_MINIWDL_REPOSITORY=aws/miniwdl-mirror -c ECR_SNAKEMAKE_ACCOUNT_ID=680431765560 -c ECR_SNAKEMAKE_REGION=us-east-2 -c ECR_SNAKEMAKE_TAG=internal-fork -c ECR_SNAKEMAKE_REPOSITORY=aws/snakemake-mirror])
2022-05-13T17:49:03Z โ†“  
2022-05-13T17:49:03Z โ†“  > cdk@0.1.0 cdk
2022-05-13T17:49:03Z โ†“  > cd $INIT_CWD && cdk "deploy" "--all" "--profile" "" "--require-approval" "never" "--toolkit-stack-name" "Agc-CDKToolkit" "--output" "/home/ec2-user/.agc/cdk/apps/context/cdk-output3517600036" "-c" "USER_EMAIL=rob.montroy@vai.org" "-c" "AGC_VERSION=1.4.0" "-c" "ENGINE_DESIGNATION=snakemake" "-c" "READ_BUCKET_ARNS=" "-c" "MAX_V_CPUS=256" "-c" "PUBLIC_SUBNETS=true" "-c" "CONTEXT=spotContext" "-c" "OUTPUT_BUCKET=agc-<acct-id>-us-east-2" "-c" "FILESYSTEM_TYPE=" "-c" "FS_PROVISIONED_THROUGHPUT=0" "-c" "ENGINE_REPOSITORY=" "-c" "ENGINE_HEALTH_CHECK_PATH=" "-c" "ADAPTER_DESIGNATION=" "-c" "USER_ID=robmontroy2GXvkm" "-c" "CUSTOM_TAGS=" "-c" "ENGINE_NAME=snakemake" "-c" "ADAPTER_NAME=" "-c" "READ_WRITE_BUCKET_ARNS=" "-c" "BATCH_COMPUTE_INSTANCE_TYPES=" "-c" "PROJECT=SnakemakeDemo" "-c" "ADAPTER_REPOSITORY=" "-c" "ARTIFACT_BUCKET=agc-<acct-id>-us-east-2" "-c" "REQUEST_SPOT_INSTANCES=true" "-c" "ECR_WES_ACCOUNT_ID=680431765560" "-c" "ECR_WES_REGION=us-east-2" "-c" "ECR_WES_TAG=0.1.0" "-c" "ECR_WES_REPOSITORY=aws/wes-release" "-c" "ECR_CROMWELL_ACCOUNT_ID=680431765560" "-c" "ECR_CROMWELL_REGION=us-east-2" "-c" "ECR_CROMWELL_TAG=64" "-c" "ECR_CROMWELL_REPOSITORY=aws/cromwell-mirror" "-c" "ECR_NEXTFLOW_ACCOUNT_ID=680431765560" "-c" "ECR_NEXTFLOW_REGION=us-east-2" "-c" "ECR_NEXTFLOW_TAG=21.04.3" "-c" "ECR_NEXTFLOW_REPOSITORY=aws/nextflow-mirror" "-c" "ECR_MINIWDL_ACCOUNT_ID=680431765560" "-c" "ECR_MINIWDL_REGION=us-east-2" "-c" "ECR_MINIWDL_TAG=v0.1.11" "-c" "ECR_MINIWDL_REPOSITORY=aws/miniwdl-mirror" "-c" "ECR_SNAKEMAKE_ACCOUNT_ID=680431765560" "-c" "ECR_SNAKEMAKE_REGION=us-east-2" "-c" "ECR_SNAKEMAKE_TAG=internal-fork" "-c" "ECR_SNAKEMAKE_REPOSITORY=aws/snakemake-mirror"
2022-05-13T17:49:03Z โ†“  
2022-05-13T17:49:22Z ๐’Š  current credentials could not be used to assume 'arn:aws:iam::<acct-id>:role/cdk-agc-lookup-role-<acct-id>-us-east-2', but are for the right account. Proceeding anyway.
2022-05-13T17:49:22Z ๐’Š  current credentials could not be used to assume 'arn:aws:iam::<acct-id>:role/cdk-agc-lookup-role-<acct-id>-us-east-2', but are for the right account. Proceeding anyway.
2022-05-13T17:49:22Z ๐’Š  current credentials could not be used to assume 'arn:aws:iam::<acct-id>:role/cdk-agc-lookup-role-<acct-id>-us-east-2', but are for the right account. Proceeding anyway.
2022-05-13T17:49:38Z ๐’Š  current credentials could not be used to assume 'arn:aws:iam::<acct-id>:role/cdk-agc-lookup-role-<acct-id>-us-east-2', but are for the right account. Proceeding anyway.
2022-05-13T17:49:53Z ๐’Š  /home/ec2-user/.agc/cdk/node_modules/aws-cdk-lib/aws-ec2/lib/vpc.ts:401
2022-05-13T17:49:53Z ๐’Š        throw new Error(`There are no '${subnetType}' subnet groups in this VPC. Available types: ${availableTypes}`);
2022-05-13T17:49:53Z ๐’Š              ^
2022-05-13T17:49:53Z ๐’Š  Error: There are no 'Private' subnet groups in this VPC. Available types: Public
2022-05-13T17:49:54Z ๐’Š      at LookedUpVpc.selectSubnetObjectsByType (/home/ec2-user/.agc/cdk/node_modules/aws-cdk-lib/aws-ec2/lib/vpc.ts:401:13)
2022-05-13T17:49:54Z ๐’Š      at LookedUpVpc.selectSubnetObjects (/home/ec2-user/.agc/cdk/node_modules/aws-cdk-lib/aws-ec2/lib/vpc.ts:357:22)
2022-05-13T17:49:54Z ๐’Š      at LookedUpVpc.selectSubnets (/home/ec2-user/.agc/cdk/node_modules/aws-cdk-lib/aws-ec2/lib/vpc.ts:249:26)
2022-05-13T17:49:54Z ๐’Š      at new ComputeEnvironment (/home/ec2-user/.agc/cdk/node_modules/@aws-cdk/aws-batch-alpha/lib/compute-environment.ts:171:45)
2022-05-13T17:49:54Z ๐’Š      at Batch.renderComputeEnvironment (/home/ec2-user/.agc/cdk/lib/constructs/batch.ts:187:14)
2022-05-13T17:49:54Z ๐’Š      at new Batch (/home/ec2-user/.agc/cdk/lib/constructs/batch.ts:104:36)
2022-05-13T17:49:54Z ๐’Š      at SnakemakeEngineConstruct.renderBatch (/home/ec2-user/.agc/cdk/lib/stacks/engines/snakemake-engine-construct.ts:164:12)
2022-05-13T17:49:54Z ๐’Š      at new SnakemakeEngineConstruct (/home/ec2-user/.agc/cdk/lib/stacks/engines/snakemake-engine-construct.ts:34:27)
2022-05-13T17:49:54Z ๐’Š      at ContextStack.renderSnakemakeStack (/home/ec2-user/.agc/cdk/lib/stacks/context-stack.ts:121:5)
2022-05-13T17:49:54Z ๐’Š      at new ContextStack (/home/ec2-user/.agc/cdk/lib/stacks/context-stack.ts:56:14)
2022-05-13T17:49:54Z ๐’Š  Subprocess exited with error 1
2022-05-13T17:49:54Z โœ˜  Failed to deploy context 'spotContext'. Below is the log for that deployment
2022-05-13T17:49:54Z โœ˜  current credentials could not be used to assume 'arn:aws:iam::<acct-id>:role/cdk-agc-lookup-role-<acct-id>-us-east-2', but are for the right account. Proceeding anyway.
2022-05-13T17:49:54Z โœ˜  current credentials could not be used to assume 'arn:aws:iam::<acct-id>:role/cdk-agc-lookup-role-<acct-id>-us-east-2', but are for the right account. Proceeding anyway.
2022-05-13T17:49:54Z โœ˜  current credentials could not be used to assume 'arn:aws:iam::<acct-id>:role/cdk-agc-lookup-role-<acct-id>-us-east-2', but are for the right account. Proceeding anyway.
2022-05-13T17:49:54Z โœ˜  current credentials could not be used to assume 'arn:aws:iam::<acct-id>:role/cdk-agc-lookup-role-<acct-id>-us-east-2', but are for the right account. Proceeding anyway.
2022-05-13T17:49:54Z โœ˜  /home/ec2-user/.agc/cdk/node_modules/aws-cdk-lib/aws-ec2/lib/vpc.ts:401
2022-05-13T17:49:54Z โœ˜        throw new Error(`There are no '${subnetType}' subnet groups in this VPC. Available types: ${availableTypes}`);
2022-05-13T17:49:54Z โœ˜              ^
2022-05-13T17:49:54Z โœ˜  Error: There are no 'Private' subnet groups in this VPC. Available types: Public
2022-05-13T17:49:54Z โœ˜      at LookedUpVpc.selectSubnetObjectsByType (/home/ec2-user/.agc/cdk/node_modules/aws-cdk-lib/aws-ec2/lib/vpc.ts:401:13)
2022-05-13T17:49:54Z โœ˜      at LookedUpVpc.selectSubnetObjects (/home/ec2-user/.agc/cdk/node_modules/aws-cdk-lib/aws-ec2/lib/vpc.ts:357:22)
2022-05-13T17:49:54Z โœ˜      at LookedUpVpc.selectSubnets (/home/ec2-user/.agc/cdk/node_modules/aws-cdk-lib/aws-ec2/lib/vpc.ts:249:26)
2022-05-13T17:49:54Z โœ˜      at new ComputeEnvironment (/home/ec2-user/.agc/cdk/node_modules/@aws-cdk/aws-batch-alpha/lib/compute-environment.ts:171:45)
2022-05-13T17:49:54Z โœ˜      at Batch.renderComputeEnvironment (/home/ec2-user/.agc/cdk/lib/constructs/batch.ts:187:14)
2022-05-13T17:49:54Z โœ˜      at new Batch (/home/ec2-user/.agc/cdk/lib/constructs/batch.ts:104:36)
2022-05-13T17:49:54Z โœ˜      at SnakemakeEngineConstruct.renderBatch (/home/ec2-user/.agc/cdk/lib/stacks/engines/snakemake-engine-construct.ts:164:12)
2022-05-13T17:49:54Z โœ˜      at new SnakemakeEngineConstruct (/home/ec2-user/.agc/cdk/lib/stacks/engines/snakemake-engine-construct.ts:34:27)
2022-05-13T17:49:54Z โœ˜      at ContextStack.renderSnakemakeStack (/home/ec2-user/.agc/cdk/lib/stacks/context-stack.ts:121:5)
2022-05-13T17:49:54Z โœ˜      at new ContextStack (/home/ec2-user/.agc/cdk/lib/stacks/context-stack.ts:56:14)
2022-05-13T17:49:54Z โœ˜  
2022-05-13T17:49:54Z โœ˜  Subprocess exited with error 1 
2022-05-13T17:49:54Z โœ˜   error="1 context deployment failures"
Error: an error occurred invoking 'context deploy'
with variables: {contexts:[spotContext] deployAll:false}
caused by: 1 context deployment failures
suggestion: To resolve failure 1, determine the cause of: exit status 1

Operating System: AL2
AGC Version: 1.4.0
Was AGC setup with a custom bucket: no
Was AGC setup with a custom VPC: no

rmontroy commented

@multimeric amirite?

multimeric commented

Yes, that looks like the right fix. Actually it needs to be changed for all the engine constructs. I'm not sure how this ever worked, actually, but I know that it did in my own testing. Very strange.

rmontroy commented

@markjschreiber This would have been nice to know before I started down this path:

amazon-genomics-cli/packages/cdk/lib/stacks/context-stack.ts

Lines 74 to 75 in 3a5a9b5

if (contextParameters.usePublicSubnets) {
throw Error(`'snakemake is not currently supported using public subnets, please file a github issue detailing your use case'`);

multimeric commented

Oh, I wonder what the issue is with using public subnets for all these workflow engines?

markjschreiber commented

Snakemake, and MiniWDL engines run as Fargate tasks. The tasks are created by the WES adapter Lambda function and a public IP must be requested at this time otherwise the engine will have a private IP and nothing will be able to talk to it (because there will be no NAT gateway). The fix would be to have the CLI signal that a public IP is needed when it submits the workflow. Not sure how well this will play with the WES API though. The WES adapter python code would then need to take that into account when creating the engines Fargate task.

  • ๐Ÿ‘1
markjschreiber commented

Using public subnets is actually a pretty tricky thing to get right for most of the engines.

rmontroy commented

Not sure what the difference is, but if I comment out that restriction above, I'm not able to reproduce the original error using the latest code.

rmontroy commented

Snakemake, and MiniWDL engines run as Fargate tasks. The tasks are created by the WES adapter Lambda function and a public IP must be requested at this time otherwise the engine will have a private IP and nothing will be able to talk to it (because there will be no NAT gateway). The fix would be to have the CLI signal that a public IP is needed when it submits the workflow. Not sure how well this will play with the WES API though. The WES adapter python code would then need to take that into account when creating the engines Fargate task.

@markjschreiber I took a stab at removing this limitation in this PR: #470. Let me know if I'm on the right track, and how to add tests if possible.

  • ๐Ÿ‘€1