CFN Flip unable to parse AWS Amplify CLI Function CloudFormation templates

Question

CFN Flip unable to parse AWS Amplify CLI Function CloudFormation templates

paolo573 opened this issue 3 years ago · 11 comments

I am trying to add CloudWatch alarms to my Amplify Functions so that I get paged when certain events happen.

I was able to find the appropriate CloudFormation config, and applied it manually to the yaml (to test; this approach would get overwritten by the CLI, so is not a permanent solution) and the alarms and topics created successfully, so i began work to script this.

I am already using AWS cfn_tools to modify the Amplify Auth CF template, and cfn_tools is able to load and dump the Auth package CF template without any issues.

When I took the same approach with the Amplify Function CF templates (there is one for each Function) I ran into the complaint from cfn_tools.

Steps to reproduce:

Create a Lambda Resolver with AWS Amplify CLI
cfn_tools.load_yaml the Function's CF template
Try to dump that loaded yaml as below:

dumper = cfn_flip.yaml_dumper.get_dumper(clean_up=True, long_form=False)
raw_markup = yaml.dump(
        markup,
        Dumper=dumper,
        default_flow_style=False,
        allow_unicode=True
)

The array for ShouldNotCreateEnvResources condition seems to be the offender.

System details:

Amplify 4.44.2
Python Lambda Resolvers

Originally posted this to the amplify-cli repo aws-amplify/amplify-cli#7138

Answer 1 · 2021-04-20T21:21:14.000Z

Hi Paolo,

The cfn_flip provide tools to convert from json to yaml.
Can you test your code using our functions to_yaml?

For example:

from cfn_flip import to_yaml

raw_markup = to_yaml(markup, clean_up=True, long_form=False)

If possible, can you share here the content of markup in your scenario?

Answer 2 · 2021-04-22T20:49:38.000Z

Hey @koiker, the issue I'm having is with the loading of the serialized CF template, so that I can add a CW alarm and SNS topic.

The to_yaml function you're proposing still just gives me a serialized output. There's no indication that cfn_tools.load_yaml is having trouble with that part, so when i feed the (still) serialized output from to_yaml to the loader, I run into the same issue.

The CF templates we're trying to modify are out of the box Amplify Functions, so creating an Amplify app and adding a Python Lambda Resolver to it will produce a similar template. Here's a copy of one of our Lambda Resolver CF templates (ie. what we are trying to load and add objects to, and then dump back to file):

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "Lambda Function resource stack creation using Amplify CLI",
  "Parameters": {
    "CloudWatchRule": {
      "Type": "String",
      "Default": "NONE",
      "Description": " Schedule Expression"
    },
    "env": {
      "Type": "String"
    },
    "storagexBucketName": {
      "Type": "String",
      "Default": "storagexBucketName"
    },
    "deploymentBucketName": {
      "Type": "String"
    },
    "s3Key": {
      "Type": "String"
    }
  },
  "Conditions": {
    "ShouldNotCreateEnvResources": {
      "Fn::Equals": [
        {
          "Ref": "env"
        },
        "NONE"
      ]
    }
  },
  "Resources": {
    "LambdaFunction": {
      "Type": "AWS::Lambda::Function",
      "Metadata": {
        "aws:asset:path": "./src",
        "aws:asset:property": "Code"
      },
      "Properties": {
        "Handler": "index.handler",
        "FunctionName": {
          "Fn::If": [
            "ShouldNotCreateEnvResources",
            "YLambdaResolver",
            {
              "Fn::Join": [
                "",
                [
                  "YLambdaResolver",
                  "-",
                  {
                    "Ref": "env"
                  }
                ]
              ]
            }
          ]
        },
        "Environment": {
          "Variables": {
            "ENV": {
              "Ref": "env"
            },
            "REGION": {
              "Ref": "AWS::Region"
            },
            "STORAGE_X_BUCKETNAME": {
              "Ref": "storagexBucketName"
            }
          }
        },
        "Role": {
          "Fn::GetAtt": [
            "LambdaExecutionRole",
            "Arn"
          ]
        },
        "Runtime": "python3.8",
        "Layers": [],
        "Timeout": "25",
        "Code": {
          "S3Bucket": {
            "Ref": "deploymentBucketName"
          },
          "S3Key": {
            "Ref": "s3Key"
          }
        }
      }
    },
    "LambdaExecutionRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "RoleName": {
          "Fn::If": [
            "ShouldNotCreateEnvResources",
            "xpocLambdaRolec8",
            {
              "Fn::Join": [
                "",
                [
                  "xpocLambdaRolec8",
                  "-",
                  {
                    "Ref": "env"
                  }
                ]
              ]
            }
          ]
        },
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": [
                  "lambda.amazonaws.com"
                ]
              },
              "Action": [
                "sts:AssumeRole"
              ]
            }
          ]
        }
      }
    },
    "lambdaexecutionpolicy": {
      "DependsOn": [
        "LambdaExecutionRole"
      ],
      "Type": "AWS::IAM::Policy",
      "Properties": {
        "PolicyName": "lambda-execution-policy",
        "Roles": [
          {
            "Ref": "LambdaExecutionRole"
          }
        ],
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
              ],
              "Resource": {
                "Fn::Sub": [
                  "arn:aws:logs:${region}:${account}:log-group:/aws/lambda/${lambda}:log-stream:*",
                  {
                    "region": {
                      "Ref": "AWS::Region"
                    },
                    "account": {
                      "Ref": "AWS::AccountId"
                    },
                    "lambda": {
                      "Ref": "LambdaFunction"
                    }
                  }
                ]
              }
            }
          ]
        }
      }
    },
    "AmplifyResourcesPolicy": {
      "DependsOn": [
        "LambdaExecutionRole"
      ],
      "Type": "AWS::IAM::Policy",
      "Properties": {
        "PolicyName": "amplify-lambda-execution-policy",
        "Roles": [
          {
            "Ref": "LambdaExecutionRole"
          }
        ],
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "s3:GetObject",
                "s3:ListBucket"
              ],
              "Resource": [
                {
                  "Fn::Join": [
                    "",
                    [
                      "arn:aws:s3:::",
                      {
                        "Ref": "storagexBucketName"
                      },
                      "/*"
                    ]
                  ]
                },
                {
                  "Fn::Join": [
                    "",
                    [
                      "arn:aws:s3:::",
                      {
                        "Ref": "storagexBucketName"
                      }
                    ]
                  ]
                }
              ]
            }
          ]
        }
      }
    }
  },
  "Outputs": {
    "Name": {
      "Value": {
        "Ref": "LambdaFunction"
      }
    },
    "Arn": {
      "Value": {
        "Fn::GetAtt": [
          "LambdaFunction",
          "Arn"
        ]
      }
    },
    "Region": {
      "Value": {
        "Ref": "AWS::Region"
      }
    },
    "LambdaExecutionRole": {
      "Value": {
        "Ref": "LambdaExecutionRole"
      }
    }
  }
}

Answer 3 · 2021-04-27T02:35:37.000Z

Hi Paolo,

I'm not able to reproduce your issue.

Here is the steps that I took to reproduce your environment:

Generate a file test_amplify_lambda_resolver.json
Put this file in the examples/ folder in the repo
Create a pytest file: test_amplify.py with the test:

import cfn_flip
import cfn_tools
import yaml


def test_amplify_lambda_resolver_2():
    with open("examples/test_amplify_lambda_resolver.json", "r") as f:
        json_file = f.read()
        markup_json = cfn_tools.load_json(json_file)
        print(markup_json)
        print("Converting to YAML")
        markup_yaml = cfn_tools.dump_yaml(markup_json)
        markup = cfn_tools.load_yaml(markup_yaml)
        dumper = cfn_flip.yaml_dumper.get_dumper(clean_up=True, long_form=False)
        raw_markup = yaml.dump(
            markup,
            Dumper=dumper,
            default_flow_style=False,
            allow_unicode=True
        )
        print("raw_markup---------------------")
        print(raw_markup)
        assert 1 == 1

The output of my tests don't generate any error from the dump, just this yaml output:

ODict([('AWSTemplateFormatVersion', '2010-09-09'), ('Description', 'Lambda Function resource stack creation using Amplify CLI'), ('Parameters', ODict([('CloudWatchRule', ODict([('Type', 'String'), ('Default', 'NONE'), ('Description', ' Schedule Expression')])), ('env', ODict([('Type', 'String')])), ('storagexBucketName', ODict([('Type', 'String'), ('Default', 'storagexBucketName')])), ('deploymentBucketName', ODict([('Type', 'String')])), ('s3Key', ODict([('Type', 'String')]))])), ('Conditions', ODict([('ShouldNotCreateEnvResources', ODict([('Fn::Equals', [ODict([('Ref', 'env')]), 'NONE'])]))])), ('Resources', ODict([('LambdaFunction', ODict([('Type', 'AWS::Lambda::Function'), ('Metadata', ODict([('aws:asset:path', './src'), ('aws:asset:property', 'Code')])), ('Properties', ODict([('Handler', 'index.handler'), ('FunctionName', ODict([('Fn::If', ['ShouldNotCreateEnvResources', 'YLambdaResolver', ODict([('Fn::Join', ['', ['YLambdaResolver', '-', ODict([('Ref', 'env')])]])])])])), ('Environment', ODict([('Variables', ODict([('ENV', ODict([('Ref', 'env')])), ('REGION', ODict([('Ref', 'AWS::Region')])), ('STORAGE_X_BUCKETNAME', ODict([('Ref', 'storagexBucketName')]))]))])), ('Role', ODict([('Fn::GetAtt', ['LambdaExecutionRole', 'Arn'])])), ('Runtime', 'python3.8'), ('Layers', []), ('Timeout', '25'), ('Code', ODict([('S3Bucket', ODict([('Ref', 'deploymentBucketName')])), ('S3Key', ODict([('Ref', 's3Key')]))]))]))])), ('LambdaExecutionRole', ODict([('Type', 'AWS::IAM::Role'), ('Properties', ODict([('RoleName', ODict([('Fn::If', ['ShouldNotCreateEnvResources', 'xpocLambdaRolec8', ODict([('Fn::Join', ['', ['xpocLambdaRolec8', '-', ODict([('Ref', 'env')])]])])])])), ('AssumeRolePolicyDocument', ODict([('Version', '2012-10-17'), ('Statement', [ODict([('Effect', 'Allow'), ('Principal', ODict([('Service', ['lambda.amazonaws.com'])])), ('Action', ['sts:AssumeRole'])])])]))]))])), ('lambdaexecutionpolicy', ODict([('DependsOn', ['LambdaExecutionRole']), ('Type', 'AWS::IAM::Policy'), ('Properties', ODict([('PolicyName', 'lambda-execution-policy'), ('Roles', [ODict([('Ref', 'LambdaExecutionRole')])]), ('PolicyDocument', ODict([('Version', '2012-10-17'), ('Statement', [ODict([('Effect', 'Allow'), ('Action', ['logs:CreateLogGroup', 'logs:CreateLogStream', 'logs:PutLogEvents']), ('Resource', ODict([('Fn::Sub', ['arn:aws:logs:${region}:${account}:log-group:/aws/lambda/${lambda}:log-stream:*', ODict([('region', ODict([('Ref', 'AWS::Region')])), ('account', ODict([('Ref', 'AWS::AccountId')])), ('lambda', ODict([('Ref', 'LambdaFunction')]))])])]))])])]))]))])), ('AmplifyResourcesPolicy', ODict([('DependsOn', ['LambdaExecutionRole']), ('Type', 'AWS::IAM::Policy'), ('Properties', ODict([('PolicyName', 'amplify-lambda-execution-policy'), ('Roles', [ODict([('Ref', 'LambdaExecutionRole')])]), ('PolicyDocument', ODict([('Version', '2012-10-17'), ('Statement', [ODict([('Effect', 'Allow'), ('Action', ['s3:GetObject', 's3:ListBucket']), ('Resource', [ODict([('Fn::Join', ['', ['arn:aws:s3:::', ODict([('Ref', 'storagexBucketName')]), '/*']])]), ODict([('Fn::Join', ['', ['arn:aws:s3:::', ODict([('Ref', 'storagexBucketName')])]])])])])])]))]))]))])), ('Outputs', ODict([('Name', ODict([('Value', ODict([('Ref', 'LambdaFunction')]))])), ('Arn', ODict([('Value', ODict([('Fn::GetAtt', ['LambdaFunction', 'Arn'])]))])), ('Region', ODict([('Value', ODict([('Ref', 'AWS::Region')]))])), ('LambdaExecutionRole', ODict([('Value', ODict([('Ref', 'LambdaExecutionRole')]))]))]))])
Converting to YAML
raw_markup---------------------
AWSTemplateFormatVersion: '2010-09-09'
Description: Lambda Function resource stack creation using Amplify CLI
Parameters:
  CloudWatchRule:
    Type: String
    Default: NONE
    Description: ' Schedule Expression'
  env:
    Type: String
  storagexBucketName:
    Type: String
    Default: storagexBucketName
  deploymentBucketName:
    Type: String
  s3Key:
    Type: String
Conditions:
  ShouldNotCreateEnvResources: !Equals
    - !Ref 'env'
    - NONE
Resources:
  LambdaFunction:
    Type: AWS::Lambda::Function
    Metadata:
      aws:asset:path: ./src
      aws:asset:property: Code
    Properties:
      Handler: index.handler
      FunctionName: !If
        - ShouldNotCreateEnvResources
        - YLambdaResolver
        - !Join
          - ''
          - - YLambdaResolver
            - '-'
            - !Ref 'env'
      Environment:
        Variables:
          ENV: !Ref 'env'
          REGION: !Ref 'AWS::Region'
          STORAGE_X_BUCKETNAME: !Ref 'storagexBucketName'
      Role: !GetAtt 'LambdaExecutionRole.Arn'
      Runtime: python3.8
      Layers: []
      Timeout: '25'
      Code:
        S3Bucket: !Ref 'deploymentBucketName'
        S3Key: !Ref 's3Key'
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !If
        - ShouldNotCreateEnvResources
        - xpocLambdaRolec8
        - !Join
          - ''
          - - xpocLambdaRolec8
            - '-'
            - !Ref 'env'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
  lambdaexecutionpolicy:
    DependsOn:
      - LambdaExecutionRole
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: lambda-execution-policy
      Roles:
        - !Ref 'LambdaExecutionRole'
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - logs:CreateLogGroup
              - logs:CreateLogStream
              - logs:PutLogEvents
            Resource: !Sub
              - arn:aws:logs:${region}:${account}:log-group:/aws/lambda/${lambda}:log-stream:*
              - region: !Ref 'AWS::Region'
                account: !Ref 'AWS::AccountId'
                lambda: !Ref 'LambdaFunction'
  AmplifyResourcesPolicy:
    DependsOn:
      - LambdaExecutionRole
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: amplify-lambda-execution-policy
      Roles:
        - !Ref 'LambdaExecutionRole'
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - s3:GetObject
              - s3:ListBucket
            Resource:
              - !Join
                - ''
                - - 'arn:aws:s3:::'
                  - !Ref 'storagexBucketName'
                  - /*
              - !Join
                - ''
                - - 'arn:aws:s3:::'
                  - !Ref 'storagexBucketName'
Outputs:
  Name:
    Value: !Ref 'LambdaFunction'
  Arn:
    Value: !GetAtt 'LambdaFunction.Arn'
  Region:
    Value: !Ref 'AWS::Region'
  LambdaExecutionRole:
    Value: !Ref 'LambdaExecutionRole'

Could you provide more information about your environment?

Which Python version are you using?
Which aws-cfn-template-flip are you using?
Can you share the error message that you are receiving

Answer 4 · 2021-04-27T19:22:27.000Z

@koiker for the Amplify Lambda Resolver functions in my project, the resultant CF template is rejected with this error during the amplify publish:

Found whitespace in your key name (use quotes to include) at line 19,6 >>>    - !Ref 'env'
    ...
An error occurred during the push operation: Found whitespace in your key name (use quotes to include) at line 19,6 >>>    - !Ref 'env'
    ...

I originally found this comment aws-amplify/amplify-cli#5286 and added my own, which led to the creation of aws-amplify/amplify-cli#7138, where i was asked to create an issue here in awslabs/aws-cfn-template-flip

Answer 5 · 2021-04-27T19:33:42.000Z

Am running on a recent version of python3 and the install line is pip install cfn_flip so probably getting the latest release

Answer 6 · 2021-05-20T20:08:53.000Z

@koiker is this sufficient information? This is all happening in an up to date Amplify env

Answer 7 · 2021-05-20T21:56:44.000Z

Hi Paolo,

I'm trying to replicate your issue. Getting just the template that you shared don't generate any error when I convert.
Can you provided detailed step by step from the amplify init to the step that you run the conversion with cfn-flip and the amplify push where the error occur?

Answer 8 · 2021-05-26T16:29:17.000Z

Sure thing, here are the steps: (I've left out the auth step because i don't think it is necessary to reproduce, but we do use Cognito Groups auth on the lambda resolver)

amplify init
amplify add api
Choose GraphQL
amplify add function
Choose python lambda resolver
wire the function to the api with @function
amplify push
Now you will have the cloudformation templates in /amplify/backend/function//cftemplate.json

My script runs during the CI pipeline, in this order:

Script modifies the CF template using approach outlined in OP
amplify publish --yes

We already have this working fine for modification to the auth package CF template, but there is something unique to the function package CF template, which user pashka4281 also encountered in aws-amplify/amplify-cli#5286

Answer 9 · 2021-05-27T18:15:49.000Z

Thanks for the answer Paolo,

I will try to replicate your environment and get the error.

Answer 10 · 2021-06-02T03:42:18.000Z

Hi Paolo,

I can reproduce the error and now I'm looking into the issue.
I will get back soon.

Answer 11 · 2021-06-02T14:50:36.000Z

Hi Paolo,

I found the issue.
When you run the amplify push the code will load the Cloudformation template but using a json reader. No matter if the file extension is .yaml or .json
The issue in the code: https://github.com/aws-amplify/amplify-cli/blob/78854ebd4a3d41d34d68736d6556045302101265/packages/amplify-provider-awscloudformation/src/push-resources.ts#L573 where the cfnMeta is populated with a readJson method instead of running a check first to choose between json or yaml.

Please, report back to the Amplify project that they need change the amplify CLI to check first if the template is json or yaml and then load the template.