Different behavior between local invocation with SAM and AWS

[csreddy]

I'm running a SAM project locally. The project code contains Chromium and Puppeteer attached as Lambda layer. When lambda is invoked locally Chromium crashes with below error. However the exact same code runs successfully when deployed to AWS.

Note: I am bundling all essential shared objects and binaries for Chromium to run successfully which I verified in Lambda function.

This is happening in both nodejs18.x and python3.8 runtimes.

Error

[1218/204108.825770:ERROR:nss_util.cc(239)] Error initializing NSS with a persistent database (sql:/opt/fonts/.pki/nssdb): NSS error code: -8023
[1218/204108.825919:ERROR:nss_util.cc(124)] Error initializing NSS without a persistent database: NSS error code: -8023
[1218/204108.825931:FATAL:nss_util.cc(126)] nss_error=-8023, os_error=0

template.yml

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
  Sample SAM Template for testing Synthetics NodeJS canary locally

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  CanaryExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument: 
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns: 
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: CanaryExecutionPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - s3:PutObject
                  - s3:GetObject
                Resource:
                  - arn:aws:s3:::cw-syn-results-*/*
              - Effect: Allow
                Action:
                  - s3:GetBucketLocation
                Resource:
                  - arn:aws:s3:::cw-syn-results-*/*
              - Effect: Allow
                Action:
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                  - logs:CreateLogGroup
                Resource:
                  - arn:aws:logs:*:*:log-group:/aws/lambda/cwsyn-*
              - Effect: Allow
                Action:
                  - s3:ListAllMyBuckets
                  - xray:PutTraceSegments
                Resource:
                  - "*"
              - Effect: Allow
                Resource: "*"
                Action: cloudwatch:PutMetricData
                Condition:
                  StringEquals:
                    cloudwatch:namespace: CloudWatchSynthetics

  #  Canary lambda that runs when invoked with sam local invoke
  NodeJSPuppeteerCanary:
    Type: AWS::Serverless::Function
    Properties:
      Description: Local Synthetics NodeJS Dev Canary
      Handler: cw-synthetics.handler
      Runtime: nodejs18.x
      Architectures:
        - x86_64
      CodeUri: src/
      MemorySize: 3000
      Timeout: 300
      Environment:
        Variables:
          MODE: "Local canary test"
      Layers:
         - arn:aws:lambda:us-west-2:<redacted>:layer:Synthetics:42
      Role: !GetAtt CanaryExecutionRole.Arn

I have verified with old and new SAM CLI versions and all of them have same behavior locally.

• 1.105
• 1.94
• 1.80

Not sure if anything changed in RIE that would behave differently locally

[csreddy]

It appears that the recent images has changes that caused the crash. Resolved the issue by using older nodejs and python images. Not sure if this is expected or if this should be treated as bug in the image.

nodejs: public.ecr.aws/sam/build-nodejs18.x:1.65-x86_64.
python: public.ecr.aws/sam/build-python3.8:1.65.0-x86_64

[valerena]

This was apparently an issue during last December with an update on the Lambda base images aws/aws-lambda-base-images#131

It was addressed in a newer version of Chromium release at that time: Sparticuz/chromium#187, Sparticuz/chromium#229 (comment)