[Misc] initial request slow

[js-kyle]

Hi there,

We are running FWOA with provisioned concurrency. For each initial request in a series of requests we are seeing, that the first request has a much higher latency than the rest. It seems like FWOA is performing some async initialization on the first request, however adding some logging around this, it seems like that is not the culprit.

Is this behaviour that is currently known/explained? Ideally we would not need to implement synthetic checks on the fhirServer to keep the app 'warm' in addition to the lambda runtime 'warm' via Provisioned concurrency

Example cloudwatch logs for a slow initial request, followed by a faster subsequent request

REPORT RequestId: 89332962-5679-46e3-b2eb-a85d29c22d95	Duration: 3379.19 ms	Billed Duration: 3380 ms	Memory Size: 512 MB	Max Memory Used: 229 MB	Init Duration: 3972.11 ms	
XRAY TraceId: 1-6332510d-592cc6b01a74b9641524d948	SegmentId: 67a37db15d23b38f	Sampled: true	

REPORT RequestId: 540816de-ce02-4b2e-8e48-cf7b40e3d5e4	Duration: 86.91 ms	Billed Duration: 87 ms	Memory Size: 512 MB	Max Memory Used: 229 MB	

Updates:

After some investigation, the chokepoint seems to be the call to the FHIR Validator lambda via aws-sdk const lambdaResponse = await this.lambdaClient.invoke(lambdaParams).promise();. As seen in the XRay trace above, the fhir validator lambda invocation time is ok, but instrumenting the code it seems like the initial request of the aws-sdk Lambda client is delaying this somehow.

This problem will occur for the first FHIR resource which requires validation via the hapi validator, but not on the /metadata endpoint for example.

When adding this.lambdaClient.getAccountSettings(() => {}); to the constructor the issue goes away, but there is a warning printed Error: Missing AWS Lambda trace data for X-Ray . This is an attempt at 'warming up' the sdk (which I know is incorrect usage), pre-request. Is there a better way to do achieve the same result of 'warming up' the sdk?

    constructor(hapiValidatorLambdaArn: string) {
        this.hapiValidatorLambdaArn = hapiValidatorLambdaArn;
        this.lambdaClient = new AWS.Lambda({
            httpOptions: {
                timeout: TIMEOUT_MILLISECONDS,
            },
        });
        // eslint-disable-next-line @typescript-eslint/no-empty-function
        this.lambdaClient.getAccountSettings(() => {});
    }

[ssvegaraju]

Hi,

A few questions about your setup:

1. Which version of FWoA are you running this on? (If >v5.0.0/v3.0.0-smart please specify if using CDK or Serverless)
2. Which Implementation Guides are you running the validator with?

I think this may be something you would want to reach out to the Lambda team about, as I double checked our Xray Traces but wasn't able to find any similar traces with long first-request latency. Their team may have more information on troubleshooting the Lambda warming issue.

[Zambonilli]

Make sure you're pointed at the validator lambda alias and not a specific version of the validator lambda. Only the validator lambda alias has provisioning and the constructor for the HAPI validator takes a long time.

[js-kyle]

Which version of FWoA are you running this on? (If >v5.0.0/v3.0.0-smart please specify if using CDK or Serverless)

5.0.0 deployed via Serverless

Which Implementation Guides are you running the validator with?

The default setting/IG

Make sure you're pointed at the validator lambda alias and not a specific version of the validator lambda. Only the validator lambda alias has provisioning and the constructor for the HAPI validator takes a long time.

Yup, it is using the :provisioned alias for VALIDATOR_LAMBDA_ALIAS

To clarify I do not believe the issue is with the validator lambda itself, as seen in the xray traces it is responding in a reasonable time, but it seems to be the aws-sdk Lambda client causing the delay in it's initialization. I will try to isolate this further

[nisankep]

FHIR Works on AWS has been moved to maintenance mode. While in maintenance, we will not add any new features to this solution. All security issues should be reported directly to AWS Security at [aws-security@amazon.com] (mailto:security@amazon.com). If you are new to this solution, we advise you to explore using [HealthLake] (https://aws.amazon.com/healthlake), which is our managed service for building FHIR based transactional and analytics applications. You can get started by contacting your AWS Account team. If you are an existing customer
of FHIR Works on AWS, and have additional questions or need immediate help, please reach out to fwoa-migration-support@amazon.com or contact your AWS Account team.