Fetching last added Observation via rest API doesn't return the resultset on first call

[archtryingtocode]

Describe the bug
Add a new Observation. FHIR server returns successful insert operation with added record and id. Call API to get all the records within a specified datetime interval. The API on the first call doesn't return the last inserted Observation.

To Reproduce
Steps to reproduce the behavior:

1. Add a new Observation
2. Call API to get the all the Observations with a specified date - e.g. Today
3. API returns results without the last inserted Observation
4. Call API again and this time it returns the last updated Observation

Expected behavior
Calling API to get Observation should return all the records - including the last inserted one. This should happen on the first call to API and not on the second call to API.

Additional context
Could this be due to the type of read consistency thats by default enabled on DynamoDB ?

[sevu1]

@archtryingtocode , thank you for reaching out. The delay that you observe is a cumulative delay between inserting data in DynamoDB, kicking of DynamoDB Streams and finally writing data in OpenSearch index (at this point API will return inserted data).
There are few things that we can suggest:

1. update stage in cdk.json, use e.g. "prod"
2. Review and update elastic search cluster parameters
3. Increase provisionedConcurrency for ddbToEs lambda function
4. Depending on use case - whether this is single item insert or insert in large batches, modify increase/decrease batchSize

Let us know if you need more details

[vcareit]

Thanks for your reply @sevu1

1. I tried using the "prod" as an environment it doesnt help.
2. I am trying to insert single Observation at a time.

I think the issue is related to the DynamoDB and configurations for the reads. How do I enable "strongly consistent reads" on DynamoDB ?

If the API calls cannot result the latest updated records then this must be fixed. It doesn't matter how many processes are run behind the scene, if the an health app cannot display the newly inserted records. If the strongly consistent read doesn't solve the issue then some alternative solution needs to be figured out.

[Zambonilli]

If you're doing a search for your read like you indicated with getting observations by date, then it's like @sevu1 said there's write propagation delay between dynamodb and elastic search. The best solution to date is to run with provisioned concurrency on the ddbToES lambda because you do not have a lambda cold start in the loop. However, you are still going to have an nondeterministic delay measured most likely in 10s of milliseconds and even more depending on the size of the batch being written to elastic.

If you can do a direct GET based on Observation/:id you will be doing all operations against DynamoDB. The default eventual consistency of DynamoDB has been driven low enough that it is measured in single digit milliseconds out of the box. If that still isn't good enough for your needs then you could update your version persistence-ddb to set strong consistency.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html