WojciechMatuszewski/stockwatch-app

StockWatchApp

Inspired by this video with my own spin.

The main reason for creating this POC were:

• Learning how the new event source mapping filtering works.

• Trying out the Amazon API Gateway integration with AWS Step Functions.

Architecture

The architecture I've built significantly differs from the one presented in the video I was inspired by. This is mostly because there are usually multiple ways to achieve desired result in AWS :)

Instead of watching stock prices, this architecture looks at the BINANCE:BTCUSDT and BINANCE:BTCUSDT pairs.

Deployment

1. cd infra

2. npm run bootstrap

3. npm run deploy

4. Amend the value of the SymbolsAPIKeyParameter SSM parameter. Use the APIKey from https://finnhub.io/.

5. Either manually execute the SymbolsPriceOrchestrator state machine couple of times (with 1-minute interval), or enable the EventBridge scheduled rule.

6. You might want to change the test SNS subscription in SymbolsEventDispatcher.

Please remember that this code is far from production-grade quality. This is just me playing around and learning new concepts.

Learnings

• The property in the Properties block of a given CloudFormation resource does not have only to be of type string. It can also be an array or other structure.

  • Having said that, I think it's better always to pass strings. In Golang, it's easier to unmarshal the structure rather than guess its type.

• The Test tab within the APIGW console is handy. It will print all the logs related to a given integration execution.

  • If you are trying to debug an integration error, this is very helpful.

  • You should set up API Gateway logging, but having the quick and dirty way to check them works as well.

• When creating API Gateway HTTPIntegration, you are forced to hardcode the returned status codes (unless you want to map every possible combination).

  • Check out this StackOverflow answer for more details.

• The VTL in the APIGW is not the same as VTL in AppSync.

  • It looks to me that AppSync extends the VTL language by adding many helpers. API Gateway is also doing that, but the catch is that what is available in AppSync might not be available in API Gateway VTL.

  • There are multiple hints regarding this in the AppSync documentation.

    AWS AppSync defines a set of variables and functions for working with resolver mapping templates. This makes logical operations on data easier with GraphQL.

• The values you specify in the requestOverrides for querystrings have to be strings. If they are not, APIGW will silently reject the override and proceed as if you did not specify it in your VTL template.

  • I hate it when tools "silently fail". It is so much better to "fail in the open".

• The type coercion in API Gateway is weird.

  • If you specify a variable that is a number in quotation marks, it will become a string.

    "$context.requestTimeEpoch" // OK
    "$context.requestTimeEpoch - 60" // "123456 + - 60" // WTF :D

• With the SDK Integrations we can finally use DynamoDB Query operation within the StepFunctions!

  • This is excellent news. You no longer have to write a Lambda function to perform that operation.

• The APIGW StepFunctions optimized integration requires arrays as properties for QueryParameters.

  How can one use the JSONPath and the States.Format with the array requirement?

  "symbol.$": "States.StringToJson(States.Format('[\"{}\"]', $.symbols[0].SK.S))"
  

• The APIGW StepFunctions optimized integration does not support the parameters (Like in Pass state) property. This makes it hard to format the results correctly.

  • If you use resultSelector and resultPath, you will end up with a nested object passed to your next state.

  • One could use Pass state just after the Task to format the results

• The DynamoAttributeValue.fromNumber in StepFunctions is broken!

  • See this issue for more details

• The new event source mappings filtering capabilities are not supported in L2 Lambda CDK constructs.

  • You either have to use the CfnEventSourceMapping or add a manual override as per this blog post.

  • The filtering capabilities are neat! And the supported syntax is the same as EventBridge.

  • You can learn a bit more about the technology behind the feature by reading this blog post.

• The EventBridge PutEvents API is a bit awkward to use with the ReportBatchItemFailures setting (for example, in the context of DynamoDB).

  • The response from PutEvents contains the EventID attribute, but this ID is the internal EventBridge event ID, not the ID of the event you are reading from the source.

  • To monitor failed events, you can either use DLQ or rely on a metric that EventBridge updates.

  • Note that the DLQ is set in the context of a rule and not in the context of a bus. This means that if the PutEvents call fails, you need to handle it separately from the events that are pushed to the DLQ.

• The PutRule API call does not support adding the target.

  • This is very different from how the AWS::Events::Rule CloudFormation resource works. There you can specify the target along with the rule.

• The EventBridge to SNS integration does not support setting the MessageAttributes property.

  • It would be awesome if that support was present. I would not have to resort to using a Lambda function in-between those two services just to set the MessageAttributes property.

• The SNS MessageDeduplicationId parameter is only allowed for FIFO topics. Makes sense.

  • It would be neat to have MessageIdempotencyId or something similar available to us.