quiver/blog-sam-eventbridge-pipes-kinesis-lambda

AWS Kinesis Data Stream to AWS Lambda using EventBridge Pipes

This pattern will connect Kinesis Data Streams to AWS Lambda consumer with EventBridge Pipes.

Learn more about this pattern at the following blog: How to Connect Kinesis to Lambda using EventBridge Pipes

Requirements

• Create an AWS account if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
• AWS CLI installed and configured
• Git Installed
• AWS Serverless Application Model (AWS SAM) installed

Deployment Instructions

1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:

   git clone https://github.com/quiver/blog-sam-eventbridge-pipes-kinesis-lambda
   

2. Change directory to the pattern directory:

   cd blog-sam-eventbridge-pipes-kinesis-lambda
   

3. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yml file:

   sam deploy --guided
   

4. During the prompts:

   • Enter a stack name
   • Enter the desired AWS Region
   • Allow SAM CLI to create IAM roles with the required permissions.

   Once you have run sam deploy -guided mode once and saved arguments to a configuration file (samconfig.toml), you can use sam deploy in future to use these defaults.

5. Note the outputs from the SAM deployment process. These contain the resource names and/or ARNs which are used for testing.

Testing

1. Use the following dummy JSON payload to trigger your producer Lambda function:

   $ aws kinesis put-record \
     --stream-name YOUR-STREAM-ARN \
     --cli-binary-format raw-in-base64-out \
     --partition-key bar \
     --data '{ "reqId": "1-2-3-4", "status": 200, "client": "abc"}'
   
   {
       "ShardId": "shardId-000000000000",
       "SequenceNumber": "49640567222783951011097960734367787140450389581107298306"
   }
   

2. Observe the logs of the consumer Lambda function to verify if the data pushed by the producer is received or not.

   

Cleanup

1. For deleting the stack you can use sam delete from SAM CLI -

   sam delete
   

---

This template is heavily inspired by GitHub : aws-samples/serverless-patterns.