serverless-express-typescript-sqs
This project provides a full starter kit for a serverless producer consumer queue setup implemented with AWS Lambda and SQS. The goal is that you can use this code base to add your domain logic on top of it.
- serverless-express-typescript-sqs
- Abstract overview
- Purpose
- Content
- Project structure
- Development guides
- Create a new service or controller
- Add a new http endpoint
- Define message processing logic
- Change Dead Letter Queue behavior
- Change message processing retry interval time
- Add a new serverless function
- Change handling of unexpected errors during endpoint access
- Change validation error response format
- Define environment variables (local)
- Define secrets (AWS Lambda)
- Run locally
- Deploy on AWS
- Known issues
- Contribution
Abstract overview
The project consists out of two serverless functions:
Producer function
- Provides http endpoints that can be called to produce messages
- Producer endpoints will enqueue messages into a queue
Consumer function
- Will be triggered when a message was passed to the queue
- Payload of the functions are enqueued messages
- Each message will be processed (e.g. by sending to another http api)
- Proccessing can lead to two kind of errors:
- Retriable errors
- Non retriable errors
- If processing of a message leads to a retriable error the message will stay in the queue
- If processing of a message leads to a non retriable error the message will be deleted from the queue
- If processing of a message was successful the message will be also deleted from the queue
- After a message caused n times a retriable error it will be moved to a so called Dead Letter Queue
- Message will stay in that queue for a specific time (Maximum 7 days) and will not be processed anymore
Purpose
There are scenarios in which the processing of a message in the consumer could fail unexpectedly. For the example an api could become temporarly unavailable. This queue architecture makes sure that the messages will be kept and processing retried multiple times.
Content
The project was created with TypeScript and Node.js. It contains the following content:
- Serverless setup for AWS Lambda with Serverless framework
- SQS interaction with AWS SDK
- Queue error handling setup with AWS SDK
- Http server setup with Express
- Request validation with Express validator
- Swagger setup with Swagger UI Express
- Unit test setup with Jest
- Formatting setup with Prettier
Project structure
Configuration files
Path | Description |
---|---|
.env.example |
Example environment variable configuration |
.prettierignore |
Files to exclude from formatting |
.prettierrc |
Prettier formatting config |
jest.config.js |
Jest config file for unit tests |
secrets.yml.example |
Example secret configuration to run functions on AWS Lambda |
serverless.offline.yml |
Serverless configuration to run functions locally |
serverless.yml |
Servlerless configuration to run functions on AWS Lambda |
tsconfig.json |
TypeScript configuration file |
/docs
folder
/docs
contains everything that has to do with documentation about the project/docs/api-spec.yml
contains the swagger specification for the http app
/test
folder
/test
contains all unit tests executed with Jest- Has also similar folder structure to
src
env.test
contains environment variable setup for tests
/src
folder
/src
contains the main source code of the project.- Folder structure similar to typical Express projects
Development guides
The following section provides you a short guide on how to change or add functionalities. You can also read the comments in the source code of the files to get more information about the specific implementation.
Create a new service or controller
- Create a new folder with the service or controller name (e.g
/src/services/log
) - Create a file for the class (e.g
/src/services/log/log.service.ts
)- You can use dependency injection to access other services
- Create a file that contains an exported instance of that class (e.g
/src/services/index.ts
).- Here you then can import exported instances of other services or controllers to inject them
- Create a unit test file (e.g
/test/services/log.service.spec.ts
)
Add a new http endpoint
- Create a new router or add a route to an existing router in
/src/routers
- Create a new validator for that route in
/src/validators
and use it as middleware for that route- Do not forget to always add the validation check middleware (
/src/validator.middleware.ts
) right after you used a new validation middleware
- Do not forget to always add the validation check middleware (
- Add a controller as shown in the section before that will be called from the new route
- Add a service as shown in the section before that will called from the new controller
- Register the router in
/src/http-app.ts
- Update the swagger documentation in
/docs/api-spec.yml
Define message processing logic
- Go to file
/src/services/event/event.service.ts
- In the function
processMessage : (message: DequeuedMessage) => Promise<void>
of classEventService
you can define how to process the message - Or you can create also a seperate service for message processing and use it with dependency injection
- Check out the file for more specific information
Change Dead Letter Queue behavior
- Go to file
serverless.yml
into sectionresources.Resources
resources.Resources.MessagesQueue.Properties.RedrivePolicy.maxReceiveCount
represents the number of times a message that caused a retriable error should be retriedresources.Resources.DeadLetterMessagesQueue.Properties.MessageRetentionPeriod
represents the maximum time in seconds a message should be stored in the dead letter queue
Change message processing retry interval time
- Go to file
serverless.yml
into sectionresources.Resources.MessagesQueue.Properties
- After a minimum time of
VisibilityTimeout - consumer execution time
(in seconds) the consumer will pick up a failed message that should be retried again
Add a new serverless function
- Add a new file with the function in
src
(e.g/src/example-function.ts
) - To call a service you can import an exported instance
- Add the serverless function to
functions
section inserverless.yml
Change handling of unexpected errors during endpoint access
- Modify the function in
/src/middlewares/error.middleware.ts
Change validation error response format
- Modify the function in
/src/middlewares/validator.middleware.ts
Define environment variables (local)
- Add the new environment variable to
.env
file (and also to.env.example
as reference) - Add the variable to
/test/.env.test
- Use the variable in the
provider.environment
section inserverless.offline.yml
- Use the variable in
/src/constants/environment.constants.ts
Define secrets (AWS Lambda)
- Follow the steps which are explained in the local section
- Add the variable to
secrets.yml
file for the different stages (and also tosecrets.yml.example
as reference) - Use the variable in the
provider.environment
section inserverless.yml
Run locally
Right now there is no configuration setup to run this project in the exact same way as on AWS.
You can check out LocalStack to simulate AWS services on your local computer.
When running the project locally without LocalStack there is no automatic triggering of serverless functions. So you have to call the functions manually to test them. If you run the project with ENVIRONMENT=local
every interaction with SQS will be realized with a mocked queue service (/services/queue/local-queue.service.ts
). You can change that behavior in /services/queue/index.ts
to always use the real service (/services/queue/sqs-queue.service.ts
).
Install and start project
- Clone the repository
- Move into cloned directory
- Run
npm i
to install all packages - Create a
.env
file based on.env.example
- Run
npm run start:serverless
to start the serverless offline application
Produce message
- Open
http://localhost:3000/local/apispec/
in browser - Use the endpoint
/produce-message
to enqueue a message
Consume messages
- Install the AWS CLI
- Create a payload file that can contains mocked queue messages
{
"Records": [
{
"messageId": "message-id1",
"receiptHandle": "message-handle1",
"body": "{\"payload\":\"message-payload1\"}"
}
]
}
- Run the following command to send the mocked messages to the consumer:
aws lambda invoke \
--output json {absolute output file path} \
--endpoint-url http://localhost:3100 \
--function-name serverless-express-typescript-sqs-local-serverless-consumer \s
--payload file://{relative payload file path} \
--cli-binary-format raw-in-base64-out
Run tests
- Run
npm run test
to run all tests - Run
npm run test:watch
to tests in interactive mode
Format
- Run
npm run format
to format all files with Prettier - Run
npm run format:check
if all files are formatted correctly
Deploy on AWS
- Read about serverless deploying
- Create a
secrects.yml
file based onsecrects.yml.example
- Run
npm run serverless -- deploy --stage={stage to deploy for}
to deploy on AWS
Known issues
- TypeScript compiling on changes is quite slow (Check out this GitHub issue)
Contribution
Feel free to open an issue if you found any error or to create a pull request if want to add additional content.