Datadog + Serverless Example Project

This example is adapted from Serverless

This example demonstrates how to setup a RESTful Web Services allowing you to create, list, get, update and delete Todos. DynamoDB is used to store the data. This is just an example and of course you could use any data storage as a backend.

Structure

This service has a separate directory for all the todo operations. For each operation exactly one file exists e.g. todos/delete.py. In each of these files there is exactly one function defined.

The idea behind the todos directory is that in case you want to create a service containing multiple resources e.g. users, notes, comments you could do so in the same service. While this is certainly possible you might consider creating a separate service for each resource. It depends on the use-case and your preference.

Use-cases

API for a Web Application
API for a Mobile Application

Setup

npm install -g serverless
npm install
cp .env.example .env
# edit .env with your Datadog API keys found at https://app.datadoghq.com/account/settings#api

You can also add environment variables specific to your staging / production environments.

Note: The dotenv plugin will inject all environment variables into your Lambda functions by default. This means that you don't need to explicitly define your API keys as environment variables for every function. To override this behavior, you can define a whitelist.

Datadog Lambda Layer

Currently, the Serverless Framework requires that you define Lambda Layers on a per-function basis, like this:

For saner defaults, we've included a plugin that makes Layers available to all functions in a provider by default:

provider:
  name: aws
  runtime: python3.6
  region: us-east-1
  layers:
    - arn:aws:lambda:${self:provider.region, 'us-east-1'}:464622532012:layer:Datadog-Python36-metric:1

X-Ray tracing

To make configuration even easier, this template project uses serverless-plugin-tracing to enable X-Ray tracing by default for all of your Lambda functions. In the AWS Console, this is the "Enable active tracing" option under Advanced Settings when viewing a Lambda function.

To disable this per-function:

functions:
  mainFunction: # inherits tracing settings from "provider"
    handler: src/app/index.handler
  aBoringFunction:
    tracing: false # overrides provider settings (opt out)

Also remember to install and import AWS' X-Ray client libraries for full tracing:

Python

from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all

Node

const awsXRay = require('aws-xray-sdk');
const awsSdk = awsXRay.captureAWS(require('aws-sdk'));

Deploy

In order to deploy the endpoint simply run

serverless deploy  # or sls deploy

The expected result should be similar to:

Serverless: Packaging service…
Serverless: Uploading CloudFormation file to S3…
Serverless: Uploading service .zip file to S3…
Serverless: Updating Stack…
Serverless: Checking Stack update progress…
Serverless: Stack update finished…

Service Information
service: serverless-rest-api-with-dynamodb
stage: dev
region: us-east-1
api keys:
  None
endpoints:
  POST - https://45wf34z5yf.execute-api.us-east-1.amazonaws.com/dev/todos
  GET - https://45wf34z5yf.execute-api.us-east-1.amazonaws.com/dev/todos
  GET - https://45wf34z5yf.execute-api.us-east-1.amazonaws.com/dev/todos/{id}
  PUT - https://45wf34z5yf.execute-api.us-east-1.amazonaws.com/dev/todos/{id}
  DELETE - https://45wf34z5yf.execute-api.us-east-1.amazonaws.com/dev/todos/{id}
functions:
  serverless-rest-api-with-dynamodb-dev-update: arn:aws:lambda:us-east-1:488110005556:function:serverless-rest-api-with-dynamodb-dev-update
  serverless-rest-api-with-dynamodb-dev-get: arn:aws:lambda:us-east-1:488110005556:function:serverless-rest-api-with-dynamodb-dev-get
  serverless-rest-api-with-dynamodb-dev-list: arn:aws:lambda:us-east-1:488110005556:function:serverless-rest-api-with-dynamodb-dev-list
  serverless-rest-api-with-dynamodb-dev-create: arn:aws:lambda:us-east-1:488110005556:function:serverless-rest-api-with-dynamodb-dev-create
  serverless-rest-api-with-dynamodb-dev-delete: arn:aws:lambda:us-east-1:488110005556:function:serverless-rest-api-with-dynamodb-dev-delete

Usage

You can create, retrieve, update, or delete todos with the following commands:

Create a Todo

curl -X POST https://XXXXXXX.execute-api.us-east-1.amazonaws.com/dev/todos --data '{ "text": "Learn Serverless" }'

No output

List all Todos

curl https://XXXXXXX.execute-api.us-east-1.amazonaws.com/dev/todos

Example output:

[{"text":"Deploy my first service","id":"ac90feaa11e6-9ede-afdfa051af86","checked":true,"updatedAt":1479139961304},{"text":"Learn Serverless","id":"206793aa11e6-9ede-afdfa051af86","createdAt":1479139943241,"checked":false,"updatedAt":1479139943241}]%

Get one Todo

# Replace the <id> part with a real id from your todos table
curl https://XXXXXXX.execute-api.us-east-1.amazonaws.com/dev/todos/<id>

Example Result:

{"text":"Learn Serverless","id":"ee6490d0-aa11e6-9ede-afdfa051af86","createdAt":1479138570824,"checked":false,"updatedAt":1479138570824}%

Update a Todo

# Replace the <id> part with a real id from your todos table
curl -X PUT https://XXXXXXX.execute-api.us-east-1.amazonaws.com/dev/todos/<id> --data '{ "text": "Learn Serverless", "checked": true }'

Example Result:

{"text":"Learn Serverless","id":"ee6490d0-aa11e6-9ede-afdfa051af86","createdAt":1479138570824,"checked":true,"updatedAt":1479138570824}%

Delete a Todo

# Replace the <id> part with a real id from your todos table
curl -X DELETE https://XXXXXXX.execute-api.us-east-1.amazonaws.com/dev/todos/<id>

No output

Scaling

AWS Lambda

By default, AWS Lambda limits the total concurrent executions across all functions within a given region to 100. The default limit is a safety limit that protects you from costs due to potential runaway or recursive functions during initial development and testing. To increase this limit above the default, follow the steps in To request a limit increase for concurrent executions.

DynamoDB

When you create a table, you specify how much provisioned throughput capacity you want to reserve for reads and writes. DynamoDB will reserve the necessary resources to meet your throughput needs while ensuring consistent, low-latency performance. You can change the provisioned throughput and increasing or decreasing capacity as needed.

This is can be done via settings in the serverless.yml.

  ProvisionedThroughput:
    ReadCapacityUnits: 1
    WriteCapacityUnits: 1

In case you expect a lot of traffic fluctuation we recommend to checkout this guide on how to auto scale DynamoDB https://aws.amazon.com/blogs/aws/auto-scale-dynamodb-with-dynamic-dynamodb/

Misc

From StackOverflow, update the Python dependencies with:

pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U