dynamodb-lambda-autoscale

Autoscale AWS DynamoDB using an AWS Lambda function

5 minute setup process
Serverless design
Flexible code over configuration style
Autoscale table and global secondary indexes
Autoscale multiple tables
Optimised performance using concurrent queries
Statistics via 'measured'
AWS credential configuration via 'dotenv'
Optimised lambda package via 'webpack'
ES7 code
100% Flow static type checking coverage

Disclaimer

Any reliance you place on dynamodb-lambda-autoscale is strictly at your own risk.

In no event will we be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this code.

Getting started

Note: dynamodb-lambda-autoscale uses Flow extensively for static type checking, we highly recommend you use Nuclide when making modification to code / configuration. Please see the respective websites for advantages / reasons.

Build and package the code
Fork the repo
Clone your fork
Create a new file in the root folder called 'config.env.production'
Put your AWS credentials into the file in the following format, only if you want to run a local test (not needed for lambda)

```javascript
AWS_ACCESS_KEY_ID="###################"
AWS_SECRET_ACCESS_KEY="###############"
```

Update Region.json to match the region of your DynamoDB instance
Run 'npm install'
Run 'npm run build'
Verify this has created a 'dist.zip' file
Optionally, run a local test by running 'npm run start'

Running on AWS Lambda

Follow the steps in 'Running locally'
Create an AWS Policy and Role
Create a policy called 'DynamoDBLambdaAutoscale'

Use the following content to give access to dynamoDB, cloudwatch and lambda logging

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "dynamodb:ListTables",
        "dynamodb:DescribeTable",
        "dynamodb:UpdateTable",
        "cloudwatch:GetMetricStatistics",
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Create a role called 'DynamoDBLambdaAutoscale'
Attach the newly created policy to the role
Create a AWS Lambda function
Skip the pre defined functions step
Set the name to 'DynamoDBLambdaAutoscale'
Set the runtime to 'Node.js 4.3'
Select upload a zip file and select 'dist.zip' which you created earlier
Set the handler to 'index.handler'
Set the Role to 'DynamoDBLambdaAutoscale'
Set the Memory to the lowest value initially but test different values at a later date to see how it affects performance
Set the Timeout to approximately 5 seconds (higher or lower depending on the amount of tables you have and the selected memory setting)
Once the function is created, attach a 'scheduled event' event source and make it run every minute. Event Sources > Add Event Source > Event Type = Cloudwatch Events - Schedule. Set the name to 'DynamoDBLambdaAutoscale' and the schedule expression to 'rate(1 minute)'

Configuration

The default setup in the Provisioner.js allows for a quick no touch setup. A breakdown of the configuration behaviour is as follows:

AWS region is set to 'us-east-1' via Region.json configuration
Autoscales all tables and indexes
Autoscaling 'Strategy' settings are defined in DefaultProvisioner.json and are as follows
- Separate 'Read' and 'Write' capacity adjustment strategies
- Separate asymmetric 'Increment' and 'Decrement' capacity adjustment strategies
- Read/Write provisioned capacity increased
  - when capacity utilisation > 90%
  - by 3 units or to 110% of the current consumed capacity, which ever is the greater
  - with hard min/max limits of 1 and 10 respectively
- Read/Write provisioned capacity decreased
  - when capacity utilisation < 30% AND
  - when at least 60 minutes have passed since the last increment AND
  - when at least 60 minutes have passed since the last decrement AND
  - when the adjustment will be at least 5 units AND
  - when we are allowed to utilise 1 of our 4 AWS enforced decrements
  - to the consumed throughput value
  - with hard min/max limits of 1 and 10 respectively

Strategy Settings

The strategy settings described above uses a simple schema which applies to both Read/Write and to both the Increment/Decrement. Using the options below many different strategies can be constructed:

ReadCapacity.Min : (Optional) Define a minimum allowed capacity, otherwise 1
ReadCapacity.Max : (Optional) Define a maximum allowed capacity, otherwise unlimited
ReadCapacity.Increment : (Optional) Defined an increment strategy
ReadCapacity.Increment.When : (Required) Define when capacity should be incremented
ReadCapacity.Increment.When.UtilisationIsAbovePercent : (Optional) Define a percentage utilisation upper threshold at which capacity is subject to recalculation
ReadCapacity.Increment.When.UtilisationIsBelowPercent : (Optional) Define a percentage utilisation lower threshold at which capacity is subject to recalculation, possible but non sensical for increments however.
ReadCapacity.Increment.When.AfterLastIncrementMinutes : (Optional) Define a grace period based off the previous increment in which capacity adjustments should not occur
ReadCapacity.Increment.When.AfterLastDecrementMinutes : (Optional) Define a grace period based off the previous decrement in which capacity adjustments should not occur
ReadCapacity.Increment.When.UnitAdjustmentGreaterThan : (Optional) Define a minimum unit adjustment so that only capacity adjustments of a certain size are allowed
ReadCapacity.Increment.By : (Optional) Define a 'relative' value to change the capacity by
ReadCapacity.Increment.By.ConsumedPercent : (Optional) Define a 'relative' percentage adjustment based on the current ConsumedCapacity
ReadCapacity.Increment.By.ProvisionedPercent : (Optional) Define a 'relative' percentage adjustment based on the current ProvisionedCapacity
ReadCapacity.Increment.By.Units : (Optional) Define a 'relative' unit adjustment
ReadCapacity.Increment.To : (Optional) Define an 'absolute' value to change the capacity to
ReadCapacity.Increment.To.ConsumedPercent : (Optional) Define an 'absolute' percentage adjustment based on the current ConsumedCapacity
ReadCapacity.Increment.To.ProvisionedPercent : (Optional) Define an 'absolute' percentage adjustment based on the current ProvisionedCapacity
ReadCapacity.Increment.To.Units : (Optional) Define an 'absolute' unit adjustment

A sample of the strategy setting json is...

{
  "ReadCapacity": {
    "Min": 1,
    "Max": 10,
    "Increment": {
      "When": {
        "UtilisationIsAbovePercent": 90
      },
      "By": {
        "Units": 3
      },
      "To": {
        "ConsumedPercent": 110
      }
    },
    "Decrement": {
      "When": {
        "UtilisationIsBelowPercent": 30,
        "AfterLastIncrementMinutes": 60,
        "AfterLastDecrementMinutes": 60,
        "UnitAdjustmentGreaterThan": 5
      },
      "To": {
        "ConsumedPercent": 100
      }
    }
  },
  "WriteCapacity": {
    "Min": 1,
    "Max": 10,
    "Increment": {
      "When": {
        "UtilisationIsAbovePercent": 90
      },
      "By": {
        "Units": 3
      },
      "To": {
        "ConsumedPercent": 110
      }
    },
    "Decrement": {
      "When": {
        "UtilisationIsBelowPercent": 30,
        "AfterLastIncrementMinutes": 60,
        "AfterLastDecrementMinutes": 60,
        "UnitAdjustmentGreaterThan": 5
      },
      "To": {
        "ConsumedPercent": 100
      }
    }
  }
}

Advanced Configuration

This project takes a 'React' style code first approach over declarative configuration traditionally used by other autoscaling community projects. Rather than being limited to a structured configuration file or even the 'strategy' settings above you have the option to extend the ProvisionerBase.js abstract base class for yourself and programmatically implement any desired logic.

The following three functions are all that is required to complete the provisioning functionality.
As per the 'React' style, only actual updates to the ProvisionedCapacity will be sent to AWS.

getDynamoDBRegion(): string {
  // Return the AWS region as a string
}

async getTableNamesAsync(): Promise<string[]> {
  // Return the table names to apply autoscaling to as a string array promise
}

async getTableUpdateAsync(
  tableDescription: TableDescription,
  tableConsumedCapacityDescription: TableConsumedCapacityDescription):
  Promise<?UpdateTableRequest> {
  // Given an AWS DynamoDB TableDescription and AWS CloudWatch ConsumedCapacity metrics
  // return an AWS DynamoDB UpdateTable request
}

DescribeTable.ResponseSyntax UpdateTable.RequestSyntax

Flexibility is great, but implementing all the logic required for a robust autoscaling strategy isn't something everyone wants to do. Hence, the default 'Provisioner' builds upon the base class in a layered approach. The layers are as follows:

Provisioner.js concrete implementation which provides very robust autoscaling logic which can be manipulated with a 'strategy' settings json object
ProvisionerConfigurableBase.js abstract base class which breaks out the 'getTableUpdateAsync' function into more manageable abstract methods
ProvisionerBase.js the root abstract base class which defines the minimum contract

Rate Limited Decrement

AWS only allows 4 table decrements in a calendar day. To account for this we have an included an algorithm which segments the remaining time to midnight by the amount of decrements we have left. This logic allows us to utilise each 4 decrements as efficiently as possible. The increments on the other hand are unlimited, so the algorithm follows a unique 'sawtooth' profile, dropping the provisioned capacity all the way down to the consumed throughput rather than gradually. Please see RateLimitedDecrement.js for full implementation.

Capacity Calculation

As well as implementing the correct Provisioning logic it is also important to calculate the ConsumedCapacity for the current point in time. We have provided a default algorithm in CapacityCalculator.js which should be good enough for most purposes but it could be swapped out with perhaps an improved version. The newer version could potentially take a series of data points and plot a linear regression line through them for example.

Dependencies

This project has the following main dependencies (n.b. all third party dependencies are compiled into a single javascript file before being zipped and uploaded to lambda):

aws-sdk - Access to AWS services
dotenv - Environment variable configuration useful for lambda
measured - Statistics gathering

Licensing

The source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.

root360/dynamodb-lambda-autoscale