elb-instance-reboot

An AWS lambda function that reboots ELB instances that are out of service. The function can be automatically triggered by a CloudWatch alarm via an SNS topic message.

Prerequisites

AWS account with access to: IAM, Lambda, SNS, CloudWatch, ELB, and EC2

The following installed on your local machine (for deployment):

Node (with NPM) Installed
Claudia NPM package npm install -g claudia
AWS CLI Installed
AWS CLI credentials configured Just run aws configure

Deployment and Configuration

To deploy and configure the lambda function, run the following commands from the top-level project directory.

Step 1: Deploy the lambda function

Substitute the region of your elb for {region}, e.g. us-west-2

(Note: AWS calls often take a few seconds to return. So we need to bump the lambda function timeout to 60 seconds.)

claudia create --handler elb-instance-reboot.handler --policies policies --timeout 60 --region {region}

Step 2: Create an SNS Topic

aws sns create-topic --name elb-unhealthy-instances

Step 3: Subscribe the lambda function to the SNS Topic

Use the TopicArn value returned from the last step for {topic}. It will be in the form arn:aws:sns:{region}:{owner id}:elb-unhealthy-instances

claudia add-sns-event-source --topic {topic}

Step 4: Configure a CloudWatch alarm

The following command will create a CloudWatch alarm that is triggered when one or more EC2 instances are out of service for more than 1 minute. You can alter the values for threshold, period, and evaluation-periods to suit your needs.

Substitute the name of the load balancer you want to monitor for {elb name}. Substitute the TopicArn value from step 3 for {topic}.

aws cloudwatch put-metric-alarm --alarm-name unhealthy-instances \
 --metric-name UnHealthyHostCount \
 --namespace AWS/ELB \
 --statistic Minimum \
 --period 60 \
 --threshold 1 \
 --comparison-operator GreaterThanOrEqualToThreshold \
 --evaluation-periods 1 \
 --alarm-actions {topic} \
 --dimensions "Name=LoadBalancerName,Value={elb name}"

If you have multiple load balancers that you want to monitor, set a separate alarm for each one. These alarms can all point to the same SNS Topic and utilize the same lambda function.

Test

You can confirm that the service is working by manually taking one of your EC2 instances out of service, e.g. by logging in and killing the service that responds to the healthcheck endpoint.

Monitoring

You can subscribe to the SNS topic via email to be notified when the alarm is triggered.

To see the results of any executions of the lambda function, look at the /aws/lambda/elb-instance-reboot log group under Cloudwatch Logs.

jcowley/elb-instance-reboot