Manage Amazon Elastic Compute Cloud (Amazon EC2) spot instances using webtask.io
This is a work in progress. The quick instructions are below. Developed with Node version 5.x. Not yet on Windows. Webpack, and the wt-cli babel option make it possible to run as a webtask.io container
git clone https://github.com/glennschler/spotspec-wt.git
npm install
As a proof of concept, create a webtask which launches a given machine instance type in a given region. This will show that AWS Identity and Access Management (IAM) user credentials can be securely stored for use in a webtask. Once the webtask is proven and understood through these steps, a larger goal to fully manage EC2 spot instance will be possible.
- Webpack the
./lib/wt-spotter.js
to a single file which is required by webtask.io. The resultingbuild/wt-spotter-packed.js
is file to be used when calling wt-create.
npm run webpack
- NPM install command has already installed all dependencies locally. Now initialize wt-cli, if it has not been done in the past.
- There are detailed instructions at https://webtask.io/cli.
npm run wt-init
- Create an EC2 IAM user following this aws guide. Here are the quick steps:
-
Sign in to the AWS Management Console and open the IAM console.
-
In the navigation pane, choose Users, and then choose Create New Users.
-
Enter a user name. Check Generate an access key, and choose Create.
-
Once created, choose Show User Security Credentials. Save the credentials for the webtask. You will not have access to this secret access key again after you close.
-
Attach a policy to limit the user permissions to specific AWS resources. For more information, see Attaching Managed Policies. Assign a policy which only allows the spot request action. The spotspec README shows a good policy. Here is a much shorter example:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [ "ec2:RequestSpotInstances" ],
"Effect": "Allow",
"Resource": "*",
"Condition": {
"Bool": { "aws:SecureTransport": "true" },
"StringEquals": {
"ec2:Region": [ "us-east-1", "us-west-2" ]
}
}
}]
}
- To create a webtask token, the webtask.io CLI command
wt create
uploads code along with the EC2 credentials. Both are encrypted and stored. This create command returns a url which represents the new webtask token. Even though the code and secrets are cryptographically protected, the created webtask token url still needs be well protected.
- A well formed EC2 IAM policy, such as specified above, is an additional level of protection to restrict access.
- The code uploaded in this example is part of this repository and it's dependencies, so is publicly available for your review. Understand that any code that is used to create a webtask token needs to be trusted with your IAM credentials.
# Set to where the webtask code exists. This is a file in this repository
export WT_GITHUB=.
# The file which was created with the `npm run webpack` command
export WT_CODE=$WT_GITHUB/build/wt-spotter-packed.js
- In this JSON string replace the enclosed {secret} in both the accessKeyId and secretAccessKey with the real IAM user's credentials. Plus the optional aws MFA token device serial numbers
# Prepare base64 encoded cloud-init user data to launch with the new AWS instances
export WT_USERDATA=$(npm run -s encodeFn -- node_modules/spotspec/test/userDataDockerAWSLinux.txt)
export WT_SECRET=$(npm run -s secretsToJson -- --accessKeyId=<<secret>> \
--secretAccessKey=<<secret>> --serialNumber=<<secret>> \
--userData=$WT_USERDATA)
- Call the webtask.io CLI command
wt create
. - The optional exp=+10 parameter instructs the webtask token to expire in 10 minutes. Only when evaluating these steps.
- The above JSON $WT_SECRET is sent using the wt --secret parameter.
export WT_OPTS='--exp=+10'
# This will do this -> wt create $WT_CODE $WT_OPTS --secret wtData=$WT_SECRET
npm run -s wt-create > wt-create.log
# Echo the previous output to view the created webtask token url
echo wt-create.log
https://webtask.it.auth0.com/api/run/{container}/{jt-name}?webtask_no_cache=1
```
- Now the webtask request is available to execute remotely, and repeatedly. This example shows using the
CURL
command line. Another better example is an IFTTT recipe which makes a similar HTTP request, though instead is triggered by an incoming SMS or email.
- Replace the post data JSON arguments "region", "type", "price", etc... as needed.
- Request the $WT_URL which was created during the previous
wt create
step. - To format the output, optionally pipe the output to a python command as demonstrated here.
# Every minute the token code changes
export WT_TOKEN=123456
# send the LAUNCH task request
curl -s $(cat wt-create.log) --trace-ascii "./out.log" \
-H "Content-Type: application/json" \
-X POST -d \ '{"construct":{"keys":{"region":"us-west-1"},"upgrade":{"tokenCode":"'$WT_TOKEN'"}},"attributes":{"type":"m3.large","dryRun":"false","isLogging":"true","ami":"ami-d5ea86b5","keyName":"yourKeyName","securityGroups":[],"price":"0.0083","task":"launch"}}' \
| python -mjson.tool
{
"code": 400,
"details": "Error: Failed to initialize: {\"message\":\"MultiFactorAuthentication failed with invalid MFA one time pass code......
}
```