OVERVIEW
We use a CloudFormation template to help track all the moving parts
(and clean them up when we're done). The moving parts include:
The lease:
* A CloudWatch metric, specific to each annex, whose namespace includes
the stack's name.
* A CloudWatch alarm, specific to each annex, which triggers when the
custom metric becomes 0 or misses "enough" updates.
* An SNS topic, specific to each annex, which ties to the preceding trigger
to the following Lambda function.
* A Lambda function, the same for all annexes, triggered by the alarm. The
Lambda function checks if the alarm was properly triggered (to work
around an AWS bug where the alarm is evaluated on an update before the
update is evaluated): if so, it deletes the stack, otherwise it
resets the alarm.
The instances:
* One or more AutoScaling Groups. (Up to eight, given the current
limitations of the template engine.)
* One LaunchConfiguration per AutoScaling Group, each created in its
own substack (the substack is only used as a template engine).
* A Security Group...
Some plumbing:
* A Lambda function, the same for all annexes, invoked as a custom
resource, that adjusts the permissions on the previous Lambda function
so that the SNS topic can invoke it when the alarm triggers and calls it.
* A Lambda function, the same for all annexes, invoked as a custom
resource, that inserts a data point into the lease metric. (The alarm
will otherwise never leave its initial INSUFFICIENT (data) state.)
* Three IAM roles, one for each Lambda function, each the same for all
annexes.
* An IAM Role and IAM Role Profile to grant instances permission to
read from the private S3 bucket we created to share security tokens.
LAUNCH CONFIGURATION
Amazon does some things that make it very difficult to to automatically
generate launch configurations:
* Some instance types don't exist in some availability zones.
* Some instance types may require VPC. You'd think we could work around this,
but we need subnet IDs... and a subnet is availability-zone specific. Of
course, we don't know the names of the availability zones -- or even the
count -- since they vary from region to region. We could create mappings,
but how do we create loops? The subnet resource doesn't allow a list in the
availability zone parameter. Could we create another custom resource whose
Lambda function creates a subnet in each AZ for the region it runs in? Even
if we could, how would we create the launch group(s)?
* AMIs vary by availability zone. This can be worked around using mappings --
see Amazon's standard example templates -- but it makes everything harder.
In the glorious future, when we have our own AMIs, we can use the Amazon trick
to choose one, but it's wildly unclear how to deal with the VPC subnet
requirements and launch group variability.
[Update: what we do now is list all selected VPC subnets in each launch
configuration and let Amazon figure it out. This works as far as it goes,
but of course requires VPC. (Do any instance types require EC2 Classic?)
This doesn't help with the problem of some instance types not being available
in some AZs, but since VPC subnets are tied to AZs, each AutoScaling Group
is at least "trying" to launch its instance types in each AZ.]
DEPENDENCIES
Note that the "DependsOn" attributes exist to force the LeaseFunctionRole
to be deleted last, because once it's deleted, CloudFormation no longer has
the permissions to delete anything else.
FIXME
[] We need to look into which users on the machine (current set up has
user nobody running jobs) can act as the role associated with the machine,
e.g., download the credentials from S3.
[] If condor_annex fails to create the stack, it should check once more if
the stack already exists (to help catch simultaneous uses).
FEATURE REQUESTS
* When creating an AWS::SNS::Topic resource with a Lamba-protocol subscription,
automatically add permissions on the referenced function to allow that topic
to Invoke([.*]?) it.
* The Cloud Formation web console should check parameter validity earlier
and write out constraint failure messages next to the corresponding parameter.
* Add an intrinsic function to refer to arbitrary parameters in other resources.
* An AWS::Lambda::Function should be able to specify the duration of its logs.
It's really hard to do this right now, because the log won't actually exist
until the Lambda function runs, which could happen at any time. This may
actually be a feature request for CloudWatch -- I'd like to set the default
retention policy for a log group based on its name (and/or source).
* Delete generated subscriptions with the rest of the stack.
* Access to the ARN of a security group. (Using Fn::Join is awkward.)
* Deletion order dependency. In particular, if I create an IAM role for a
Lambda function, it has to be the last thing deleted, because otherwise
CloudFormation loses the privileges to finish the deletion. However, that
IAM role should have its resources restricted based on resources created
for the stack...
- I can't grant an instance permission to detach only itself, or even any
instance only from its own autoscaling group. :(
* [2016-02-18] CloudFormation doesn't (yet) allow per-instance type bids when
creating a Spot Fleet.
* [2016-02-18] It would be nice to be able to create Spot Fleets with an
initial size of 0.
* [2016-02-18] It would be really nice to be able to specify defaults with
functions (Fn::Join, Fn::Ref, and Fn::FindInMap in particular). This
would be doubly nice with a function to look up arbitrary values from
any resource in the template (see above).
* [2016-02-18] foreach() would be nice for use with comma-separated lists.
* [2016-02-18] It would be nice if AWS::NoValue disappeared from inside
object lists, instead of giving Spot Fleet fits.
* [2016-02-18] Some of these issues may be better resolved by allowing me
to add (existing) resources to a stack, rather than predeclare them all.
* [2016-02-18] Being able to specify a periodic timer (CloudWatch Event)
in CloudFormation would be really nice.
? [2016-02-18] Specify propogated user tag(s) for the stack. (Currently
not propogated to EBS volumes.) Unclear how or if possible to tag the
whole stack.
* [2016-03-11] In particular, it looks like a tagged autoScaling Group
or Spot Fleet that propogates tags to its instances doesn't tag the
Spot instance requests by which it acquires those instances.
NOTES
* I could possibly use code to create the log group each Lambda function
will by convention create, and set the retention policy there?
* I would like to narrow the resource permissions granted to the IAM roles,
but some actions don't support that, and others cause circular dependency
problems -- see above.