Feature request: ECS graceful shutdown/instance draining

[itsdalmo]

When running ECS clusters on AWS, a zero-downtime rolling update of the underlying VM's can only be done gracefully by using lifecycle hooks and calling UpdateContainerInstancesState to set the instance state to DRAINING, and then waiting for it to have zero running tasks before completing the lifecycle action.

This pattern is shown here using a Lambda:

• AWS blog post
• Code example from blog

I'm wondering if perhaps this is something that could be handled by lifecycled instead. As always there are some pros and cons for users by doing it this way instead of using a Lambda...

Pros

• Single way of handling scale-in for regular instances and ECS clusters.
• Draining can take longer than 5min (max runtime for Lambda) without needing recursive invocations of the Lambda (or step functions).

Cons

• The instance would need additional permissions, which would give any running tasks the same permissions unless users were careful to disallow it.
• Instances would fail to launch if lifecycled was installed from the Github releases and GH was down. So it makes the ECS instances more "brittle".

If this is something you think belongs in Lifecycled (and it seems like a good practice), I think we could add a new flag --ecs-cluster and implement a new handler (ECSHandler?) which would drain the instance before completing the lifecycle hook. We could probably hardcode it to run before the FileHandler (aka the handler script).

What do you think @lox?

[lox]

Yup, I love that idea!

[lox]

The other thing that would be neat is hibernation support: https://github.com/aws/ec2-hibernate-linux-agent