Provisioned Concurrence is not Scaling down/in when there is no traffic/utililzation

[rsshilli]

I've noticed that my provisioned concurrency is scaling up properly in production but it never scales back down at night when there are no users on our site. It seems this is because the alarm is in Insufficient Data state and by default, the provisioned concurrency is set to Treat missing data as missing, so nothing happens.

I've recorded this Loom video to help explain it: https://www.loom.com/share/6030e49b9ea542d2888bf54f54d5e1f7

I also previously opened this stack overflow issue with a lot of details, before I figured out that it's the configuration being output by this plugin (or perhaps the lack of configuration being output), that's the issue.

Here's the place where this configuration needs to be changed:

[claydanford]

Hi @rsshilli, thanks for opening this issue.

Those alarms are auto-generated by AWS when creating the target tracking and scaling policy. The description says, "DO NOT EDIT OR DELETE". It's not intended to be manually edited or deleted.

The fix for this is to go to AWS and ask them to change the default alarm in the autoscaling service, or roll your own alarm (or an additional alarm) to make it scale down. This plugin is designed to only create the necessary resources for autoscaling as a managed service. This plugin does not create alarms.

I hope that is helpful, but sorry it might not be the answer you were looking for.

Kindly,

CRD

[rsshilli]

Oh. I thought this plugin was creating the alarm. Thank you! This is helpful. I guess I need to go read the source code. What is this plugin creating?

Also, if I was to roll my own, is there an example somewhere of how to reference the auto scaling resources?

[claydanford]

@rsshilli You can view this blog, scroll towards the bottom to see some cloudformation. This plugin just generates that cloudformation (AWS::ApplicationAutoScaling::ScalingPolicy, AWS::ApplicationAutoScaling::ScalableTarget). https://aws.amazon.com/blogs/aws/new-provisioned-concurrency-for-lambda-functions/

The easiest thing to recreate this alarm with the proposed change is to add an alarm in the resources section referencing this cloudformation. You can use the plugin alarm as a reference for the values (see console or screenshot above). https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cw-alarm.html

hth,

CRD

[pavan-tw]

@rsshilli what did you end up doing for this? We have a similar problem.

[rsshilli]

Well, I'm glad you asked @pavan-tw. I just realized that we never really fixed this. It's still a ticket in our backlog.