Add vertical cluster autoscaler functional testing to the pipeline

[ncheneweth]

As a Engineering Platform Labs PSK product team engineer
I want the lab-plaform-eks-core-services pipeline to include cluster-autoscaler functional testing
So that with each deployment and with the nightly conformance tests, functional health of vertical autoscaling is confirmed to be healthy

Enhancement

The current cluster-autoscaler deployment is only tested for service health status within kubernetes. This is an insufficient testing level. Additionally, the expected behavior of the service should be tested by triggering one or more events expected to cause the managed node group to add additional nodes.

Acceptance Criteria

• The post-deployment functional and integration tests within the lab-platform-eks-core-services pipeline will include cluster-autoscaler functional test that demonstrate the autoscaler performs as expected within the cluster.

• These tests should run both during the change-triggered run of the pipeline, as well as during the nightly (recurring) desired-state testing.

Technical notes

A common approach for this functional test is to query the cluster for available pod capacity, then deploy the hello-restful app with sufficient replicas and resource requests that a scale up within the managed node group ASG occurs. Then remove the deployment in order to observe a scale down.