Amazon EC2 Spot Instances offer spare compute capacity available in the AWS Cloud at steep discounts compared to On-Demand prices. EC2 can interrupt Spot Instances with two minutes of notification when EC2 needs the capacity back. You can use Spot Instances for various fault-tolerant and flexible applications. Some examples are analytics, containerized workloads, high-performance computing (HPC), stateless web servers, rendering, CI/CD, and other test and development workloads.
One of the best practices to successfully adopt Spot Instances is to implement Spot Instance diversification as part of your configuration. Spot Instance diversification helps to procure capacity from multiple Spot Instance pools, both for scaling up and for replacing Spot Instances that may receive a Spot Instance termination notification. A Spot Instance pool is a set of unused EC2 instances with the same Instance type, operating system and Availability Zone (for example, m5.large on Red Hat Enterprise Linux in us-east-1a).
In this tutorial you will learn how to add Spot Instances to your kOps Kubernetes clusters, while adhering to Spot Instance best practices. This will allow you to run applications without compromising performance or availability. Kubernetes Operations (kOps) is an open source project that provides a cohesive set of tools for provisioning, operating, and deleting Kubernetes clusters in the cloud. As part of the tutorial, you will deploy a kOps Kubernetes deployment and autoscale it on your Spot Instance worker nodes by using Kubernetes Cluster-Autoscaler and Karpenter.
- How to set up and use the kOps CLI to create a Kubernetes cluster with On-Demand nodes
- How to add Instance Groups with Spot Instances to your cluster, automatically leveraging best practices
- How to deploy the AWS Node Termination Handler
- How to deploy the Kubernetes Cluster Autoscaler
- How to deploy a sample application, test that it is running on Spot Instances and that it properly scales
- How to clean up your resources
Step 1: Set up AWS CLI, kOps, and kubectl
In this step you will install all the dependencies that you will need during the tutorial.
-
Install version 2 of the AWS CLI by running the following commands — if you’re using Linux — or follow the instructions in the AWS CLI installation guide for different operating systems.
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" unzip awscliv2.zip sudo ./aws/install
-
kOps requires that you have AWS credentials configured in your environment. The
aws configurecommand is the fastest way to set up your AWS CLI installation for general use. Run the command and follow the prompts. You can use Administrator IAM policy, but if you want to limit the permissions required by kOps, the minimum required IAM privileges you will need are:
- AmazonEC2FullAccess
- AmazonRoute53FullAccess
- AmazonS3FullAccess
- IAMFullAccess
- AmazonVPCFullAccess
- Events:
- DeleteRule
- ListRules
- ListTargetsByRule
- ListTagsForResource
- PutEvents
- PutRule
- PutTargets
- RemoveTargets
- TagResource
- SQS:
- CreateQueue
- DeleteQueue
- GetQueueAtttributes
- ListQueues
- ListQueueTags
-
Install kOps in your environment. You can also follow this guide to install kOps for other architectures and platforms. At the time of writing, the latest version of kOps is v1.23.1
export KOPS_VERSION=v1.23.1 curl -LO https://github.com/kubernetes/kops/releases/download/${KOPS_VERSION}/kops-linux-amd64 chmod +x kops-linux-amd64 sudo mv kops-linux-amd64 /usr/local/bin/kops kops version
-
Install Kubectl. You can also follow this guide for other architectures and platforms. You should use the same major kubectl version as the kOps version selected.
export KUBECTL_VERSION=v1.23.6 sudo curl --silent --location -o /usr/local/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl sudo chmod +x /usr/local/bin/kubectl kubectl version
-
In addition to kOps and kubectl, install yq, a portable command-line YAML processor. You can follow yq installation instructions for your system. On Cloud9 and Linux, you can install yq with the command on the right. The command requires that Go tools are installed in your environment. You can run
go versionto check if Go is already installed in your environment; if it is not, install go tools before proceeding with this step.GO111MODULE=on go get github.com/mikefarah/yq ; export PATH=$PATH:~/go/bin
Step 2: Set up kOps Cluster environment and state store
In this step you will configure some of the environment variables that will be used to set up your environment, and create and configure the S3 bucket that kOps will use as states store.
-
Export environment variables according to the following requirements:
- The name of your cluster will be “spot-kops-cluster”. To reduce the dependencies on other services, in this tutorial you will create your cluster using Gossip DNS, hence the cluster domain will be k8s.local and the fully qualified name of the cluster spot-kops-cluster.k8s.local.
- You will also create an S3 bucket where kOps configuration and the cluster's state will be stored. You will use uuidgen to generate a unique S3 bucket name.
- In the above command, you will set the environment variables that will be used across the rest of the session.
export NAME=spot-kops-cluster.k8s.local export KOPS_STATE_PREFIX=spot-kops-$(uuidgen) export KOPS_STATE_STORE=s3://${KOPS_STATE_PREFIX}
-
Additionally you will set a few other environment variables that define the region and availability zones where your cluster will be deployed. In this tutorial, the region will be “us-east-1”, you can change this and point it to the region where you would prefer running your cluster.
export AWS_REGION=us-east-1 export AWS_REGION_AZS=$(aws ec2 describe-availability-zones \ --region ${AWS_REGION} \ --query 'AvailabilityZones[0:3].ZoneName' \ --output text | \ sed 's/\t/,/g')
-
Now that you have the name of your cluster and S3 State Store bucket defined, let's create the S3 bucket.
aws s3api create-bucket \ --bucket ${KOPS_STATE_PREFIX} \ --region ${AWS_REGION}
-
Once the bucket has been created, you can apply one of kOps best practices by enabling S3 Versioning on the bucket. S3 is acting as the state store, and by enabling versioning on the bucket you will be able to recover your cluster back to a previous state and configuration.
aws s3api put-bucket-versioning \ --bucket ${KOPS_STATE_PREFIX} \ --region ${AWS_REGION} \ --versioning-configuration Status=Enabled
Step 3: Cluster creation and On-Demand node configuration
In this step you will create the cluster control plane and a kOps InstanceGroup with OnDemand instances. You will also add some labels to the group, so that you can place pods accordingly later on.
-
It is now time to create the cluster. You will build a Highly Available (HA) cluster using m5.large instances for the kubernetes masters spread across three Availability Zones. Additionally you will create an InstanceGroup with two t3.large OnDemand worker nodes, that you will use to demonstrate how you can configure your applications to run on Spot or OnDemand Instances, depending on the type of workflow.
kops create cluster \ --name ${NAME} \ --state ${KOPS_STATE_STORE} \ --cloud aws \ --master-size m5.large \ --master-count 3 \ --master-zones ${AWS_REGION_AZS} \ --zones ${AWS_REGION_AZS} \ --node-size t3.large \ --node-count 2 \ --dns private
-
Great! The output of the previous command displays all the resources that will be created. You can check that the cluster configuration has been written to the kOps state S3 bucket. The following command should showcase the cluster state, and yield and an output similar to the following one:
aws s3 ls --recursive ${KOPS_STATE_STORE}2022-05-11 00:08:21 0 spot-kops-cluster.k8s.local/clusteraddons/default 2022-05-11 00:08:21 1723 spot-kops-cluster.k8s.local/config 2022-05-11 00:08:21 454 spot-kops-cluster.k8s.local/instancegroup/master-us-east-1a 2022-05-11 00:08:21 454 spot-kops-cluster.k8s.local/instancegroup/master-us-east-1b 2022-05-11 00:08:21 454 spot-kops-cluster.k8s.local/instancegroup/master-us-east-1c 2022-05-11 00:08:21 450 spot-kops-cluster.k8s.local/instancegroup/nodes-us-east-1a 2022-05-11 00:08:21 450 spot-kops-cluster.k8s.local/instancegroup/nodes-us-east-1b 2022-05-11 00:08:21 450 spot-kops-cluster.k8s.local/instancegroup/nodes-us-east-1c
-
As for the two nodes in the InstanceGroup that you created, you should label those as OnDemand nodes by adding a lifecycle label. kOps created an instance group per AZ for your nodes, so you will apply the changes to each of them. To merge the new configuration attributes to the cluster nodes, you will use yq.
for availability_zone in $(echo ${AWS_REGION_AZS} | sed 's/,/ /g') do NODEGROUP_NAME=nodes-${availability_zone} echo "Updating configuration for group ${NODEGROUP_NAME}" cat << EOF > ./nodes-extra-labels.yaml spec: nodeLabels: kops.k8s.io/lifecycle: OnDemand EOF kops get instancegroups --name ${NAME} ${NODEGROUP_NAME} -o yaml > ./${NODEGROUP_NAME}.yaml yq merge --overwrite --inplace ./${NODEGROUP_NAME}.yaml ./nodes-extra-labels.yaml aws s3 cp ${NODEGROUP_NAME}.yaml ${KOPS_STATE_STORE}/${NAME}/instancegroup/${NODEGROUP_NAME} done
-
You can validate the result of your changes by running the following command, and verifying that the labels have been added to the spec.nodeLabels section. The output of this command should be:
- Instancegroup nodes-us-east-1a contains label kops.k8s.io/lifecycle: OnDemand
- Instancegroup nodes-us-east-1b contains label kops.k8s.io/lifecycle: OnDemand
- Instancegroup nodes-us-east-1c contains label kops.k8s.io/lifecycle: OnDemand
for availability_zone in $(echo ${AWS_REGION_AZS} | sed 's/,/ /g') do NODEGROUP_NAME=nodes-${availability_zone} kops get ig --name ${NAME} ${NODEGROUP_NAME} -o yaml | grep "lifecycle: OnDemand" > /dev/null if [ $? -eq 0 ] then echo "Instancegroup ${NODEGROUP_NAME} contains label kops.k8s.io/lifecycle: OnDemand" else echo "Instancegroup ${NODEGROUP_NAME} DOES NOT contains label kops.k8s.io/lifecycle: OnDemand" fi done
-
Aside from validating that the lifecycle label is set up, you can inspect one of the nodegroup's configuration. Run the following command to view it.
kops get ig --name ${NAME} nodes-$(echo ${AWS_REGION_AZS}|cut -d, -f 1) -o yaml
Step 4: Adding Spot workers with kOps toolbox instance-selector
Until January of 2021, to adhere to Spot best practices using kOps, users were required to select a group of Spot instances to diversify manually. They then had to configure a MixedInstancePolicy InstanceGroup in order to apply diversification within the instance group. In that date, we introduced a new tool: kOps toolbox instance-selector. This tool is distributed as part of the standard kOps distribution, and it simplifies the creation of kOps Instance Groups, by creating groups that fully adhere to Spot Instances best practices.
In order to tap into multiple Spot capacity pools, you will create two Instance Groups, each containing multiple instance types. Diversifying into more capacity pools increases the chances of achieving the desired scale, and maintaining it if some of the capacity pools get interrupted (when EC2 needs the capacity back). Each Instance Group (EC2 Auto Scaling group) will launch instances using Spot pools that are optimally chosen based on the available Spot capacity.
-
The following command creates an Instance Group, which will be called spot-group-base-4vcpus-16gb. To create the group, use kOps toolbox instance-selector, which saves the effort of manually configuring the new group for diversification. In this case, use the
--instance-type-basewith m5.xlarge as your base instance, made up of pools from the latest generations. You can get more information about which parameterskops toolbox instance-selectoruses by running kops toolbox instance-selsector –-helpkops toolbox instance-selector "spot-group-base-4vcpus-16gb" \ --usage-class spot --cluster-autoscaler \ --base-instance-type "m5.xlarge" --burst-support=false \ --deny-list '^?[1-3].*\..*' --gpus 0 \ --node-count-max 20 --node-count-min 1 \ --name ${NAME}
-
Now let’s create the second Instance Group. This time, you will create the group spot-group-base-2vcpus-8gb, using a different approach as in the previous step. Instead of defining a base instance type, now you will specify the amount of memory and vcpus:
kops toolbox instance-selector "spot-group-base-2vcpus-8gb" \ --usage-class spot --cluster-autoscaler \ --burst-support=false \ --vcpus=2 --memory=8GB \ --gpus 0 --node-count-max 20 \ --node-count-min 1 --name ${NAME}
-
Before proceeding with the final instantiation of the cluster, let’s validate and review the newly created Instance Group's configuration. Run the following command to display the configuration of the spot-group-base-2vcpus-8gb Instance Group.
kops get ig spot-group-base-2vcpus-8gb --name $NAME -o yaml -
Your cluster is now configured with all the resources depicted in the architecture diagram below.
However, you have only configured the cluster up to this point. To actually instantiate it, you must execute the following command:
kops update cluster --state=${KOPS_STATE_STORE} --name=${NAME} --yes --admin
⚠️ If your environment previously had a kubeconfig file, you may need to runkops export kubeconfig ${NAME} --adminto store the configuration and change the config. ] -
The command in the previous step will start requesting for all the cluster resources, and end up with an output similar to the following one. This may take around five minutes.
Cluster is starting. It should be ready in a few minutes. Suggestions: * validate cluster: kops validate cluster --wait 10m * list nodes: kubectl get nodes --show-labels * ssh to the master: ssh -i ~/.ssh/id_rsa ubuntu@api.spot-kops-cluster.k8s.local * the ubuntu user is specific to Ubuntu. If not using Ubuntu please use the appropriate user based on your OS. * read about installing addons at: https://kops.sigs.k8s.io/addons.
-
You can run the kOps validate cluster command to evaluate the state of the cluster a few times per minute, capturing the progress of its creation.
kops validate cluster --wait 10m
The command will raise errors until the cluster is created.
-
Once the cluster is in a healthy state, you can list nodes to check that the cluster and all its associated resources are up and running.
kubectl get nodes --show-labels
Step 5: Deploying the aws-node-termination-handler
When an interruption happens, EC2 sends a Spot interruption notification to the instance, giving the application two minutes to gracefully handle that interruption, and minimize the impact to its availability or performance. Also, recently the new Instance Rebalance Recommendation signal was made available, which notifies you when a Spot Instance is at elevated risk of interruption; it can arrive sooner that the Spot interruption notice, giving you extra time to proactively manage the Spot Instance, by rebalancing to new or existing Spot Instances that are not at risk. In order to gracefully handle either scenario on Kubernetes, we will deploy the aws-node-termination-handler in this section.
Let's proceed to installing the aws-node-termination-handler in Queue Processor mode, with the help of kOps. The Handler will continuously poll an Amazon SQS queue, which receives events emitted by Amazon EventBridge that can lead to the termination of the nodes in your cluster (Spot Interruption/Rebalance events, maintenance events, Auto-Scaling Group lifycle hooks and more). This enables the Handler to cordon and drain the node - also issuing a SIGTERM to the Pods and containers running on it, in order to achieve a graceful application termination.
-
kOps facilitates the deployment of the aws-node-termination-handler, allowing you to add its configuration as an addon to the kOps cluster spec. This addon also takes care of deploying all the necessary AWS infrastructure for you: SQS Queue, EventBridge rules, and the necessary Auto-Scaling group Lifecycle hooks. Deploy the aws-node-termination-handler addon with the following command:
kops get cluster --name ${NAME} -o yaml > ~/environment/cluster_config.yaml cat << EOF > ./node_termination_handler_addon.yaml spec: nodeTerminationHandler: cpuRequest: 200m enabled: true enableSQSTerminationDraining: true managedASGTag: "aws-node-termination-handler/managed" EOF yq merge --overwrite --inplace ~/environment/cluster_config.yaml ~/environment/node_termination_handler_addon.yaml aws s3 cp ~/environment/cluster_config.yaml ${KOPS_STATE_STORE}/${NAME}/config kops update cluster --state=${KOPS_STATE_STORE} --name=${NAME} --yes --admin
-
To check that the aws-node-termination-handler has been deployed successfully, execute the following command.
kubectl get deployment aws-node-termination-handler -n kube-system -o wide
Step 6: Deploy the Kubernetes Cluster Autoscaler
Cluster Autoscaler is a Kubernetes controller that dynamically adjusts the size of the cluster. If there are pods that can't be scheduled in the cluster due to insufficient resources, Cluster Autoscaler will issue a scale-out action. When there are nodes in the cluster that have been under-utilized for a period of time, Cluster Autoscaler will scale-in the cluster. Internally Cluster Autoscaler evaluates a set of instance groups to scale up the cluster. When Cluster Autoscaler runs on AWS, instance groups are implemented using Auto Scaling Groups. To calculate the number of nodes to scale-out/in when required, Cluster Autoscaler assumes all the instances in an instance group are homogenous (i.e. have the same number of vCPUs and memory size).
Metrics Server is a scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines. These metrics will drive the scaling behavior of the deployments.
-
Before installing Cluster Autoscaler, install the metric server using a kOps addon:
kops get cluster --name ${NAME} -o yaml > ~/environment/cluster_config.yaml cat << EOF > ./metric_server_addon.yaml spec: certManager: enabled: true metricsServer: enabled: true EOF yq merge --overwrite --inplace ~/environment/cluster_config.yaml ~/environment/metric_server_addon.yaml aws s3 cp ~/environment/cluster_config.yaml ${KOPS_STATE_STORE}/${NAME}/config kops update cluster --state=${KOPS_STATE_STORE} --name=${NAME} --yes --admin
-
Wait until the metric server is up and running:
kubectl get deploy metrics-server -n kube-system
-
kOps facilitates the deployment of the Cluster Autoscaler, allowing you to add its configuration as an addon to the kOps cluster spec. Deploy the Cluster Autoscaler addon with the following command:
kops get cluster --name ${NAME} -o yaml > ~/environment/cluster_config.yaml cat << EOF > ./cluster_autoscaler_addon.yaml spec: clusterAutoscaler: awsUseStaticInstanceList: false balanceSimilarNodeGroups: false cpuRequest: 100m enabled: true expander: random image: us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.23.0 memoryRequest: 300Mi newPodScaleUpDelay: 0s scaleDownDelayAfterAdd: 10m0s skipNodesWithLocalStorage: true skipNodesWithSystemPods: true EOF yq merge --overwrite --inplace ~/environment/cluster_config.yaml ~/environment/cluster_autoscaler_addon.yaml aws s3 cp ~/environment/cluster_config.yaml ${KOPS_STATE_STORE}/${NAME}/config kops update cluster --state=${KOPS_STATE_STORE} --name=${NAME} --yes --admin
-
You can also check the logs and steps taken by Cluster Autoscaler with the following command. This command will display Cluster Autoscaler logs.
kubectl logs -f deployment/cluster-autoscaler -n kube-system --tail=10
Step 7: Deploy a Sample Application
Finally let's deploy a test application and scale our cluster. To scale our application, we will use a Deployment and a Horizontal Pod Autoscaler.
-
Deploy an application and expose as a service on TCP port 80. The application is a custom-built image based on the php-apache image. The index.php page performs calculations to generate CPU load. More information can be found here
kubectl create deployment php-apache --image=us.gcr.io/k8s-artifacts-prod/hpa-example kubectl set resources deploy php-apache --requests=cpu=1000m kubectl expose deploy php-apache --port 80 kubectl get pod -l app=php-apache -
Create an Hpa resource. This HPA scales up when CPU exceeds 30% of the allocated container resource.
kubectl autoscale deployment php-apache \ --cpu-percent=30 \ --min=1 \ --max=50
-
View the HPA using kubectl. You probably will see
<unknown>/30%for 1-2 minutes and then you should be able to see0%/30%.kubectl get hpa
-
Open a new terminal window and create a new container:
kubectl run -i --tty load-generator-1 --image=busybox /bin/sh
-
Execute a while loop to generate load in the application:
while true; do wget -q -O - http://php-apache; done
-
In the previous terminal, watch the HPA with the following command:
kubectl get hpa -w
You should see your environment scaling.
Step 8: Clean Up
-
Remove the kOps cluster:
kops delete cluster --name ${NAME} --yes -
In the console, remove the S3 bucket. Read "Deleting a single object" section of the AWS Documentation to find out how to delete a bucket from the console.