Kubernetes resource deletion validation webhook

AdmissionControl validating webhook to block deletion of resources based on resource labels.

What this solution does

For Configured Kubernetes namespaces and resources (deployments, crds etc) deletion requests are first processed by this webhook, and based on labels associated with the resource the resource deletion requests (kubectl delete resource .) are either rejected or allowed. The label keys and values for which resource deletion requests are rejected can also be configured.

Solution Overview

In the above diagram the validating webhook has been configured to monitor deployments and myCRDs in the default namespace
So if deletion request for all 6 resources shown in the diagram above are made to the kubernetes API, the requests for the 2 deployments and myCRD resource in the default namespace will be forwarded to our webhook.
Since the webhook has been configured to reject resource deletion requests where resources have a label deleteLock=enabled the deletion requests for the deployment and myCRD highlighted in orange will be rejected.
For the kubernetes core components to communicate with the webhook, TLS certificate and TLS key need to be created and associated with the webhook

Reference solution

The avast/k8s-admission-webhook is a good reference for create and update resource validations.
The avast solution has been referenced for the initialization / configuration of this application (using cobra, viper etc). The script to generate the certificate key files has also been referenced

Key Functions of webhook implementation

webhook.go->validate : This operation receives the AdmissionReview request object from Kubernetes. It then extracts the name of the resource to be deleted, the namespace, the API group, the API version, and the resource type. These are then passed to the getResourceLabels Operations. The labels are then passed to the validate.go->isDeletionRequestToBeBlocked operation to check if the deletion is to be blocked
webhook.go->getResourceLabels : This operation uses the REST client to fetch the labels for resource and returns the map or labels
validate.go->isDeletionRequestToBeBlocked : This checks if the resource labels indicate that the resource is to be deleted or not.

Installation

Building and Pushing the container image :The Make target docker-build or docker-build-local can be used to create the container image. The make docker-push makefile target can be used to push the container image to the container registry. With the make docker-build-local makefile target you need dependencies like glide on your machine, the make docker-build makefile target uses a multi stage build for building the go binary. Make sure you change the values of CONTAINER_NAME, CONTAINER_VERSION in the Make file.
The deployments/webhook-k8s-resources.template.yaml is the kubernetes manifest template for this solution. The main kubernetes resources to be created are ValidatingWebhookConfiguration, a deployment and a service. The template has place holders for the TLS Certificate, the TLS Key, the CA Bundle and the container image.
The Steps mentioned of the avast repository explain how to replace values in the yaml template file. Instead of manually doing it you can using the make gen-k8s-manifests Makefile target from this solution. This is described in more detail as follows
The makefile target make gen-k8s-manifests in this solution has all steps to replace values in the template, and as an output it generates the deployments/webhook-k8s-resources.yaml which has certificate, key and the ca bundle in the yaml. To execute this make target you need to have access to the target kubernets cluster (KUBECONFIG or ./kube/config). Before running the make target, verify that the values of the CONTAINER_NAME, CONTAINER_VERSION, WEBHOOK_NAMESPACE and WEBHOOK_SERVICE_NAME in the Makefile are correct.
After this applying the generated deployments/webhook-k8s-resources.yaml file creates all required kubernetes resources. By default entry for this generated file is in the .gitignore file.

Commands

Clone the repo

git clone https://github.com/maniSbindra/k8s-delete-validation-webhook.git

Modify Makefile values for CONTAINER_NAME, CONTAINER_VERSION, WEBHOOK_NAMESPACE and WEBHOOK_SERVICE_NAME
Make sure you are logged in to the container registry and have access to the kubernetes cluster
Build and Push container image
```
make docker-build
make docker-push
```
Generate certs, keys and generate actual yaml from yaml template.
```
make gen-k8s-manifests
```

Apply the yaml

kubectl apply -f deployments/webhook-k8s-resources.yaml

Verify installation

Check the deployment k8s-delete-validation-webhook, for which a pod should be running
```
kubectl get deploy k8s-delete-validation-webhook
```
add the label webhook=enabled to the default namespace. Note this is the default value and can be changed in the namespace selector
```
kubectl label namespace default webhook=enabled
```

Create Deployment with deleteLock=enabled label

kubectl run nginx --image=nginx --port=80 --labels=deleteLock=enabled

Trying to delete this deployment should throw following error

kubectl delete deploy nginx
Error from server: admission webhook "k8s-deletion-validation-webhook.test.com" denied the request: The deployment cannot be deleted as deletions are locked for this deployment

Configuring the namespaces, resources and labels

By modifying the rules section of the validatingwebhookconfiguration resource we can change the namespaces, api groups, api versions and resource types which this webhook handles. We also need to change the namespaces, and resource permissions for the cluster role in the Cluster Role and Binding section
By modifying the environment settings of the webhook deployment, we can control the delete rejection message, and the key and value of the label used to determine if resource deletion requests are rejected
You need to add the label specified in the namespace selector to any namespace where you expect the webhook to perform the validation