K8S Mutating Webhook

What are Admission Webhooks.

Read kubernetes official documentation for more details about admission webhooks.

Steps marked OPTIONAL STEP are mainly for users using Windows host machine for development. These steps are optional for other platform users.

Environment

Minikube for development k8s cluster.
Cert Manager for managing CA Certs
OPTIONAL STEP: Docker (debian) for building go project. As k8s cluster is using *nix environment, we want to make sure to have similar development environment.
Source code is in src directory.
All K8S config required for this tutorial is available in configs directory

Installing Minikube K8S Cluster

Note: My local machine is macOS based. As per docker official documentation, we can access service on host machine from container using host.docker.internal. Hence, we need to make sure, we add this domain to k8s cluster certificate.

Please refer here for installing minikube.
Start minikube server.
1. OPTIONAL STEP: Start with api-server name host.docker.internal. This tell minikube to add additional hostname as SAN to cert. Windows users can also use host network to access host services from containers.
```
 minikube start --apiserver-names=host.docker.internal  
```
2. If OPTIONAL STEP steps is not followed, then run
```
minikube start  
```
OPTIONAL STEP: Copy ~/.kube/config to kube directory at the root of this project and update server as https://host.docker.internal. Do not change the port number.
For Webhook, we need to have a CA which can sign certificates for TLS. We are using cert-manager for this. Alternatively you can also use Cloudflare CFSSL but require lots of manual configuration. cert-manager is highly recommended.
1. Run from container if OPTIONAL STEP steps were followed, else you can directly run from host machine. Access to k8s cluster is required.
```
  kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml 
  kubectl get pods -n cert-manager  # Make sure all containers are running.  
```

Configuring Dockerfile

We will use this for building our app as well as for hosting webhook code.

Change directory to src
Create a Dockerfile

FROM golang:1.17-alpine as dev-env  
  
WORKDIR /app  
RUN apk add --no-cache curl && \  
 curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl && \ 
 chmod +x ./kubectl && \ 
 mv ./kubectl /usr/local/bin/kubectl

OPTIONAL STEP Docker for development.
```
docker build . -t webhook
```

Please make sure to replace image name with your preferred name.

Start a Dev Container

This section is OPTIONAL STEP

Start a dev container with Volume.

From src directory, run:

docker run -it --rm -p 80:80 -p 8443:8443 -v ${PWD}/../kube/:/root/.kube/ -v ${PWD}:/app -v /Users/`whoami`/.minikube:/Users/`whoami`/.minikube webhook sh

${PWD}/../kube/:/root/.kube/: This contains updated kubeconfig file with cluster domain host.docker.internal
${PWD}:/app: Mount current src directory to container. Used for building app.
/Users/`whoami`/.minikube:/Users/`whoami`/.minikube: Contains all certificate which are required to connect to k8s cluster from dev container.

Note: All go build needs to be executed inside dev containers

Dev environment verification

Note: Windows users should run this from docker dev container.

Let's define our main module and a web server inside src directory.
```
go mod init sample-mutating-webhook
```

Create a main.go inside src with package main. Let's add a minimal webserver code to it

package main  
  
import (  
 "log" 
 "net/http"
)  
  
func main() {  
 http.HandleFunc("/", HandleRoot) 
 http.HandleFunc("/mutate", HandleMutate) 
 log.Fatal(http.ListenAndServe(":80", nil))
}  
  
func HandleRoot(w http.ResponseWriter, r *http.Request){  
 w.Write([]byte("HandleRoot!"))
}  
  
func HandleMutate(w http.ResponseWriter, r *http.Request){  
 w.Write([]byte("HandleMutate!"))
}

Build:

export CGO_ENABLED=0 go build -o webhook
./webhook

Verify, if you can access http://localhost/mutate from Host machine browser.

Also, verify if you can access k8s cluster running on host machine

$ kubectl get nodes

Webhook Development

As we will receive webhook events from kubernetes, we need to translate those requests to an understandable format such as Objects or Struct. For this we need to deserialize them using K8S serializer.

// imports added:  
  
"k8s.io/apimachinery/pkg/runtime"  
"k8s.io/apimachinery/pkg/runtime/serializer"  
  
// code:  
  
var (  
 universalDeserializer = serializer.NewCodecFactory(runtime.NewScheme()).UniversalDeserializer()
 )

Now to access K8S, we need to have kube config. We will use k8s config GetConfigOrDie function.

// As global variable  
  
k8sConfig *rest.Config  
k8sClientSet *kubernetes.Clientset
  
// Inside main:  
  
k8sConfig = config.GetConfigOrDie()  
clientSet, err := kubernetes.NewForConfig(k8sConfig)

To test if configs are working, we added a test function inside podscount.go

For podscount to work, we also need to have a k8s client. Let's define that first in main.go

k8sClientSet = createClientSet()

Build the app and test it.

# ./webhook  
Total pod running in cluster: 12

We will also provide a way to override server port, tls location.

// ServerParameters : we need to enable a TLS endpoint  
// Let's take some parameters where we can set the path to the TLS certificate and port number to run on.  
type ServerParameters struct {  
 port           int    // webhook server port 
 certFile       string // path to the x509 certificate for https keyFile        
 string // path to the x509 private key matching `CertFile`
 }  
  
serverParameters ServerParameters  
  
  
// Inside main:  
  
flag.IntVar(&serverParameters.port, "port", 8443, "Webhook server port.")  
flag.StringVar(&serverParameters.certFile, "tlsCertFile", "/etc/webhook/certs/tls.crt", "File containing the x509 Certificate for HTTPS.")  
flag.StringVar(&serverParameters.keyFile, "tlsKeyFile", "/etc/webhook/certs/tls.key", "File containing the x509 private key to --tlsCertFile.")  
flag.Parse()

By Default, TLS files /etc/webhook/certs/tls.crt and /etc/webhook/certs/tls.key are injected using secret sidecar-injector-certs created using cert-manager.

Change Listener to TLS.

Change from

log.Fatal(http.ListenAndServe(":80", nil))

log.Fatal(http.ListenAndServeTLS(":" + strconv.Itoa(serverParameters.port), serverParameters.certFile, serverParameters.keyFile, nil))

Kubernetes sends us an AdmissionReview and expects an AdmissionResponse back. Lets us write logic to get AdmissionReview Request, pass it to universal decoder and and use it inside HandleMutate.

func getAdmissionReviewRequest(w http.ResponseWriter, r *http.Request) admissionv1.AdmissionReview {

	// Grabbing the http body received on webhook.
	body, err := ioutil.ReadAll(r.Body)
	if err != nil {
		panic(err.Error())
	}

	// Required to pass to universal decoder.
	// v1beta1 also needs to be added to webhook.yaml
	var admissionReviewReq admissionv1.AdmissionReview

	if _, _, err := universalDeserializer.Decode(body, nil, &admissionReviewReq); err != nil {
		w.WriteHeader(http.StatusBadRequest)
		_ = fmt.Errorf("could not deserialize request: %v", err)
	} else if admissionReviewReq.Request == nil {
		w.WriteHeader(http.StatusBadRequest)
		_ = errors.New("malformed admission review: request is nil")
	}
	return admissionReviewReq
}

// inside HandleMutate  
  
// func getAdmissionReviewRequest, grab body from request, define AdmissionReview
// and use universalDeserializer to decode body to admissionReviewReq

admissionReviewReq := getAdmissionReviewRequest(w, r)

We now need to capture Pod object from the admission request

var pod v1.Pod  
  
err = json.Unmarshal(admissionReviewReq.Request.Object.Raw, &pod)  
  
if err != nil {  
 _ = fmt.Errorf("could not unmarshal pod on admission request: %v", err)
 }

To perform a mutation on the object before the Kubernetes API sees the object, we need to apply a patch to the operation

var sideCarConfig *Config  
sideCarConfig = getNginxSideCarConfig()  
patches, _ := createPatch(pod, sideCarConfig)  
  
// getNginxSideCarConfig: Return Config object which contains SideCar Container and Volume Information  
// This inturn calls getPodVolumes and generateNginxSideCarConfig

Add patchBytes to the admission response

// Once you have completed all patching, convert the patches to byte slice:
 patchBytes, err := json.Marshal(patches)
 if err != nil {
     _ = fmt.Errorf("could not marshal JSON patch: %v", err)
 }

 // Add patchBytes to the admission response
 admissionReviewResponse := admissionv1.AdmissionReview{
     Response: &admissionv1.AdmissionResponse{
         UID: admissionReviewReq.Request.UID,
         Allowed: true,
     },
 }
 admissionReviewResponse.Response.Patch = patchBytes

Submit the response

bytes, err := json.Marshal(&admissionReviewResponse)  
if err != nil {  
   fmt.Errorf("marshaling response: %v", err)  
}  
  
w.Write(bytes)

Build the app

Publish Changes to Docker Hub

We now need to publish changes to docker hub so that it can be downloaded in k8s cluster.

Change Dockerfile as below:

FROM golang:1.17-alpine as dev-env

WORKDIR /app
RUN apk add --no-cache curl && \
    curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl && \
    chmod +x ./kubectl && \
    mv ./kubectl /usr/local/bin/kubectl

FROM dev-env as build-env
COPY go.mod /go.sum /app/
RUN go mod download

COPY . /app/

RUN CGO_ENABLED=0 go build -o /webhook

FROM alpine:3.10 as runtime

COPY --from=build-env /webhook /usr/local/bin/webhook
RUN chmod +x /usr/local/bin/webhook

ENTRYPOINT ["webhook"]

Build and Deploy

docker build . -t yks0000/sample-mutating-webhook:v2
docker push yks0000/sample-mutating-webhook:v2

Change name of image accordingly.

Creating K8S Resources

Create Certificate for Webhook

$ kubectl apply -f configs/certs.yaml
issuer.cert-manager.io/selfsigned-issuer unchanged
certificate.cert-manager.io/sidecar-injector-certs unchanged

Deploy RBAC

$ kubectl apply -f configs/rbac.yaml
serviceaccount/sample-mutating-webhook created
clusterrole.rbac.authorization.k8s.io/sample-mutating-webhook created
clusterrolebinding.rbac.authorization.k8s.io/sample-mutating-webhook created

Create Deployment

Make sure you update image to yks0000/sample-mutating-webhook:v1

$ kubectl apply -f configs/deployment.yaml
deployment.apps/sample-mutating-webhook created

Verify Pods:

$ kubectl -n default get pods | grep sample-mutating-webhook
sample-mutating-webhook-5d8666ffc7-4ljdh 1/1     Running   0          39s

Check Logs of pod, should emit log showing total number of pods

$ kubectl logs sample-mutating-webhook-5d8666ffc7-4ljdh
Total pod running in cluster: 13

Deploy Service

$ kubectl apply -f configs/service.yaml
service/sample-mutating-webhook created

Deploy Webhook

Make sure, you have the following annotation

annotations:  
 cert-manager.io/inject-ca-from: default/sidecar-injector-certs

In default/sidecar-injector-certs, default is namespace and sidecar-injector-certs is name of certificate that we created using certs.yaml

$ kubectl apply -f configs/webhook.yaml
mutatingwebhookconfiguration.admissionregistration.k8s.io/sample-mutating-webhook created

Test Mutation

In our example, we are adding a label nginx-sidecar to pod definition and injecting nginx container as sidecar before API server sent it to controller to schedule it.

As we also added objectSelector to webhook.yaml, we need to make sure inject-nginx-sidecar: "true" label is added to pod definition, otherwise our mutating webhook will ignore the request.

$ kubectl apply -f configs/example-pod.yaml

$ kubectl get pods --show-labels | grep example-pod
example-pod                                2/2     Running   0          66m   inject-nginx-sidecar=true,nginx-sidecar=applied-from-mutating-webhook

We can see here that nginx-sidecar=applied-from-mutating-webhook is added to Pod spec by Mutating webhook though it was not part of spec initially.

Also, we can see that the container count is 2, whereas in example-pod.yaml we only have 1 container. The other container is nginx container which is injected by mutating webhook written by us.

To access the service from browser, lets expose the port. We will expose port 443 to make sure, request is handled by injected Nginx sidecar which is doing SSL termination.

kubectl expose pod example-pod --type=LoadBalancer --port=443

As we are using minikube, we need to make sure we enable tunnel

minikube tunnel

$ curl https://127.0.0.1 -k
GET / HTTP/1.1
Host: 127.0.0.1
User-Agent: curl/7.79.1
Accept: */*
Connection: close
Time: 2022-06-01 07:35:21.4040591 +0000 UTC m=+4195.517198901
X-Forwarded-For: 172.17.0.1

Verifying Logs

We are using stern for streaming multi-container logs

$ stern example-pod
+ example-pod › rest-api
+ example-pod › nginx-webserver
example-pod rest-api 2022/06/01 07:36:05 Echoing back request made to / to client (127.0.0.1:44724)
example-pod nginx-webserver 172.17.0.1 - - [01/Jun/2022:07:36:05 +0000] "GET / HTTP/1.1" 200 184 "-" "curl/7.79.1"

We can see that there are two log lines, one from rest-api and other from nginx-webserver container.

Good to Know

If you wish to recreate pods from Deployment, you can just delete them. Not advisable in prod environment

kubectl delete pods --all

yks0000/k8s-mutating-webhooks