A simple webhook endpoint server that can be used to validate the images being created inside of the kubernetes cluster.
It works with two different types of Kubernetes admission controller:
- ImagePolicyWebhook
- GenericAdmissionWebhook (which starting from Kubernetes 1.9 has been renamed ValidatingAdmissionWebhook.
This admission controller will reject all the pods that are using images with
the latest tag.
I personally find the documentation of these admission controllers vague, confusing and missing some details.
I found the documentation about GenericAdmissionWebhook and
ValidatingAdmissionWebhook more troublesome to understand compared to the
one of ImagePolicyWebhook. You have to combine the 1.9 documentation of
ValidatingAdmissionWebhook together with the 1.8 documentation of
GenericAdmissionWebhook. The latter one references the
Dynamic Admission Control
which has more details.
This document will try to shed some light about these admission controllers, especially about how to deploy them.
I promise I'll try to improve upstream documentation as well :)
Note well: during the 1.8 -> 1.9 transition, in addition to being renamed,
the GenericAdmissionWebhook has also been promoted from being an alpha1
resource to be a beta1 one. That caused some changes in terms of the request
format sent from the API server to the webhook endpoint and in terms of
expected response.
Right now this document (and the code) focuses on version 1.9.
The ImagePolicyWebhook is an admission controller that evaluates only images.
Good things about ImagePolicyWebhook:
- The API server can be instructed to reject the images if the webhook endpoint is not reachable.
Bad things about ImagePolicyWebhook:
- More configuration files are expected on the API server node(s) compared to
GenericAdmissionWebhook. - It's a bit tricky to deploy the service providing the webhook endpoint on the kubernetes cluster (more on that later).
The GenericAdmissionWebhook (which starting from Kubernetes 1.9 has been renamed ValidatingAdmissionWebhook can evaluate all kind of resources.
Good things about ValidatingAdmissionWebhook:
- Only two changes are required on the kubernetes master node(s).
- Part of the configuration is defined by a Kubernetes object.
- It's incredibly easy to deploy the service providing the webhook endpoint on the kubernetes cluster.
Bad things about ValidatingAdmissionWebhook:
- Starting from the 1.8 release it's no longer possible to instruct the API server to reject the images if the webhook endpoint is not reachable. Hence, when the endpoint is not reachable, all the resources are going to be automatically accepted.
To build the project just do:
$ go get github.com/flavio/kube-image-bouncer
The project dependencies are tracked inside of this repository and are managed using dep.
This application is distributed also as a Docker image:
$ docker pull flavio/kube-image-bouncer
The webhook endpoint must be secured by tls to be used by kubernetes. This certificate can also be a self-signed one.
Create a server key and certificate with the following command:
$ openssl req -nodes -new -x509 -keyout webhook-key.pem -out webhook.pem
The API server uses a certificate to prove its identity. This certificate can also be a self-signed one.
Create a server key and certificate with the following command:
$ openssl req -nodes -new -x509 -keyout apiserver-client-key.pem -out apiserver-client.pem
Ensure the ImagePolicyWebhook admission controller is enabled. Refer to
the official
documentation.
Create an admission control configuration file named
/etc/kubernetes/admission_configuration.json file with the following
contents:
{
"imagePolicy": {
"kubeConfigFile": "/etc/kubernetes/kube-image-bouncer.yml",
"allowTTL": 50,
"denyTTL": 50,
"retryBackoff": 500,
"defaultAllow": false
}
}Note well: this configuration file will automatically reject all the images
if the server referenced by the webhook configuration is not reachable
(see the defaultAllow: false directive).
Create a kubeconfig file /etc/kubernetes/kube-image-bouncer.yml with the
following contents:
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority: /etc/kubernetes/kube-image-bouncer/webhook.pem
server: https://bouncer.local.lan:1323/image_policy
name: bouncer_webhook
contexts:
- context:
cluster: bouncer_webhook
user: api-server
name: bouncer_validator
current-context: bouncer_validator
preferences: {}
users:
- name: api-server
user:
client-certificate: /etc/kubernetes/kube-image-bouncer/apiserver-client.pem
client-key: /etc/kubernetes/kube-image-bouncer/apiserver-client-key.pemThis configuration file instructs the API server to reach the webhook server
at https://bouncer.local.lan:1323 and use its /image_policy endpoint.
Note that the certificates and keys we previously generated have been copied
under /etc/kubernetes/kube-image-bouncer.
Copy the webhook.pem and webhook-key.pem files to node running the webhook
service.
Start the webhook by doing:
$ kube-image-bouncer --cert webhook.pem --key webhook-key.pem
When using the Docker image:
$ docker run --rm -v `pwd`/webhook-key.pem:/certs/webhook-key.pem:ro -v `pwd`/webhook.pem:/certs/webhook.pem:ro -p 1323:1323 flavio/kube-image-bouncer -k /certs/webhook-key.pem -c /certs/webhook.pem
This will start a container with the server key and certificate mounted read-only inside of it.
If you want to perform tls termination outside of this application, just start it without providing a key and a certificate.
The webhook endpoint must be secured by tls to be used by kubernetes. This certificate can also be a self-signed one.
Create a server key and certificate with the following command:
$ openssl req -nodes -new -x509 -keyout webhook-key.pem -out webhook.pem
The API server uses a certificate to prove its identity. This certificate can also be a self-signed one.
Create a server key and certificate with the following command:
$ openssl req -nodes -new -x509 -keyout apiserver-clientkey.pem -out apiserver-client.pem
Ensure the GenericAdmissionWebhook admission controller is enabled:
the --admission-control flag must mention it.
Ensure the API server is started with the following flags:
--proxy-client-cert-file=/etc/kubernetes/apiserver-client.pem \
--proxy-client-key-file=/etc/kubernetes/apiserver-client-key.pem"
First of all you have to create a tls secret holding the webhook certificate and key:
kubectl create secret tls tls-image-bouncer-webhook \
--key webhook-key.pem \
--cert webhook.pem
Then create a kubernetes deployment for the image-bouncer-webhook:
kubectl apply -f kubernetes/image-bouncer-webhook.yaml
Finally create ValidatingWebhookConfiguration that makes use of
our webhook endpoint:
kubectl apply -f kubernetes/validating-webhook-configuration.yaml
Note well: the ExternalAdmissionHookConfiguration resource defined inside of
validating-webhook-configuration.yaml includes a CA certificate. This
is the apiserver-client.pem converted to base64.
As reported by the upstream docs:
After you create the validating webhook configuration, the system will take a few seconds to honor the new configuration.
It doesn't matter which kind of admission controller you created, the behaviour will be the same.
Create a nginx-versioned.yml file:
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx-versioned
spec:
replicas: 1
selector:
app: nginx-versioned
template:
metadata:
name: nginx-versioned
labels:
app: nginx-versioned
spec:
containers:
- name: nginx-versioned
image: nginx:1.13.8
ports:
- containerPort: 80Then create the resource:
$ kubectl create -f nginx-versioned.yml
Ensure the replication controller is actually running:
$ kubectl get rc
NAME DESIRED CURRENT READY AGE
nginx-versioned 1 1 0 2h
Now create a nginx-latest.yml file:
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx-latest
spec:
replicas: 1
selector:
app: nginx-latest
template:
metadata:
name: nginx-latest
labels:
app: nginx-latest
spec:
containers:
- name: nginx-latest
image: nginx
ports:
- containerPort: 80Then create the resource:
$ kubectl create -f nginx-latest.yml
This time the replication controller won't have all the desired pods running:
$ kubectl get rc
NAME DESIRED CURRENT READY AGE
nginx-latest 1 0 0 4s
nginx-versioned 1 1 0 2h
Get more details about the nginx-versioned replication controller:
$ kubectl describe rc nginx-latest
Name: nginx-latest
Namespace: default
Selector: app=nginx-latest
Labels: app=nginx-latest
Annotations: <none>
Replicas: 0 current / 1 desired
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=nginx-latest
Containers:
nginx-latest:
Image: nginx
Port: 80/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
ReplicaFailure True FailedCreate
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 22s (x14 over 1m) replication-controller Error creating: pods "nginx-latest-" is forbidden: image policy webhook backend denied one or more images: Images using latest tag are not allowed
The culprit is inside of the latest line of the output, the pod creation has been forbidden by our admission controller with the following message:
Images using latest tag are not allowed
The admission controller is used to vet all the containers scheduled to run
inside of the cluster. That includes containers providing core services like
kube-dns, dex, kubedash,... If the image bouncer service is unreachable these
services won't be accepted inside of the cluster (because we set defaultAllow to
false inside of /etc/kubernetes/admission_configuration.json).
We could run the image bouncer on top of the kubernetes cluster, but that would require its container to be accepted into the cluster, which leads to a "chicken-egg" problem.