Kubernetes Dynamic Admission Control

Demo of Kubernetes Dynamic Admission Control (DAC) exploring various concepts, capabilities and constraints ( focusing on validation admission webhooks ).

Guide for running this demo locally on minikube.

Example

This repo contains an example of all moving components for creating and testing Dynamic Admission Control

ValidatingWebhookConfiguration
Webhook Server
Test busybox pod for experimenting and verifying DAC works.
Service
Makefile containing some helper commands
- Script for setting up and deploying all components. make deploy
- Script for tearing down all created resources. make cleanup
- Script for building and pushing docker image in the local registry. make docker-image
Minikube guide containing all steps to run this example locally on Minikube.

Webhook server

Webhook-server allows some configurable behaviour, but you can tweak it and add your business and experiment with different scenarios.

You can set env ALLOW_SCHEDULING to "true" and instead of an error the DAC will emit a warning

Warning: Team label not set on pod

You can set env SKIP_NAMESPACE to a comma separated string for a list of namespace to skip. By default "kube-public" and "kube-system" are skipped.

Observations and notes

Namespace Selector

You can limit which requests are reaching your webhook-server depending on resources' namespaces namespaceSelector.

I couldn't find a way to whitelist namespaces by their name, except of building this logic inside the webhook-server.

For the namespaceSelector you can test it with adding this to webhook configuration

namespaceSelector:
  matchExpressions:
    - key: webhook-skip
      operator: DoesNotExist

and then add the label to the namespaces that you want to skip e.g. webhook-demo

kubectl label ns webhook-demo webhook-skip=true

Reliability

Dynamic Admission Controller can be on the critical path, depending how you your Webhook configured e.g. can block various actions on Kubernetes resources.

There are couple of things someone can do increase the reliability and the availability of the admission controller.

This is not an exhaustive list

The Dynamic Admission Controller is backed by a Server, that means all concepts for making a stateless deployment more reliable in Kubernetes (if it is hosted on the kubernetes)
- Use load balancing techniques that provides high availability and performance benefits [Availability].
- Rolling update strategy and Pod Disruption Budget.
  
  Rolling updates allow Deployments' update to take place with zero downtime by incrementally updating Pods instances with new ones
- Horizontal Pod Autoscaling for matching increasing demand.
Avoiding deadlock in self hosted webhooks
Avoiding operating on the kube-system namespace
It is recommended that admission webhooks should evaluate as quickly as possible (typically in milliseconds), since they add to API request latency. It is encouraged to use a small timeout for webhooks.
Configure Failure Policy depending your needs.
The API server exposes Prometheus metrics from the /metrics endpoint, which can be used for monitoring and diagnosing API server status.

e.g. apiserver_admission_webhook_admission_duration_seconds_sum, apiserver_admission_webhook_rejection_count