kube-mgmt
Policy-based control for Kubernetes deployments.
About
kube-mgmt
manages instances of the Open Policy Agent on top of Kubernetes. Use kube-mgmt
to:
- Load policies into OPA (via Kubernetes APIs)
- Replicate Kubernetes resources into OPA
- Deploy OPA as an Admission Controller
- Deploy OPA as an Initializer
NOTE: kube-mgmt
is currently in alpha. Join the discussion on slack.openpolicyagent.org.
Deployment Guide
Hello World
-
Create a new Namespace to deploy OPA into:
kubectl create namespace opa
-
Create a new Deployment that includes OPA and
kube-mgmt
(manifests/deployment.yml
):kubectl -n opa create -f https://raw.githubusercontent.com/open-policy-agent/kube-mgmt/master/manifests/deployment.yml
-
Define a simple policy (
example.rego
) with the following content:package kubernetes example = "Hello, Kubernetes!"
-
Create a ConfigMap containing the policy:
kubectl -n opa create configmap hello-world --from-file example.rego
-
Create a Service to expose OPA:
kubectl -n opa expose deployment opa --type=NodePort
-
Execute a policy query against OPA:
OPA_URL=$(minikube service -n opa opa --url) curl $OPA_URL/v1/data/kubernetes/example
Policies
kube-mgmt
automatically discovers policies stored in ConfigMaps in Kubernetes
and loads them into OPA. kube-mgmt
assumes a ConfigMap contains policies if
the ConfigMap is:
- Created in a namespace listed in the --policies option.
- Labelled with
openpolicyagent.org/policy=rego
.
When a policy has been successfully loaded into OPA, the
openpolicyagent.org/policy-status
annotation is set to {"status": "ok"}
.
If loading fails for some reason (e.g., because of a parse error), the
openpolicyagent.org/policy-status
annotation is set to {"status": "error", "error": ...}
where the error
field contains details about the failure.
Caching
kube-mgmt
can be configured to replicate Kubernetes resources into OPA so that
you can express policies over an eventually consistent cache of Kubernetes
state.
Replication is enabled with the following options:
# Replicate namespace-level resources. May be specified multiple times.
--replicate=<[group/]version/resource>
# Replicate cluster-level resources. May be specified multiple times.
--replicate-cluster=<[group/]version/resource>
Example Options
The example below would replicate Deployments, Services, and Nodes into OPA:
--replicate=apps/v1beta/deployments
--replicate=v1/services
--replicate-cluster=v1/nodes
Admission Control
To use OPA as an Admission
Controller
in Kubernetes 1.7 or later, follow the steps in External Admission
Webhooks
to enable webhooks in the Kubernetes API server. Once you have configured the
Kubernetes API server and generated the necessary certificates you can start
kube-mgmt
with the following options:
--register-admission-controller
--admission-controller-ca-cert-file=/path/to/ca/cert.pem
--admission-controller-service-name=<name-of-opa-service>
--admission-controller-service-namespace=<namespace-of-opa-service>
You will need to create Secrets containing the server certificate and private key as well as the CA certificate:
kubectl create secret generic opa-ca --from-file=ca.crt
kubectl create secret tls opa-server --cert=server.crt --key=server.key
See Generating TLS Certificates below for examples of how to generate the certificate files.
The example below shows how to deploy OPA and enable admission control:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: opa
name: opa
spec:
replicas: 1
template:
metadata:
labels:
app: opa
name: opa
spec:
containers:
- name: opa
image: openpolicyagent/opa:0.5.2
args:
- "run"
- "--server"
- "--tls-cert-file=/certs/tls.crt"
- "--tls-private-key-file=/certs/tls.key"
- "--addr=0.0.0.0:443"
- "--insecure-addr=127.0.0.1:8181"
volumeMounts:
- readOnly: true
mountPath: /certs
name: opa-server
- name: kube-mgmt
image: openpolicyagent/kube-mgmt:0.4
args:
- "--register-admission-controller"
- "--admission-controller-ca-cert-file=/certs/ca.crt"
- "--admission-controller-service-name=opa"
- "--admission-controller-service-namespace=$(MY_POD_NAMESPACE)"
volumeMounts:
- readOnly: true
mountPath: /certs
name: opa-ca
env:
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumes:
- name: opa-server
secret:
secretName: opa-server
- name: opa-ca
secret:
secretName: opa-ca
---
kind: Service
apiVersion: v1
metadata:
name: opa
spec:
clusterIP: 10.0.0.222
selector:
app: opa
ports:
- name: https
protocol: TCP
port: 443
targetPort: 443
Admission control policies must produce a document at /system/main
that
represents the admission control decision (i.e., allow or deny).
Example Policy
To test that admission control is working, define a policy that rejects the
request if the test-reject
label is found:
package system
main = {
"apiVersion": "admission.k8s.io/v1alpha1",
"kind": "AdmissionReview",
"status": status,
}
default status = {"allowed": true}
status = reject {
input.spec.operation = "CREATE"
input.spec.object.labels["test-reject"]
}
reject = {
"allowed": false,
"status": {
"reason": "testing rejection"
}
}
Generating TLS Certificates
External Admission Controllers must be secured with TLS. At a minimum you must:
-
Provide the Kubernetes API server with a client key to use for webhook calls (
client.key
andclient.crt
below). -
Provide OPA with a server key so that the Kubernetes API server can authenticate it (
server.key
andserver.crt
below). -
Provide
kube-mgmt
with the CA certificate to register with the Kubernetes API server (ca.crt
below).
Follow the steps below to generate the necessary files for test purposes.
First, generate create the required OpenSSL configuration files:
client.conf:
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
IP.1 = 127.0.0.1
server.conf:
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
IP.1 = 10.0.0.222
The subjectAltName/IP address in the certificate MUST match the one configured on the Kubernetes Service.
Finally, generate the CA and client/server key pairs.
# Create a certificate authority
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"
# Create a server certiticate
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -subj "/CN=admission_server" -config server.conf
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_req -extfile server.conf
# Create a client certiticate
openssl genrsa -out client.key 2048
openssl req -new -key client.key -out client.csr -subj "/CN=admission_client" -config client.conf
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 100000 -extensions v3_req -extfile client.conf
If you are using minikube, you can specify the client TLS credentials with the following minikube start
options:
--extra-config=apiserver.ProxyClientCertFile=/path/to/client.crt # in VM
--extra-config=apiserver.ProxyClientKeyFile=/path/to/client.key # in VM
Initializers
To use OPA as an Initializer you must be running Kubernetes 1.7 or later.
Once you have configured the Kubernetes API server to enable initialization
controllers, you can start kube-mgmt
with the following options:
# Enable initializer for given namespace-level resource.
# May be specified multiple times.
--initialize=<[group/]version/resource>
# Enable initializer for given cluster-level resource. May be specified multiple times.
--initialize-cluster=<[group/]version/resource>
# Set path of initialization document to query. Defaults to /kubernetes/admission/initialize.
--initialize-path=<path-relative-to-/data>
The example below shows how to deploy OPA and enable initializers for Deployments and Services:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: opa
name: opa
spec:
replicas: 1
template:
metadata:
labels:
app: opa
name: opa
spec:
containers:
- name: opa
image: openpolicyagent/opa:0.5.2
args:
- "run"
- "--server"
- name: kube-mgmt
image: openpolicyagent/kube-mgmt:0.4
args:
- "--initialize=v1/services"
- "--initialize=apps/v1beta1/deployments"
If initializers are enabled, kube-mgmt
will register itself as an
initialization controller on the specified resource type (you do not have to
create a initializer configuration yourself.)
Example Policy
The policy below will inject OPA into Deployments that indicate they require OPA:
package kubernetes.admission
initialize = merge {
input.kind = "Deployment"
input.metadata.annotations["requires-opa"]
merge = {
"spec": {
"template": {
"spec": {
"containers": [
{
"name": "opa",
"image": "openpolicyagent/opa",
"args": [
"run",
"--server",
]
}
]
}
}
}
}
}
Development Guide
To run all of the tests and build the Docker image run make
in this directory.