Kube-valet is a custom controller for Kubernetes. It allows cluster administrators or users to:
- Dynamically allocate cluster capacity for specific workloads
- Automatically apply scheduling configuration to pods after they are created
It does this by utilizing CustomResourceDefinitions to define its behavior and a MutatingWebhook to modify pods as they are created.
Kube-valet is currently in an alpha state.
- Kubernetes v1.13 or greater
- The MutatingWebhook AdmissionController must be enabled (Default on most clusters)
- The admissionregistration.k8s.io api group must be enabled (Default on most clusters)
- Cluster administrator level access
This is the easiest and fastest way to get started with kube-valet.
git clone git@github.com:domoinc/kube-valet.git
cd kube-valet
helm install -n kube-valet --set tls.auto=true --wait ./helmIt is normal for the automatic tls installation to take a few minutes while the images download and all the resource become ready.
Note: The instructions below are for generating a new self-signed certficates. This is done using CFSSL: Cloudflare's PKI and TLS toolkit. Click here to know more. The cfssl tools used can be downloaded at https://pkg.cfssl.org/.
# Clone the repository and cd into it
git clone git@github.com:domoinc/kube-valet.git
cd kube-valet# Make a dedicated dir for the certificates and cd into it
mkdir tls
cd tls
# Init a ca with cfssl
cat <<EOF | cfssl genkey -initca - | cfssljson -bare ca
{
"CN": "kube-valet-ca",
"key": {
"algo": "ecdsa",
"size": 256
}
}
EOF
# Create a self-signed key and certificate
cat <<EOF | cfssl gencert -ca ca.pem -ca-key ca-key.pem - | cfssljson -bare server
{
"CN": "kube-valet.kube-valet.svc",
"hosts": [
"kube-valet.kube-valet.svc",
"kube-valet.kube-valet.svc.cluster.local"
],
"key": {
"algo": "ecdsa",
"size": 256
}
}
EOF
# Go back to the root of the project
cd ../# Put the certificates in the location expected by the helm templates by default
cp -r tls helm/
# Do helm install using default cert, ca, and key paths
helm install -n kube-valet --wait ./helm# Replace the cert and key placeholders in the valet secret
vim deploy/secret.yaml
# Replace the ca cert placeholder in the webhook config
vim deploy/mutatingwebhookconfiguration.yaml
# Apply the namespace
kubectl apply -f deploy/namespace.yaml
# Apply the deployment files
kubectl apply -f deploy/Kube-valet is configured entirely through the custom resources made during installation. The resources can be created, updated, and deleted just like any other kubernetes resource. Example:
kubectl create -f _examples/resources/nodeassignmentgroups/simple.yaml
kubectl get nags
kubectl delete nag simpleAvailable custom resources:
- pars - podassignmentrules
- cpars - clusterpodassignmentrules
- nags - nodeassignmentgroups
See the examples for example client-go scripts and detailed custom resource examples.
Valetctl is a tool that makes it easier to create and report on kube-valet resources.
curl -Lo /usr/local/bin/valetctl https://github.com/domoinc/kube-valet/releases/download/v2018.10.17.0/valetctl
chmod +x /usr/local/bin/valetctlSet aside some nodes and make specific pods target those nodes.
# Isolate one node for sensitive jobs and label two more for relaxed jobs
valetctl group create for-jobs sensitive:1:LabelAndTaint relaxed:2
# Make sure all jobtype=sensitive pods in any namespace run on the isolated node.
valetctl assignment create sensitivereq require -t jobtype=sensitive -A for-jobs/sensitive
# Make sure all jobtype=relaxed pods prefer to run on the separated nodes
valetctl assignment create relaxedpref prefer -t jobtype=relaxed -A for-jobs/relaxed
# Make sure all jobtype=misc pods prefer to run any for-jobs assignments
valetctl assignment create miscpref prefer -t jobtype=misc -A for-jobs
# Create a Namespace
kubectl create namespace kv-example
# Enable kube-valet in the namespace (not required for global installations)
kubectl label namespace kv-example kube-valet.io/enabled=""
# Report by nodes
valetctl group report nodes
# Run some jobtype=sensitive pods
kubectl -n kv-example run sensitive --image=alpine:latest sleep 36000 -l jobtype=sensitive --replicas=5
# Check which nodes are in the sensitive assignment column
valetctl group report nags for-jobs
# Check that the all of the pods went to the expected nodes automatically
kubectl -n kv-example get pods -l jobtype=sensitive -o wide
# Run some jobtype=relaxed pods
kubectl -n kv-example run relaxed --image=alpine:latest sleep 36000 -l jobtype=relaxed --replicas=5
# Check which nodes are in the relaxed assignment column
valetctl group report nags for-jobs
# Check that all or most of the pods went to the expected nodes automatically
kubectl -n kv-example get pods -l jobtype=relaxed -o wide
# Run some jobtype=misc pods
kubectl -n kv-example run misc --image=alpine:latest sleep 36000 -l jobtype=misc --replicas=5
# Check which nodes are in any assignment
valetctl group report nags for-jobs
# Check that all or most of the pods went to the expected nodes automatically
kubectl -n kv-example get pods -l jobtype=misc -o wideClean up the test resources
kubectl delete nag for-jobs
kubectl delete cpar sensitivereq relaxedpref miscpref
kubectl delete namespace kv-exampleIt is possible to instruct kube-valet to always ignore specific pods or nodes.
- Kube-valet will ignore pods with a label of
pod.initializer.kube-valet.io/protected=true
- Kube-valet will ignore nodes with a label of
nags.kube-valet.io/protected=true
make- Golang 1.12+
- Code change
maketo build everything ormake buildto just build kube-valet and valetctl.- Run via
./build/kube-valet --kubeconfig ~/.kube/config --loglevel="DEBUG" --no-leader-elect --key=/path/to/key.pem --cert=/path/to/cert.pem
make docker-imagemake push-docker-image
PRs and issues are encouraged and welcome.
Kube-valet follows the CNCF Code of Conduct.
Copyright (c) 2018 Domo, Inc.
Kube-valet was originally created by Carson Anderson (@carsonoid) with additional help from Jordan Davidson (@from-nibly)