ExtendedDaemonSet
!!! Alpha version !!!
ExtendedDaemonSet aims to provide a new implementation of the Kubernetes DaemonSet resource with key features:
- Canary Deployment: Deploy a new DaemonSet version with only a few nodes.
- Custom Rolling Update: Improve the default rolling update logic available in Kubernetes
batch/v1 Daemonset.
How to use it
Deployment
To use the ExtendedDaemonSet controller in your Kubernetes cluster, deploy its resources definitions in the /deploy folder.
First, deploy the Custom Resources Definitions:
$ kubectl apply -f ./deploy/crdsIf you want to deploy a specific controller version, the version can be set in the deploy/operator.yaml file. By default, the latest docker image tag is used.
$ kubectl apply -f ./deploy/By default, the controller only watches the ExtendedDaemonSet resources that are present in its own namespace. If you want to deploy the controller cluster wide, modify the deploy/operator.yaml to remove or set an empty value for the WATCH_NAMESPACE environment variable:
env:
- name: WATCH_NAMESPACE
value: ""Demo application
If you want to test and compare the advantages of the ExtendedDaemonSet over the the standard DaemonSet, you can use the demo application available in the /example folder. Follow the below scenario:
First, you need a Kubernetes cluster with several nodes; we recommend using three nodes. If you want, you can use kind.sigs.k8s.io to create a three node cluster with the following command: kind create cluster --config examples/kind-cluster-configuration.yaml.
This creates a three node cluster with one master and two worker nodes:
$ kind create cluster --config examples/kind-cluster-configuration.yaml
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.15.3) 🖼
✓ Preparing nodes 📦📦📦
✓ Creating kubeadm config 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
✓ Joining worker nodes 🚜
Cluster creation complete. You can now use the cluster with:
$ export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
ExtendedDaemonSet controller deployment
# deploy the controller needed crds
$ kubectl apply -f ./deploy/crds
customresourcedefinition.apiextensions.k8s.io/extendeddaemonsets.datadoghq.com created
customresourcedefinition.apiextensions.k8s.io/extendeddaemonsetreplicasets.datadoghq.com created
# deploy the controller pod
$ kubectl apply -f ./deploy/
clusterrole.rbac.authorization.k8s.io/extendeddaemonset created
clusterrolebinding.rbac.authorization.k8s.io/extendeddaemonset created
deployment.apps/extendeddaemonset created
role.rbac.authorization.k8s.io/extendeddaemonset created
rolebinding.rbac.authorization.k8s.io/extendeddaemonset created
serviceaccount/extendeddaemonset created
# you should see the extendeddaemonset controller pod running
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
extendeddaemonset-855cd7c679-gpmql 1/1 Running 0 2m11s
foo ExtendedDaemonSet deployment
Create the foo app with the ExtendedDaemonSet. For demo purposes, we'll use the k8s.gcr.io/pause Docker image, which is only awaiting a terminating signal. You can look at the foo application definition in the file examples/foo-eds_v1.yaml.
$ kubectl apply -f examples/foo-eds_v1.yaml
extendeddaemonset.datadoghq.com/foo createdYou can see the state of the ExtendedDaemonSet foo with:
$ kubectl get eds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE STATUS ACTIVE RS CANARY RS AGE
foo 3 3 3 3 3 Running foo-8z7lr 44s
# Also the `extendeddaemonsetreplicaset` resource generated by the controller from the `foo` EDS instance:
$ kubectl get ers
NAME STATUS DESIRED CURRENT READY AVAILABLE NODE SELECTOR AGE
foo-8z7lr active 3 3 3 3 61s
foo ExtendedDaemonSet deployment update with canary strategy
Now we can try to update the ExtendedDaemonSet foo. The only difference between the two versions is the Docker image used in the pod template.
$ diff examples/foo-eds_v1.yaml examples/foo-eds_v2.yaml
17c17
< image: k8s.gcr.io/pause:3.0
---
> image: k8s.gcr.io/pause:3.1
$ kubectl apply -f examples/foo-eds_v2.yaml
extendeddaemonset.datadoghq.com/foo configuredAs you can see with the following command, a canary ReplicaSet is now configured for the foo ExtendedDaemonSet. Additionally, a new ExtendedReplicaSet has been created to handle the new foo pod template version.
$ kubectl get eds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE STATUS ACTIVE RS CANARY RS AGE
foo 3 3 3 3 3 Canary foo-8z7lr foo-xdj4b 85s
$ kubectl get ers
NAME STATUS DESIRED CURRENT READY AVAILABLE NODE SELECTOR AGE
foo-8z7lr active 2 2 2 2 2m
foo-xdj4b canary 1 1 1 1 40s
$ kubectl get pod -l extendeddaemonset.datadoghq.com/name=foo
NAME READY STATUS RESTARTS AGE
foo-8z7lr-bp9w8 1/1 Running 0 108s
foo-8z7lr-jlvrq 1/1 Running 0 88s
foo-xdj4b-zvss2 1/1 Running 0 8sOnly one pod is running with the ExtendedReplicaSet foo-xdj4b pod template version. This corresponds to the setting spec.canary.replicas in the ExtendedDaemonSet foo.
Rolling update after the canary deployment validation period ended
After 5 minutes, which corresponds to spec.canary.duration, the controller will set as valid and activate the foo-xdj4b ExtendedReplicaSet. It will trigger the full foo-xdj4b ExtendedReplicaSet deployment.
$ kubectl get eds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE STATUS ACTIVE RS CANARY RS AGE
foo 3 3 3 3 3 Running foo-xdj4b 9m21s
$ kubectl get ers
NAME STATUS DESIRED CURRENT READY AVAILABLE NODE SELECTOR AGE
foo-xdj4b active 3 3 3 3 8m21s
$ kubectl get pod -l extendeddaemonset.datadoghq.com/name=foo
NAME READY STATUS RESTARTS AGE
foo-xdj4b-hh6d8 1/1 Running 0 5m11s
foo-xdj4b-rgtk9 1/1 Running 0 5m31s
foo-xdj4b-zvss2 1/1 Running 0 10mKubectl plugin
To build the the kubectl ExtendedDaemonSet plugin, you can run the command: make build-plugin. This will create the kubectl-eds Go binary, corresponding to your local OS and architecture.
Then, add or move this binary to the PATH and run the command kubectl eds:
$ kubectl eds
Usage:
ExtendedDaemonset [command]
Available Commands:
get get ExtendedDaemonSet deployment(s)
get-ers get-ers ExtendedDaemonSetReplicaset deployment(s)
help Help about any command
validate validate canary replicasetHow to migrate from a DaemonSet
If you already have an application running in your cluster with a DaemonSet, it is possible to migrate to an ExtendedDaemonSet with a smooth migration path.
-
Update your
DaemonSetspecification to set a toleration that does not correspond to your node's taints. As a result, theDaemonSetpods that are already running will not be deleted, and theDaemonSetcontroller will not take any new actions on it. -
In the ExtendedDaemonSet definition, add a specific annotation to inform the
extendeddaemonset-controllerwhichDaemonSetneeds to be migrated. The controller will recognize the pods from the "old" DaemonSet as a previous version and will do a proper rolling update.
apiVersion: datadoghq.com/v1alpha1
kind: ExtendedDaemonSet
metadata:
name: foo
annotations:
extendeddaemonset.datadoghq.com/old-daemonset: foo
spec:
# ...Developers section
How to build it
This project uses go module. Ensure you have it activated: export GO111MODULE=on.
Run make install-tools to install mandatory tooling, like the operator-fwk command line and the golangci linter.
$ make build
CGO_ENABLED=0 go build -i -installsuffix cgo -ldflags '-w' -o controller ./cmd/manager/main.goHow to test it
Unit tests
$ make test
./go.test.sh
ok github.com/datadog/extendeddaemonset/pkg/controller/extendeddaemonset 1.107s coverage: 77.0% of statements
ok github.com/datadog/extendeddaemonset/pkg/controller/extendeddaemonsetreplicaset 1.098s coverage: 63.9% of statements
ok github.com/datadog/extendeddaemonset/pkg/controller/extendeddaemonsetreplicaset/strategy 1.036s coverage: 5.3% of statements
ok github.com/datadog/extendeddaemonset/pkg/controller/extendeddaemonsetreplicaset/strategy/limits 1.016s coverage: 83.3% of statements
ok github.com/datadog/extendeddaemonset/pkg/controller/utils 1.015s coverage: 100.0% of statementsEnd to end tests
For end to end tests, consult /test/README.md
Linter validation
To use the linter, run:
$ make validate
./bin/golangci-lint run ./...