Oldest Kubernetes version supported: 1.16

⚠️ Kubernetes version 1.20.x is not supported! This Kubernetes issue prevents the controller from running properly on Kubernetes 1.20.0-1.20.4. Earlier versions of Kubernetes as well as 1.20.5 and later are still supported.

Chaos Controller

Datadog runs regular chaos experiments to test the resilience of our distributed cloud applications hosted in Kubernetes. The Chaos Controller facilitates automation of these experiments by simulating common "disruptions" including but not limited to: poor network quality, exhaustion of computational resources, or unexpected node failures. All you need to do to get started is define a yaml file which contains all of the specifications needed by our custom Kubernetes Resource to run the preferred disruption Kind.

Gamedays

At Datadog, we use the Chaos Controller to run Gamedays where we intentionally introduce failures into staging and production systems to test the resilience of our critical applications.

The Chaos Controller allows us to simulate a variety of small and large scenarios (and combination of scenarios) such as a network degradation across an AZ which limits access to critical DNS resolvers (top) or a bad configuration triggering a “thundering herd” on Consul's key-value store (bottom).

Local Development

Chaos Engineering is necessarily different from system to system. We encourage you to try out this tool, and extend it for your own use cases. If you want to run the source code locally to make and test implementation changes, visit the Contributing Docs which explain how to spin up chaos-controller source code on Colima. By the way, we welcome Pull Requests.

Getting Started

Disruptions are built as short-living resources which should be manually created and removed once your experiments are done. They should not be part of any application deployment. The Disruption resource is immutable. Once applied, you can't edit it. If you need to change the disruption definition, you need to delete the existing resource and to re-create it.

Getting started is as simple as creating a Kubernetes resource:

apiVersion: chaos.datadoghq.com/v1beta1
kind: Disruption
metadata:
  name: node-failure
  namespace: chaos-demo
spec:
  selector: # a label selector used to target resources
    app: demo-curl
  count: 1 # the number of resources to target
  duration: 1h # the amount of time before your disruption automatically terminates itself
  nodeFailure:
    shutdown: false # trigger a kernel panic on the target node

To disrupt your cluster, run kubectl apply -f <disruption_file.yaml>. You can clean up the disruption with kubectl delete -f <disruption_file>.yaml. For your safety, we recommend you get started with the dry-run mode enabled.

Check out Chaosli if you want some help understanding/creating disruption configurations.

:openbook: _The features guide contains usecases and sample disruption files!

:openbook: _The development guide contains tips for developing locally!

Useful Links

💣 Disclaimer 💣