PowerfulSeal adds chaos to your Kubernetes clusters, so that you can detect problems in your systems as early as possible. It kills targeted pods and takes VMs up and down.
It follows the Principles of Chaos Engineering, and is inspired by Chaos Monkey.
Embrace the inevitable failure. Embrace The Seal.
Watch us introduce the Seal at Kubecon 2017 Austin
- works with
OpenStack
,AWS
and local machines - speaks
Kubernetes
natively - interactive and autonomous, policy-driven mode
- metric collection and exposition to
Prometheus
- minimal setup, easy yaml-based policies
- easy to extend
PowerfulSeal works in three modes: interactive, autonomous and label.
Interactive mode is designed to allow you to discover your cluster's components, and manually break things to see what happens. It operates on nodes, pods, deployments and namespaces.
Autonomous mode reads a policy file, which can contain any number of pod and node scenarios. Each scenario describes a list of matches, filters and actions to execute on your cluster.
Label mode allows you to specify which pods to kill with a small number of options by adding seal/
labels to pods. This is a more imperative alternative to autonomous mode.
Here's a sneak peek of what you can do in the interactive mode:
Autonomous reads the scenarios to execute from the policy file, and runs them:
- The matches are combined together and deduplicated to produce an initial working set
- They are run through a series of filters
- For all the items remaining after the filters, all actions are executed
Autonomous mode also comes with the ability for metrics useful for monitoring to be collected. PowerfulSeal currently has a stdout
and Prometheus collector. However, metric collectors are easily extensible so it is easy to add your own. More details can be found here.
A minimal policy file, doing nothing, looks like this:
config:
minSecondsBetweenRuns: 77
maxSecondsBetweenRuns: 100
nodeScenarios: []
podScenarios: []
The schemas are validated against the powerful JSON schema
A full featured example listing most of the available options can be found in the tests.
Label mode is a more imperative alternative to autonomous mode, allowing you to specify which specific per-pod whether a pod should be killed, the days/times it can be killed and the probability of it being killed.
Instructions on how to use label mode can be found in LABELS.md.
Setup includes:
- pointing PowerfulSeal at your Kubernetes cluster by giving it a Kubernetes config file
- pointing PowerfulSeal at your cloud by specifying the cloud driver to use and providing credentials
- making sure that PowerfulSeal can SSH into the nodes to execute commands on them
- writing a set of policies
These interactions are available:
PowerfulSeal
is available to install through pip:
pip install powerfulseal
powerfulseal --help # or seal --help
Both Python 2.7 and Python 3 are supported.
The main way to use PowerfulSeal is to write a policy file for Autonomous mode which reflects realistic failures in your system. However, PowerfulSeal comes with a demo mode to demonstrate how it can cause chaos on your Kubernetes cluster. Demo mode gets all the pods in the cluster, selects those which are using the most resources, then kills them based on a probability.
Demo mode requires Heapster. To run demo mode, use the --demo
flag along with --heapster-path
(path to heapster without a trailing slash, e.g., http://localhost:8001/api/v1/namespaces-kube-system/services/heapster/proxy
). You can also optionally specify --aggressiveness
(from 1
(weakest) to 5
(strongest)) inclusive, as well as --[min/max]-seconds-between-runs
.
PowerfulSeal uses tox to test with multiple versions on Python. The recommended setup is to install and locally activate the Python versions under tox.ini
with pyenv.
Once the required Python versions are set up and can be discovered by tox (e.g., by having them discoverable in your PATH), you can run the tests by running tox
.
More details in TESTING.md
- https://www.techatbloomberg.com/blog/powerfulseal-testing-tool-kubernetes-clusters/
- https://siliconangle.com/blog/2017/12/17/bloomberg-open-sources-powerfulseal-new-tool-testing-kubernetes-clusters/
- https://github.com/ramitsurana/awesome-kubernetes#testing
- https://github.com/ramitsurana/awesome-kubernetes#other-useful-videos
- https://github.com/dastergon/awesome-chaos-engineering#notable-tools
- https://www.linux.com/news/powerfulseal-testing-tool-kubernetes-clusters-0
- https://www.infoq.com/news/2018/01/powerfulseal-chaos-kubernetes
We found these two links to be a good start:
PowerfulSeal was inspired by Chaos Monkey, but it differs in a couple of important ways.
The Seal does:
- speak Kubernetes
- offer flexible, easy to write YAML scenarios
- provide interactive mode with awesome tab-completion
The Seal doesn't:
- need external dependencies (db, Spinnaker), apart from SSH, cloud and Kubernetes API access
- need you to setup
cron
We would love you to. In particular, it would be great to get help with:
- get more cloud drivers (currently
OpenStack
andAWS
) - get more awesome filters
- get an amazing logo
- make the PowerfulSeal more powerful
Check out our CONTRIBUTING.md file for more information about how to contribute.
It might have been inspired by this comic.