PowerfulSeal

PowerfulSeal adds chaos to your Kubernetes clusters, so that you can detect problems in your systems as early as possible. It kills targeted pods and takes VMs up and down.

It follows the Principles of Chaos Engineering, and is inspired by Chaos Monkey.

Embrace the inevitable failure. Embrace The Seal.

Watch us introduce the Seal at Kubecon 2017 Austin

Highlights

works with OpenStack, AWS and local machines
speaks Kubernetes natively
interactive and autonomous, policy-driven mode
metric collection and exposition to Prometheus
minimal setup, easy yaml-based policies
easy to extend

Introduction

PowerfulSeal works in three modes: interactive, autonomous and label.

Interactive mode is designed to allow you to discover your cluster's components, and manually break things to see what happens. It operates on nodes, pods, deployments and namespaces.

Autonomous mode reads a policy file, which can contain any number of pod and node scenarios. Each scenario describes a list of matches, filters and actions to execute on your cluster.

Label mode allows you to specify which pods to kill with a small number of options by adding seal/ labels to pods. This is a more imperative alternative to autonomous mode.

Interactive mode

Here's a sneak peek of what you can do in the interactive mode:

Autonomous mode

Autonomous reads the scenarios to execute from the policy file, and runs them:

The matches are combined together and deduplicated to produce an initial working set
They are run through a series of filters
For all the items remaining after the filters, all actions are executed

Metric Collection

Autonomous mode also comes with the ability for metrics useful for monitoring to be collected. PowerfulSeal currently has a stdout and Prometheus collector. However, metric collectors are easily extensible so it is easy to add your own. More details can be found here.

Writing policies

A minimal policy file, doing nothing, looks like this:

config:
  minSecondsBetweenRuns: 77
  maxSecondsBetweenRuns: 100

nodeScenarios: []

podScenarios: []

The schemas are validated against the powerful JSON schema

A full featured example listing most of the available options can be found in the tests.

Label mode

Label mode is a more imperative alternative to autonomous mode, allowing you to specify which specific per-pod whether a pod should be killed, the days/times it can be killed and the probability of it being killed.

Instructions on how to use label mode can be found in LABELS.md.

Setup

Setup includes:

pointing PowerfulSeal at your Kubernetes cluster by giving it a Kubernetes config file
pointing PowerfulSeal at your cloud by specifying the cloud driver to use and providing credentials
making sure that PowerfulSeal can SSH into the nodes to execute commands on them
writing a set of policies

These interactions are available:

Getting started

PowerfulSeal is available to install through pip:

pip install powerfulseal
powerfulseal --help # or seal --help

Both Python 2.7 and Python 3 are supported.

Demo mode

The main way to use PowerfulSeal is to write a policy file for Autonomous mode which reflects realistic failures in your system. However, PowerfulSeal comes with a demo mode to demonstrate how it can cause chaos on your Kubernetes cluster. Demo mode gets all the pods in the cluster, selects those which are using the most resources, then kills them based on a probability.

Demo mode requires Heapster. To run demo mode, use the --demo flag along with --heapster-path (path to heapster without a trailing slash, e.g., http://localhost:8001/api/v1/namespaces-kube-system/services/heapster/proxy). You can also optionally specify --aggressiveness (from 1 (weakest) to 5 (strongest)) inclusive, as well as --[min/max]-seconds-between-runs.

Testing

PowerfulSeal uses tox to test with multiple versions on Python. The recommended setup is to install and locally activate the Python versions under tox.ini with pyenv.

Once the required Python versions are set up and can be discovered by tox (e.g., by having them discoverable in your PATH), you can run the tests by running tox.

More details in TESTING.md

Read about the PowerfulSeal

FAQ

Where can I learn more about Chaos Engineering ?

We found these two links to be a good start:

How is it different from Chaos Monkey ?

PowerfulSeal was inspired by Chaos Monkey, but it differs in a couple of important ways.

The Seal does:

speak Kubernetes
offer flexible, easy to write YAML scenarios
provide interactive mode with awesome tab-completion

The Seal doesn't:

need external dependencies (db, Spinnaker), apart from SSH, cloud and Kubernetes API access
need you to setup cron

Can I contribute to The Seal ?

We would love you to. In particular, it would be great to get help with:

get more cloud drivers (currently OpenStack and AWS)
get more awesome filters
get an amazing logo
make the PowerfulSeal more powerful

Check out our CONTRIBUTING.md file for more information about how to contribute.

Why a Seal ?

It might have been inspired by this comic.

rgodishela/powerfulseal