Simulator

A distributed systems and infrastructure simulator for attacking and debugging Kubernetes: simulator creates a kuberntes cluster for you in your AWS account; runs scenarios which misconfigure it and/or leave it vulnerable to compromise and trains you in mitigating against these vulnerabilities.

For details on why we created this project and our ultimate goals take a look at the vision statement.

Before you start

Usage

Ensure the AWS requirements are met and configured.

The quickest way to get up and running is to simply:

source <(curl https://raw.githubusercontent.com/kubernetes-simulator/simulator/master/kubesim)

or clone this repository and run:

make run

This will drop you into a bash shell in a launch container. You will have a program on the PATH named simulator to interact with.

Refer to Simulator CLI Usage

Simulator CLI Usage

Creating Environment And Lauching Scenario

Create a remote state bucket for terraform

simulator init

You will be asked for a name of a S3 bucket, which must be globally unique as pre AWS standards. If this does not exist it will be created, else the existing bucket will be used. This is used to store Terraform state to.

Create the infra if it isn't there

simulator infra create

This will standup the infrastructure, including an initial Kubernetes Cluster

List available scenarios

simulator scenario list

This will list all currently available scenrios

Launch a scenario (sets up your cluster)

simulator scenario launch node-shock-tactics

This will launch your selected scenario.

Login the environment

simulator ssh attack

Once you run simulator ssh attack you will be logged into a container running on the Bastion host. Upon login an outline of the challenge will be displayed. In addition, short cuts for logging into the master, or nodes, of the Kubernetes cluster are displayed. Use these shortcuts to log into the correct starting point as outlined in the challenge. Note that some starting points will require you are on the cluster first to access to starting point.

From within this container you have access to helper command start_task, next_hint and show_hints.

Start task

start_task 1

The start_task command is used to inform the simulator which task you are undertaking, and therefore what hints are available to aid you with that task.

Accessing hints

next_hint

The next_hint command will provide a hint to help you complete the task you have started with the start_task command.

Viewing all hints that have been requested

show_hints

The show_hints command will display all the hints you have requested to that point, in the task you have started.

Cleaning Up Environment

Destroy your cluster when you are done

simulator infra destroy

Once you have finished you must destroy the environment to ensure that no additional costs are incurred. This command is actioned from within the launch container.

Scenarios

The following scenarios are available:

     container-ambush
     container-defeat-in-detail
     container-phalanx-formation
     etcd-inverted-wedge
     master-bait-and-bleed
     master-encirclement
     network-feint
     network-hammer-and-anvil
     network-hedgehog-defence
     network-swarming
     node-amphibious-operations
     node-raiding
     node-shock-tactics
     policy-echelon-formation
     policy-fire-support
     policy-force-dispersal
     policy-vertical-envelopment
     rbac-contact-drill
     rbac-sangar
     rbac-flanking-maneuver
     rbac-shoot-and-scoot
     secret-high-ground
     secret-tank-desant

How It All Works

Infrastructure Design

AWS Configuration

Simulator uses terraform to provision its infrastructure. Terraform in turn honours all the rules for configuring the AWS CLI. Please follow these.

You can provide your credentials via the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, environment variables, representing your AWS Access Key and AWS Secret Key, respectively. Note that setting your AWS credentials using either these (or legacy) environment variables will override the use of AWS_SHARED_CREDENTIALS_FILE and AWS_PROFILE. The AWS_DEFAULT_REGION and AWS_SESSION_TOKEN environment variables are also used, if applicable.

https://www.terraform.io/docs/backends/types/s3.html

All the AWS_* configuration environment variables you have set will be propagated into the container

Troubleshooting AWS

If you get a timeout when running simulator infra create after about 10 minutes, the region you are using is probably running slowly. You must run simulator infra destroy and then retry simulator infra create
AWS_REGION vs AWS_DEFAULT_REGION - There have been some issues with the Go AWS client region configuration
Multi-account

Terraform

Refer to the simulator terraform documentation

SSH

Simulator whether run in the launch container or on the host machine will generate its own SSH RSA key pair. It will configure the cluster to allow access only with this keypair and automates writing SSH config and keyscanning the bastion on your behalf using custom SSH config and known_hosts files. This keeps all simulator-related SSH config separate from any other configs you may have. All simulator-related SSH files are written to ~/.ssh and are files starting cp_simulator_

If you delete any of the files then simulator will recreate them and reconfigure the infrastructure as necessary on the next run

The kubesim script

kubesim is a small script written in BASH for getting users up and running with simulator as fast as possible. It pulls the latest version of the simulator container and sets up some options for running the image. It can be installed with the following steps:

# cURL the script from GitHub
curl -o kubesim https://raw.githubusercontent.com/kubernetes-simulator/simulator/master/kubesim
# Make it executeable
chmod a+x kubesim
# Place the script on your path
cp kubesim /usr/local/bin

Development Workflow

Git hooks

To ensure all Git hooks are in place run the following

make setup-dev

Development targets are specified in the Makefile.

setup-dev             Initialise simulation tree with git hooks
devtools-deps: # Install devtools dependencies 
devtools              Install devtools
test-devtools: # Run all the unit tests for the devtools 
reset                 Clean up files left over by simulator
validate-requirements  Verify all requirements are met
previous-tag:        
release-tag:         
gpg-preflight:       
run                   Run the simulator - the build stage of the container runs all the cli tests
docker-build-no-cache  Builds the launch container
docker-build          Builds the launch container
docker-test           Run the tests
dep                   Install dependencies for other targets
build                 Run golang build for the CLI program
is-in-launch-container  checks you are running in the launch container
test                  run all tests except goss tests
test-acceptance       Run tcl acceptance tests for the CLI program
test-smoke            Run expect smoke test to check happy path works end-to-end
test-unit             Run golang unit tests for the CLI program
test-cleanup          cleans up automated test artefacts if e.g. you ctrl-c abort a test run
test-coverage         Run golang unit tests with coverage and opens a browser with the results
docs                  Generate documentation
release: gpg-preflight previous-tag release-tag docker-test docker-build build

Git commits

We follow the conventional commit specification. Please ensure your commit messages adhere to this spec. If you submit a pull request without following them be aware we will squash and merge your work with a message that does.

Architecture

Launching a scenario

TODO Validating a scenario

TODO Evaluating scenario progress

Components

Simulator CLI tool - Runs in the launch container and orchestrates everything
Launch container - Isolates the scripts from the host
Terraform Scripts for infrastructure provisioning - AWS infra
Perturb.sh - sets up a scenario on a cluster
Scenarios
Attack container - Runs on the bastion providing all the tools needed to attack a cluster in the given cloud provider

Specifications

tasks.yaml format
If you need to make changes to the format you should update this documentation.
Any changes should be accompanied by a bump of the version in the kind property
Use the migrate-hints devtool to update the existing scenarios en-masse. You can make this tool available on your PATH by running make devtools

Simulator API Documentation

The simulator API docs are available on godoc.org:

Contributing

Guidelines for contributors, and instructions on working with specific parts of Simulator are in the Contributor Guide.

Temikus/simulator