Simulator
A distributed systems and infrastructure simulator for attacking and debugging
Kubernetes: simulator
creates a Kubernetes cluster for you in your AWS
account; runs scenarios which misconfigure it and/or leave it vulnerable to
compromise and trains you in mitigating against these vulnerabilities.
For details on why we created this project and our ultimate goals take a look at the vision statement.
Before you start
Usage
Ensure the AWS requirements are met and configured.
Clone this repository and run:
make run
This will drop you into a bash shell in a launch container. You will have a
program on the PATH
named simulator
to interact with.
Simulator CLI Usage
Creating Environment And Lauching Scenario
Before you launch your environment, please see the How It All Works section to ensure you have the necessary credentials and permissions in place and know what you are standing up.
Create a remote state bucket for terraform
simulator init
You will be asked for the name for a S3 bucket for the Terraform remote state, which must be globally unique as per AWS standards. If this does not exist it will be created, otherwise the existing bucket will be used.
Create the infra if it isn't there
simulator infra create
This will standup the infrastructure, including an initial Kubernetes Cluster
List available scenarios
simulator scenario list
This will list all currently available scenarios. You can filter the list by difficulty or category using the appropriate arguments
To get a better idea of what is involved in each scenario, e.g. node-shock-tactics
simulator scenario describe node-shock-tactics
Launch a scenario (sets up your cluster)
simulator scenario launch node-shock-tactics
This will launch your selected scenario.
Login to the environment
simulator ssh attack
Running simulator ssh attack
logs you into a container running on the Bastion host. Upon login, an outline of the challenge will be displayed. In addition, shortcuts for logging into the master, or nodes, of the Kubernetes cluster and how to show hints and start tasks are displayed. starting_point
logs you into the correct starting point for the task you have started in the scenario.
From within the ssh attack container you have access to a range of helper commands such as start_task
, next_hint
and show_hints
.
Start task
start_task 1
The start_task
command is used to inform the simulator which task you are undertaking with the number associated, and therefore what hints are available to aid you with that task.
Accessing hints
next_hint
The next_hint
command will provide a hint to help you complete the task you have started with the start_task
command.
Viewing all hints that have been requested
show_hints
The show_hints
command will display all the hints you have requested to that point, in the task you have started.
Ending a task
end_task
or
start_task 2
The end_task
command will end the task you are currently on and score you. This will also happen if you move onto the a new task with the start_task
command. When you end a task in one of these ways, you will be scored 100 points minus the amount of hints you have displayed for that task at a value of -10 each. This is defined in the scenario task manifest.
Cleaning Up Environment
Destroy your cluster when you are done
simulator infra destroy
Once you have finished you should destroy the environment to ensure that no additional costs are incurred. You run this command in the launch container.
Scenarios
The following scenarios are currently shipped with the simulator:
container-ambush container-defeat-in-detail container-phalanx-formation etcd-inverted-wedge master-shell-scrape master-encirclement network-feint network-hammer-and-anvil network-hedgehog-defence network-swarming node-amphibious-operations node-raiding node-shock-tactics policy-echelon-formation policy-fire-support policy-force-dispersal policy-vertical-envelopment rbac-contact-drill rbac-sangar rbac-flanking-maneuver rbac-shoot-and-scoot secret-high-ground secret-tank-desant
But you can write your own and we really welcome any contributions of new scenarios :)
How It All Works
Infrastructure Design
AWS Configuration
Simulator uses terraform to provision its infrastructure. Terraform in turn honours all the rules for configuring the AWS CLI. Please follow these.
You can provide your credentials via the AWS_ACCESS_KEY_ID
and
AWS_SECRET_ACCESS_KEY
environment variables, containing your AWS Access Key
and AWS Secret Key respectively. Note that setting your AWS credentials using
either these (or legacy) environment variables will override the use of
AWS_SHARED_CREDENTIALS_FILE
and AWS_PROFILE
. The AWS_DEFAULT_REGION
and
AWS_SESSION_TOKEN
environment variables are also used, if applicable.
https://www.terraform.io/docs/backends/types/s3.html
All the AWS_*
configuration environment variables you have set will be propagated into the container
Troubleshooting AWS
- If you get a timeout when running
simulator infra create
after about 10 minutes, the region you are using is probably running slowly. You should runsimulator infra destroy
and then retrysimulator infra create
AWS_REGION
vsAWS_DEFAULT_REGION
- There have been some issues with the Go AWS client region configuration- Multi-account
Terraform
Refer to the simulator terraform documentation
SSH
Simulator, whether run in the launch container or on the host machine, will generate its own SSH RSA key pair. It will configure the cluster to allow access only with this keypair and automates writing SSH config and keyscanning the bastion on your behalf using custom SSH config and known_hosts files. This keeps all simulator-related SSH config separate from any other SSH configs you may have. All simulator-related SSH files are written to ~/.kubesim
and are files starting cp_simulator_
If you delete any of the files then simulator will recreate them and reconfigure the infrastructure as necessary on the next run
Scenario Launching (perturb)
Refer to the scenario launch documentation
Launch Flow Sequence
Refer to the Launch scenario flow documentation
Scenario Task Definition
Refer to the Tasks YAML file Format documentation
Create a new scenario
Refer to the Create Scenario documentation
The kubesim script
kubesim
is a small script written in BASH for getting users up and running with simulator as fast as possible. It pulls the latest version of the simulator container and sets up some options for running the image. It can be installed with the following steps:
# cURL the script from GitHub curl -Lo kubesim https://github.com/kubernetes-simulator/simulator/releases/latest/download/kubesim # Make it executeable chmod a+x kubesim # Place the script on your path cp kubesim /usr/local/bin
Development Workflow
Git hooks
To ensure all Git hooks are in place run the following:
make setup-dev
Development targets are specified in the Makefile.
You can see all the available targets with descriptions by running make help
Git commits
We follow the conventional commit specification. Please ensure your commit messages adhere to this spec.
Architecture
Launching a scenario
TODO Validating a scenario
TODO Evaluating scenario progress
Components
- Simulator CLI tool - Runs in the launch container and orchestrates everything
- Launch container - Isolates the scripts from the host
- Terraform Scripts for infrastructure provisioning - AWS infrastructure
- Perturb.sh - Sets up a scenario on a cluster
- Scenarios - The scenario setup scripts and definitions
- Attack container - Runs on the bastion providing all the tools needed to attack a cluster in the given cloud provider
Specifications
-
If you need to make changes to the format you should update this documentation.
-
Any changes should be accompanied by a bump of the version in the
kind
property -
Use the
migrate-hints
devtool to update the existing scenarios en-masse. You can make this tool available on your PATH by runningmake devtools
Simulator API Documentation
The simulator API docs are available on godoc.org:
Contributing
Guidelines for contributors, and instructions on working with specific parts of Simulator are in the Contributor Guide.