Container Orchestration Benchmark

The purpose of the container orchestration benchmark (cnbm-co for short) is to provide a vendor-neutral, extendable benchmark for container orchestration systems. The current focus is on stateless workloads and we're implementing it for the following container orchestration systems (targets):

DC/OS
Kubernetes

If you want to contribute, simply fork this repo, add your implementation in pkg/ and send in a PR.

Contents:

Using
- Launching
- Availability matrix
Developing
Design

Using

Launching

In general:

$ ./cnbm-co launch -h
Launches the CNBM container orchestration benchmark

Usage:
  cnbm-co launch [flags]

Flags:
  -h, --help             help for launch
  -p, --params string    Comma separated key-value pair list of target-specific configuration parameters. For example: k1=v1,k2=v2
  -r, --runtype string   The benchmark run type. Allowed values: [scaling distribution apicalls servicediscovery recovery]
  -t, --target string    The target container orchestration system to benchmark. Allowed values: [dcos kubernetes]

Global Flags:
      --config string   config file (default is $HOME/.cnbm.yaml)

DC/OS

$ ./cnbm-co launch --runtype scaling --target dcos -p dcosurl=http://example.com,dcosacstoken=123
INFO[0000] Setting up DC/OS scaling benchmark
INFO[0000] Executing DC/OS scaling benchmark
INFO[0000] Deploying a new application
INFO[0000] RESULT:
 Target: DC/OS
 Output: {}
 Elapsed time: 0s

Kubernetes

To benchmark a Kubernetes cluster, use --target kubernetes, for example, to launch the scaling run type, do:

$ ./cnbm-co launch --runtype scaling --target kubernetes --params kubeconfig=/Users/mhausenblas/.kube/config,ns=cnbm,numpods=3,cpu=50m,mem=300Mi
INFO[0000] Setting up Kubernetes scaling benchmark
INFO[0000] Executing Kubernetes scaling benchmark
INFO[0006] Tearing down Kubernetes scaling benchmark
INFO[0008] RESULT:
 Target: Kubernetes
 Output: {benchmark succeeded}
 Elapsed time: 6.752147169s

Note the following params that are required, depending on the --runtype:

scaling
- kubeconfig … the Kubernetes config to use, for example /Users/mhausenblas/.kube/config
- ns … the namespace to use, for example cnbm
- numpods … the number of pods to launch, for example, 10
- cpu … the CPU seconds (at least 40m) per pod
- mem … the memory (at least 300Mi) per pod
servicediscovery
- kubeconfig … the Kubernetes config to use, for example /Users/mhausenblas/.kube/config
- ns … the namespace to use, for example cnbm

Availability matrix

The following matrix shows the availability of benchmark run types per target:

benchmark run type	DC/OS	Kubernetes
`scaling`	Y	Y
`distribution`	N	N
`distribution`	N	N
`apicalls`	N	N
`servicediscovery`	N	N

Developing

Building

$ make
Building the CNBM CO CLI
go build -ldflags "-X github.com/cnbm/container-orchestration/cli/cmd.releaseVersion=0.1.0" -o ./cnbm-co cli/main.go

Vendoring

We are using Go dep for dependency management. If you don't have dep installed yet, do go get -u github.com/golang/dep/cmd/dep now and then:

$ dep ensure

Testing

For unit tests we use the go test command, for example:

$ go test -v -short -run Test* .