Vault Benchmarking Scripts

This repository contains some Lua scripts for running benchmarks against Vault with the wrk tool. They are all designed to be used with Vault's KV (Key/Value) v1 secrets engine.

This repo was forked from jdfriedma/vault-benchmarking, who forked from rberlind/vault-benchmarking I'm happy to add people to this one, and/or merge into the real HashiCorp org. It didn't appear Roger or Jacob were actively working on theirs.

You will first need to install wrk on your runner:

Scripts

The following are the main test scripts:

read-secrets.lua: This script randomly reads secrets from a set of N secrets under the path secret/read-test. The default value of N is 1,000. To change this, add "-- <\N>" after the URL where <N> is the number of secrets you want to use. If there are less secrets (at KV mount secret/read-test) than N, HTTP 404 will be returned and can skew test results. It can also print the secrets if you add "-- <N> true" after the URL. Use the write-secrets.lua script to populate the secrets read by this script before running it and check that you write all secrets you expect to read. The script reads them randomly over and over until it finishes.
write-random-secrets.lua: This script randomly writes secrets to a set of N secrets under the path secret/write-random-test. The default vaue of N is 1,000. By default, each secret has one key with 10-20 bytes and a second key with 100 bytes. To change the number of distinct secrets written, add "-- <\N>" after the URL where <N> is the number of secrets you want to use. The number and size of the keys could be changed, but you would need to edit the script. The script writes them randomly over and over until it finishes. There is no need to pre-populate Vault with any data for this test.
write-delete-secrets.lua: This script sequentially writes and deletes secrets. It must be run with one thread (-t1) and one connection (-c1) to ensure deletes do not reach the Vault server before the corresponding writes. However, multiple instances of this script can be run at the same time by passing an extra argument -- <n> after the URL, being sure to use a different value of <n> for each instance. Secrets for instance <n> of the script will be written in a sequential loop to the secret/write-delete-test path and will be named "test<n>-secret-<x>" where <x> is between 1 and N (default 100). This naming convention allows multiple instances of this script as well as other scripts to be run at the same time without conflict. By default, each secret has one key with 10-20 bytes and a second key with 100 bytes. The number of distinct secrets, N, can be changed by adding an extra argument after the script identifier. In this case, you would add "-- <identifier> <N>" after the URL, using integers for both of these arguments. The number and size of the keys could be changed, but you would need to edit the script. There is no need to pre-populate Vault with any data for this test. The last secret written might not be deleted if the final request is a write.
list-secrets.lua: This script repeatedly lists all secrets on the path secret/list-test. Use the write-list.lua script to populate that path with secrets. By default, that script writes 100 secrets to that path with each secret having one key with 10 bytes. If you want to print the secrets found in each list, add "-- true" after the URL.
authenticate-and-revoke.lua: This script repeatedly authenticates a user ("loadtester") against Vault's userpass authentication method and then revokes the acquired lease. (See below for instructions to enable it.)
write-pki.lua: This issues PKI certificates at the /pki/example_pki path. See below for PKI setup steps. This is not included in the run_all_tests_and_prelims.sh script currently. To run:
```
$ wrk -t2 -c2 -d10m -H "X-Vault-Token: $VAULT_TOKEN" -s write-pki.lua $VAULT_ADDR
```
encrypt-transit.lua: Transit encryptions. This one is a little different syntax. To prepare, enable transit and create a transit key with the CLI below:
```
$ vault secrets enable -path=transitTest transit 
$ vault write -f transitTest/keys/my-key 

Run for 1 minute, max of 500000 operations
$ wrk -t1 -c1 -d1m -H "X-Vault-Token: $VAULT_TOKEN" -s encrypt-transit.lua $VAULT_ADDR/v1/transitTest/encrypt/test encrypts=500000 
```
If you are running this on HCP Vault, you'll need the namespace specified (defaults to admin):
```
$ wrk -t4 -c8 -d1m -H "X-Vault-Token: $VAULT_TOKEN" -H "X-Vault-Namespace: admin" -s encrypt-transit.lua $VAULT_ADDR/v1/transitTest/encrypt/test encrypts=500000000

Note: debug output can be enabled with debug=true after encrypts, ie:
$ wrk -t1 -c1 -d1m -H "X-Vault-Token: $VAULT_TOKEN" -s encrypt-transit.lua $VAULT_ADDR/v1/transitTest/encrypt/test encrypts=500000 debug=true
```
I'm not seeing the performance I expect!
1. You should be running wrk from another, dedicated load-blasting instance. If you run wrk from the vault instance itself, you'll quickly max CPU.
2. If you find the wrk server is maxing CPU, try changing local transitPayloadLength = 1024 to 32 characters or something less. Wrk is creating a random string each time of this size, and just doing this can quickly max out an m5.large series AWS instance for example.

We also have the following utility scripts used to populate or delete secrets used by the test scripts:

write-secrets.lua: This script writes secrets meant to be read by the read-secrets.lua script. It writes a fixed number of secrets (default 1,000) and then stops. Each secret has one key with 10-20 bytes and a second key with 100 bytes. The number of secrets written can be changed by adding "-- <N>" after the URL where <N> is the number of secrets you want to write. The number and size of the keys could also be changed, but you would need to edit the script.
write-list.lua: This script writes a list of secrets each having one key with 10 bytes to the path secret/list-test. These secrets are read by the list-secrets.lua script. By default it writes 100 secrets, but you can change this by adding "-- <N>" after the URL where <N> is the number of secrets you want to write.
delete-secrets.lua: This deletes a sequence of secrets from under a specified path. Pass the path from which you want to delete secrets by adding something like "-- secret/read-test" after the Vault URL. Do not start your path with "/v1/" or add a final "/" at the end of it since the script does this for you. The default path is "secret/test".

Finally, json.lua is used by some of the other scripts to decode the JSON responses from the Vault HTTP API.

Running the Scripts together

The run_all_tests_and_prelims.sh bash script runs the read-secrets.lua, list-secrets.lua, authenticate-and-revoke.lua, and 10 instances of the write-delete-secrets.lua script simultaneously. The number of instances of the last script can be changed to alter the mixture of reads, writes, and deletes. It can be run from multiple clients simultaneously as long as no instances of the write-delete-secrets.lua across these clients use the same identifier argument (<n>). Instances of any of the scripts can be commented out by prefacing them with a "#". Note that while each instance of the write-delete-secrets.lua script writes to 100 secrets by default, they use distinct sets; so, the total number of secrets used by the 10 instances is 1,000. The read-secrets.lua script also uses 1,000 secrets and the list-secrets.lua script uses 100 secrets. So, the combined number of secrets used by the run_all_tests_and_prelims.sh script in its default configuration is 2,100.

In its default configuration, the run_all_tests_and_prelims.sh script is designed to run a mixture of reads, lists, writes, deletes, authentications, and revocations consisting of about 85% reads, 5% lists, 4% writes, 4% deletes, 1% authentications, and 1% revocations. Of course, these ratios will vary somewhat depending on your cluster's configuration even if you run the script without altering it. Making changes to the tests run in the script including removing or adding tests or changing the thread and count parameters passed to them will obviously change the mixture as well as the total throughput of your combined test.

In general, you will want to edit the run_all_tests_and_prelims.sh script or create modified versions of it before running it so you can change the duration of the tests (with the -d parameter) and change the names of the log files.

Setting up userpass Auth Method

In order to run the authenticate-and-revoke.lua script, you need to set up the Vault userpass authentication method on your Vault cluster and add a user called "loadtester" with password "benchmark". Use these commands to do this:

vault auth enable userpass
vault write auth/userpass/users/loadtester password=benchmark policies=default

Setting up secret KV

The scripts run again the KV engine at the secret path. This must be created prior.

$ vault secrets enable -path=secret kv
Success! Enabled the kv secrets engine at: secret/

Setting up PKI

To run the write-pki.lua script to issue PKI certificates you need to setup a few things first. This enables the PKI engine, sets the max lease to a year (from 30d default), and creates a role example_pki that allows any name certificate to be created. For further reading see PKI docs here.

$ vault secrets enable -path=pki-benchmarking pki
Success! Enabled the pki secrets engine at: pki-benchmarking/

$ vault secrets tune -max-lease-ttl=8760h pki-benchmarking
Success! Tuned the secrets engine at: pki-benchmarking/

$ vault write pki-benchmarking/roles/example_pki max_ttl=8760h allow_any_name=true
Success! Data written to: pki-benchmarking/roles/example_pki

$ vault write pki-benchmarking/root/generate/internal common_name=my-website.com ttl=8760h
(certification stuff returned)

Configuration of wrk Client Nodes

To ensure adequate resources on the client nodes that run wrk, we suggest using a Linux node with 4 CPUs and 8 GB of RAM. You can run more than 1 node. Before following the wrk Linux installation instructions for Ubuntu, you should run sudo apt-get update.

Examples of Running the Test Scripts

See the run_all_tests_and_prelims.sh script for examples of running most of the test scripts. You should export a Vault token with permissions to read, list, write, and delete the secrets used by the tests to the VAULT_TOKEN environment variable with the command export VAULT_TOKEN=<your_token>.

The only test script not included in run_all_tests_and_prelims.sh is write-random-secrets.lua. It could be run and configured to write to 10,000 secrets with a command like:

nohup wrk -t4 -c16 -d1h -H "X-Vault-Token: $VAULT_TOKEN" -s write-random-secrets.lua http://<vault_url>:8200 -- 10000 > prod-test-write-1000-random-secrets-t4-c16-1hour.log &

We use "nohup" on the test scripts to ensure that the scripts continue to run if our ssh session to the node running the wrk client gets disconnected and output the results to files with the names of the files indicating the parameters used and how long they ran for.

If you want the read-secrets.lua and list-secrets.lua scripts to print the secrets they retrieve, add -- true after the Vault URL.

When running multiple instances of the write-delete-secrets.lua script simultaneously, be sure to add the argument -- <n> after the URL and to use a different value of <n> for each instance. Please also always run this script with one thread (-t1) and one connection (-c1) to ensure deletes do not reach the Vault server before the corresponding writes.

Descriptions of the wrk command line options are here.

Notes on the parameters:

The "-t" parameter gives the number of threads.
The "-c" parameter gives the number of HTTP connections used by all threads.
The "-d" parameter gives the number of seconds (s), minutes (m) or hours (h) to run the test.

Examples of Running the Utility Scripts

Here are example of running the utility scripts to populate and delete secrets needed by the test scripts:

# Command to write 1,000 secrets needed by the read-secrets.lua script:
wrk -t1 -c1 -d1m -H "X-Vault-Token: $VAULT_TOKEN" -s write-secrets.lua http://<vault_url>:8200 -- 1000

# Command to write secrets needed by the list-secrets.lua script:
wrk -t1 -c1 -d1m -H "X-Vault-Token: $VAULT_TOKEN" -s write-list.lua http://<vault_url>:8200 -- 100

# Command to delete secrets (from secret/read-test)
wrk -t1 -c1 -d1m -H "X-Vault-Token: $VAULT_TOKEN" -s delete-secrets.lua http://<vault_url>:8200 -- secret/read-test

Note that you should specify the path from which you want to delete secrets when running the delete-secrets.lua script by adding it after the URL. The default value is "secret/test".

mikegreen/vault-benchmarking