chaos-fn

A function for chaos testing with OpenFaaS

Use-cases

Test retries on certain HTTP codes
Test timeouts
Test certain lengths of HTTP request bodies

Setup

Install openfaas:

kind cluster create

arkade install openfaas \
  --set queueWorker.image=alexellis2/pro-queue-worker-demo:0.2.1

This demo image expires on 7th August 20121, feel free to request an extension.

Deploy the function in this repository:

faas-cli deploy -f chaos-fn.yml

Experiment 1

Cause the API to start failing

curl -i localhost:8080/function/chaos-fn/set --data-binary '
{	"status": 500,
	"delay": "1s",
    "body": "1 second delay, then 500"}
' --header "Content-type: application/json"

Observe it:

curl -i localhost:8080/function/chaos-fn

Fix it

curl -i localhost:8080/function/chaos-fn/set --data-binary '
{"status": 200,
	"delay": "1ms",
    "body": "1ms second delay, then 200"}
' --header "Content-type: application/json"

curl -i localhost:8080/function/chaos-fn

Experiment 2 - the PRO async queue worker

Set up a HTTP code that may be retried

curl -i localhost:8080/function/chaos-fn/set --data-binary '
{	"status": 500,
	"delay": "1s",
    "body": "1 second delay, then 500"}
' --header "Content-type: application/json"

View the logs:

kubectl logs deploy/queue-worker -n openfaas -f

Now invoke:

curl -i localhost:8080/async-function/chaos-fn -d ""

Observe the retrying mechanism.

Whenever you like, fix the error to allow the next retry to complete:

curl -i localhost:8080/function/chaos-fn/set --data-binary '
{"status": 200,
	"delay": "1ms",
    "body": "1ms second delay, then 200"}
' --header "Content-type: application/json"

Experiment 3 - set a timeout that expires

Setup a 502 error which may be seen when a function gives a timeout during scale from zero or its first invocation immediately after a deployment:

curl -i localhost:8080/function/chaos-fn/set --data-binary '
{	"status": 502,
	"delay": "5m",
    "body": "5m delay, then 502"}
' --header "Content-type: application/json"

I've picked 5 minutes 5m, but you will need to pick a number that corresponds to your extended timeouts. Bear in mind that you may also have to extend the timeout of this function itself in the chaos-fn.yml file before running faas-cli deploy again.