About

Repo for the Kubernetes Leipzig User Group meeting 05-04-2023 showcasing Kubernetes SIG e2e-framework.

The e2e-framework in this repo is used to run a suite of end-to-end tests:

Showcase how to use the framework for business applications
- Deploy a NATS JetStream server
- Deploy a message publisher (Golang app)
- Deploy a message consumer (Golang app)
- Assert the producer sends messages (*)
- Assert the consumer receives messages (*)
- Assert successful cleanup of test resources
Showcase how to use the framework with Kubernetes controllers/operators
- Deploy the AWS ACK Controller for EventBridge
- Create an EventBus resource
- Assert the resource is synchronized and successfully created in the AWS service control plane (backend)
- Assert successful cleanup of test resources

(*) The producer and consumer expose an HTTP health check which is flipped when a message was successfully sent/received. The pods will only enter the ready state if the health check probe succeeds, i.e., the deployment is considered available which is reflected in the resource's status and can be asserted on with a Wait() function in the framework. This is a very naive way of doing this, but shows how to use Kubernetes conditions for asynchronous assertations (works very nicely with Kubernetes Jobs).

Tools and Versions used

ko (0.13.0)
helm (v3.11.2)
kind (v0.18.0 - uses Kubernetes 1.26)
go (go1.20.3)
Docker (20.10.23)

Run the E2E Tests

NATS

This E2E test asserts that NATS is running in Kubernetes (deployed via helm as part of the test suite), a publisher Kubernetes deployment (Go) can send messages to a NATS JetStream topic, and a subscriber Kubernetes deployment (Go) successfully consumes messages from the stream.

# create kind cluster
kind create cluster --name e2e-meetup
export KIND_CLUSTER_NAME=e2e-meetup 
export KO_DOCKER_REPO=kind.local

# build and upload images to kind
# replace platform with your environment
export PUBLISHER_IMAGE=$(ko build -B --platform=linux/arm64 ./publisher) \
export SUBSCRIBER_IMAGE=$(ko build -B --platform=linux/arm64 ./subscriber)

# run nats tests
go test -race -count=1 -v ./e2e -args -v 4 -labels=feature=e2e-nats

Your output should be similar to

I0419 13:32:41.075876   72332 main_test.go:39] setting up test environment with kind cluster "e2e-meetup"
I0419 13:32:41.076098   72332 kind.go:93] Creating kind cluster e2e-meetup
I0419 13:32:41.867656   72332 kind.go:99] Skipping Kind Cluster.Create: cluster already created: e2e-meetup
=== RUN   TestSimpleE2E
I0419 13:32:52.556332   72332 helper_test.go:31] creating namespace "e2e-feature-f53" for feature "e2e demo with nats"
=== RUN   TestSimpleE2E/e2e_demo_with_nats
I0419 13:32:52.567830   72332 suite_test.go:48] adding nats repo "nats" using chart "https://nats-io.github.io/k8s/helm/charts/"

<snip>

=== RUN   TestSimpleE2E/e2e_demo_with_nats/nats_server_running
I0419 13:32:53.772049   72332 suite_test.go:80] waiting for nats "nats-server" in namespace "e2e-feature-f53" to become ready
=== RUN   TestSimpleE2E/e2e_demo_with_nats/publisher_running
I0419 13:33:08.791521   72332 suite_test.go:99] creating deployment "publisher"
I0419 13:33:08.812503   72332 suite_test.go:106] waiting for deployment "publisher" in namespace "e2e-feature-f53" to become ready
=== RUN   TestSimpleE2E/e2e_demo_with_nats/subscriber_received_message
I0419 13:33:13.826676   72332 suite_test.go:121] creating deployment "subscriber"
I0419 13:33:13.849208   72332 suite_test.go:128] waiting for deployment "subscriber" in namespace "e2e-feature-f53" to become ready
I0419 13:33:18.881446   72332 helper_test.go:42] deleting namespace "e2e-feature-f53" for feature "e2e demo with nats"
--- PASS: TestSimpleE2E (26.33s)
    --- PASS: TestSimpleE2E/e2e_demo_with_nats (26.31s)
        --- PASS: TestSimpleE2E/e2e_demo_with_nats/nats_server_running (15.02s)
        --- PASS: TestSimpleE2E/e2e_demo_with_nats/publisher_running (5.03s)
        --- PASS: TestSimpleE2E/e2e_demo_with_nats/subscriber_received_message (5.05s)
PASS
ok      k8s-meetup-04-05-2023/e2e       38.501s

AWS EventBridge

This E2E test asserts that the AWS EventBridge controller Helm chart is running in Kubernetes (deployed via helm as part of the test suite), an EventBus custom resource is created, synchronized (status), and created in the AWS service control plane (backend).

# create kind cluster unless it already exists
kind create cluster --name e2e-meetup
export KIND_CLUSTER_NAME=e2e-meetup 
export KO_DOCKER_REPO=kind.local

# define AWS environment variables with IAM permissions to manage event buses and authenticate with ECR
AWS_DEFAULT_REGION=eu-central-1
AWS_ACCESS_KEY_ID=<ID>
AWS_SECRET_ACCESS_KEY=<KEY>
AWS_SESSION_TOKEN=<TOKEN>

# run eventbridge tests
go test -race -count=1 -v ./e2e -args -v 4 -labels=feature=e2e-eventbridge