Teahouse

Demo setup for Spring Boot apps with Prometheus, Grafana, Loki, Tempo, Eureka, and Spring Boot Admin to demonstrate Observability use-cases.

Start dependencies

docker compose up

Stop dependencies

docker compose down

Stop dependencies and purge data

docker compose down --volumes

Remove logs

rm -rf logs

Start the apps (using in-memory H2 DB)

./gradlew bootRun

Start the apps using MySQL

This is needed if you want to inject latency on the network (see ToxiProxy).

./gradlew bootRun -Pprofiles=mysql

Start load tests

See SteepTeaSimulation.java for duration, request rate, and traffic patterns.

./gradlew :load-gen:gatlingRun

Useful URLs

Tea UI: http://localhost:8090/steep
Tea Service: http://localhost:8090
Tealeaf Service: http://localhost:8091
Water Service: http://localhost:8092
Spring Boot Admin: http://localhost:8080
Eureka: http://localhost:8761
Prometheus: http://localhost:9090
Loki, Grafana, Tempo: http://localhost:3000
ToxiProxy UI (failure injection): http://localhost:8484
MailDev (emails for alerts): http://localhost:3001
Adminer (DB Admin UI): http://localhost:8888 (credentials: root:password)

Errors simulation

When start the apps for the first time, english breakfast is missing from the DB but you can make requests through the UI using english breakfast and the load generator also sends requests containing it. Those calls will end-up with HTTP 500; approximately 10% of the requests should fail: ~0.5 rq/sec error- and ~4.5 rq/sec success rate (~5 rq/sec total throughput, see SteepTeaSimulation.java).

You should see these errors on the throughput panel of the Tea API dashboard and Grafana also alerts on them (see the emails in MailDev).

If you want to fix these errors, you need to create a record in the DB for english breakfast. The easiest way is sending an HTTP POST request to /tealeaves to create the resource (you can also log into the DB and insert the record for example using Adminer). The Makefile contains a goal for this to make it simple for you, you can run this to fix errors (httpie and jq needed):

make errors-fixed

If you want the errors back again, you need to remove the record from the DB, the Makefile contains a goal for this too, so you can run this to inject errors:

make errors

Latency simulation

If you start the apps with the mysql profile, the apps are not connected to the DB directly but through ToxiProxy so that you can inject failures (i.e.: latency) on the network. You can do this in multiple ways (e.g.: using the ToxiProxy UI or the ToxiProxy CLI). The Makefile contains a goal for this to make it simple for you, you can run this to inject latency:

make chaos

And this to eliminate the extra latency:

make order

kmandalas/teahouse