Docker-CRDB

This repo encapsulates a multi-node CockroachDB environment with integrated logging and monitoring. Containerized components such as Grafana, Prometheus, Fluentd, and LOKI are configured and used to capture the logging and metrics originating from the CRDB cluster.

Also included is Alerting which is connected via WebHooks from Grafana to a running "Alerts" container that fire requests to Twilio (SMS message notifications), and SendGrid (Email notifications). The altering container is a NodeJS web app, shown in the second architecture below.

This is a complete platform that's running in a Docker environment, leveraging a custom named bridge network designed to avoid any IP conflicts.

Overall architectures

This project is divided into distict containers to showcase the separation of duty and isolation of components. Generally speaking each container is driving a single image here for simplicity and consistency, but it makes sense to group certain services together (eg: CRDB + Fluentd) that share logical services and have tight dependencies.

Core components

Alerting components

Alerting was separated from the core architecture because it's an optional capability and requires service keys from 3rd party cloud services. In this example, Twilio is used to send email and SMS messages when Grafana triggers are defined.

Pre-Reqs: installation, binaries, tools, certificate generation

Docker, Docker Desktop, Docker-Compose: Ensure that you install Docker, Docker Desktop, and Docker-Compose. Verify that these services are operational.
NodeJS: https://nodejs.org/en/ is the homepage for runtimes, binaries. Please install this run-time platform if you wish to use Alerts.
Alert web services: Alerting relies on Twilio cloud services, and will require API keys to operate. SMS is a paid service, while the email service offers 100 sends per month for free.
CRDB Installed: Please visit https://github.com/cockroachdb/cockroach and download the latest CRDB installation (22.1.0 at the time of this commit)

Certificates for secure mode:

Cheat sheet on certificate creation for this project:

// in the crdb01 folder:
mkdir certs
mkdir ca
cockroach cert create-ca --certs-dir=certs --ca-key=ca/ca.key
cockroach cert create-node crdb-node01 crdb-node02 crdb-node03 --ca-key=ca/ca.key --certs-dir=certs
cockroach cert create-client root --certs-dir=certs --ca-key=ca/ca.key
cockroach --certs-dir=certs cert list

The ca and certs folders are populated with certs & keys. Just copy these 2 folders and paste them into the crdb02 and crdb03 folders to complete the cluster security settings.
The certs list command in the last step is informative, and should return the list of certificates similar to this:

markzlamal@crl_my_laptop crdb01 % cockroach --certs-dir=certs cert list 
Certificate directory: certs
Usage  | Certificate File |    Key File     |  Expires   |                     Notes                      | Error
---------+------------------+-----------------+------------+------------------------------------------------+--------
CA     | ca.crt           |                 | 2032/07/07 | num certs: 1                                   |
Node   | node.crt         | node.key        | 2027/07/04 | addresses: crdb-node01,crdb-node02,crdb-node03 |
Client | client.root.crt  | client.root.key | 2027/07/04 | user: root                                     |
(3 rows)

Start the cluster + logging in Docker

The start sequence of your containers is important due to the networking expectations between the components. This project expects containers to be launched in the following order. Different sequences will work, but you will needlessly generate logging related to temporary network connection problems as the contaners establish connections.

Alerts (optional)
LOKI
Prometheus
Grafana
fluentd01 (relies on LOKI)
fluentd02 (relies on LOKI)
fluentd03 (relies on LOKI)
crdb01 (relies on fluentd01)
crdb02 (relies on fluentd2)
crdb03 (relies on fluentd03)

Each component folder contains a docker-compose.yml file that is the descriptor for the container & image.

git clone https://github.com/cockroachlabs/Docker-CRDB.git
Create a user-defined bridge network
```
docker network create w2m-crdb-net
```

Additional options can be specified to define a unique subnet range if desired.

In each folder (eg: Loki, Prometheus, fluentd01, etc), running this command will create and start the associated container.
```
docker-compose up -d
```

Note that the alerts container is built, while the others are pulled from public repositories.