Kubernetes Observability

Spin up a Kubernetes cluster with Prometheus and Grafana installed using Docker, and Terraform. The repo provides IaC to create a Kubernetes cluster using Kind, but the Prometheus and Grafana setup can be used with any Kubernetes cluster.

Setup MinIO

Run the following command to start MinIO:

make minio-up

If you want to customise the MinIO setup, create a .env file in the minio directory using the default.env file as a template.

The default.env file provides default values good enough for local development.

Setup Terraform State Backend on MinIO

Connect to the MinIO web interface and create a bucket named terraform-states with
- Locking enabled
- Versioning enabled.
Create another bucket named cluster-kubeconfig to store the kubeconfig file for the Kind cluster (which will be useless at the moment, because MinIO Terraform provider does not provide a way to read from bucket yet).
Create a user with the necessary permissions to read/write to the buckets (choose the IAM setup that suits your needs best). (example here)
Create a pair of keys for the user, take note of them and download the credentials.json file. Then copy the credentials file to the root of this repository.

Create a .config.s3.tfbackend file in both terraform/kind-cluster and terraform/observability directories with the following content:

bucket = "terraform-states"
endpoints = {
  s3 = "http://localhost:9000"
}
# e.g "kind-cluster/terraform.tfstate" or "observability/terraform  tfstate"
# but it could be any value (as long as it is unique for each state file)
key = "<DIRECTORY_NAME>/terraform.tfstate"

access_key = "<YOUR_ACCESS_KEY>"
secret_key = "<YOUR_SECRET_KEY>"

region                      = "main"
skip_credentials_validation = true
skip_metadata_api_check     = true
skip_region_validation      = true
skip_requesting_account_id  = true
use_path_style              = true

Create Cluster

The cluster IaC is in the terraform/kind-cluster directory. A dev.tfvars file is provided to set up a cluster using a pre-packaged configuration.

Run the following command to create the cluster:

# file path is relative to the terraform/kind-cluster directory
make init STACK=kind ARGS="-backend-config=.config.s3.tfbackend"
make plan STACK=kind ARGS="-var-file=dev.tfvars"
make apply STACK=kind ARGS="-var-file=dev.tfvars"

Setup Prometheus And Grafana

Once that the cluster is created, run the following command to set up Prometheus and Grafana:

make init STACK=observability ARGS="-backend-config=.config.s3.tfbackend"
make plan STACK=observability ARGS="-var-file=dev.tfvars"
make apply STACK=observability ARGS="-var-file=dev.tfvars"

Example IAM Setup

Go to the MinIO console (e.g. http://localhost:9001 if you're using the default configuration).
Navigate to Identity > Groups and create a group named terraform-admins by

Clicking on the Create Group button
Setting the group name to terraform-admins
Clicking on the Save button

Set the policy for the group by

Clicking on the newly created group terraform-admins
Clicking on the Policies tab and then clicking on the Set Policies button
From the policies list, select consoleAdmin, diagnostics and readwrite policies, then click Save

Navigate to Identity > Users and create a new user by

Clicking on the Create User button
Setting the User Name to terraform, and setting the Password to whatever you want
Assign the terraform-admins group in the Assing Group section
Clicking on the Save button

From the Users page create a new pair of credentials by

Clicking on the newly created user terraform
Clicking on the Service Accounts tab of the user
Clicking on the Create Access Key button
Clicking on the Create button
Taking note of the Access Key and Secret Key and downloading the credentials.json file

Eventually, further restrict the service account permissions by

Clicking on the Service Account Access Key that you want to restrict, and, for example, restrict the buckets that can be accessed by applying the following change

// in the statements section of the policy
"Statement": [
  {
   "Effect": "Allow",
   "Action": [
    "s3:*"
   ],
   "Resource": [
    // change the default "arn:aws:s3:::*" to the following
    "arn:aws:s3:::cluster-kubeconfig*",
    "arn:aws:s3:::terraform-states/*"
   ]
  },
]