fluent-bit with k8s to YC Cloud Logging

Description

Terraform module creates the Cloud Logging log group and deploys fluent-bit with required configuration in the cluster configured in ~/.kube/config. Kubernetes cluster ID is required for filtering purposes in Cloud Logging dashboard. This module requires two key.json files placed in ./helm and ./ folders respectively. The first one will be used by fluent-bit service, and the second one is required to deploy Terraform module (OAuth token can be used instead).

Prerequisites

Cloud Logging service account key.json file with the following folder roles:

logging.writer
monitoring.editor

Kubernetes cluster ID (optional)
Kubernetes authentication configured (~/.kube/config must already exist)
YC configuration variables: folder_id, cloud_id (can be obtained through yc config list)
Terraform provider authentication through service account key.json file

Steps

Save Cloud Logging service account key.json file to ./helm folder
Save Terraform provider service account key.json file to the top folder ./
Fill private.auto.tfvars file with the following data:

cloud_id = "xxx"
folder_id = "xxx"
kube_cluster_id = "xxx"

Run terraform init and terraform apply

Using Helm only

You can use Helm chart directly if you want. Just fill the ./helm/values.yaml file with the Cloud Logging group ID and Kubernetes cluster ID. And place the Cloud Logging service account key.json file in the ./helm folder. After that, run helm install command.

Notes

config.yaml file in the root of this folder is temporary and exists only to display various options available to configure in fluent-bit plugin. Configuration file that is actually used during deployment through Terraform/Helm is located at ./helm/templates/config-map.yaml