/fluent-bit-kubernetes-logging

Fluent Bit Kubernetes Daemonset

Kubernetes Logging with Fluent Bit

Fluent Bit is a lightweight and extensible Log Processor that comes with full support for Kubernetes:

  • Read Kubernetes/Docker log files from the file system or through systemd Journal
  • Enrich logs with Kubernetes metadata
  • Deliver logs to third party storage services like Elasticsearch, InfluxDB, HTTP, etc.

This repository contains a set of Yaml files to deploy Fluent Bit which consider namespace, RBAC, Service Account, etc.

Getting started

Fluent Bit must be deployed as a DaemonSet so that it will be available on every node of your Kubernetes cluster. To get started run the following commands to create the namespace, service account and role setup:

$ kubectl create namespace logging
$ kubectl apply -k ./base

(./base can be replaced with a URL if #90 gets merged)

If you are deploying fluent-bit on openshift, you additionally need to run:

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-openshift-security-context-constraints.yaml

The base only configures Stdout output. Log records are found using kubectl logs on fluentbit pods. This is a good way to experiment with configuration. Switch to Format msgpack to also see the "tag" (helpful when developing your pipeline).

Logs ingested to Loki

This repository serves as example of Kubernetes yaml for a log processing stack, not as example of how Fluent-bit can process, refine and forward log entries.

Use the loki base to replace Stdout with forwarding to a standalone Loki instance. Loki can do some processing at query time which helps keep this repo light on pipeline config.

The loki-plugin base uses an out-of-tree plugin with DropSingleKey so that the log records forwarded are the actual logged lines from pods, but still queryable on labels like pod and container.

See the lokitest Job for some examples of how to consume logs through Loki.

The loki folders also serve as example of how to use Kustomize to override selected parts of the base's *.conf.

Get in touch with us!

Your contribution to testing is highly appreciated. We aim to make logging cheaper for everybody so your feedback is fundamental. Please get in touch on: