/logging-service

EFK Setup

Components

  • Logging operator (https://banzaicloud.com/ )
  • Elastic operator
  • Elasticsearch for database
  • Enterprise search
  • Fluentd and Fluentbit for logs prosessor and logs collector(already included with logging operator but customized yaml files included in this setup )
  • Kibana for visualization
  • scalable setup

Architecture:

image description

Create a Namespace logging

1. kubectl create ns logging

Install helm

* Install: (brew install helm) https://helm.sh/docs/intro/install/ 
* Configure KUBECONFIG evn variable into your env variable: https://stackoverflow.com/a/45276283

Deploy loggin-operator

1. helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
2. helm repo update
3. helm upgrade --install --wait --create-namespace --namespace logging logging-operator banzaicloud-stable/logging-operator

Note:- Deleting a helm chart command (loggin-operator is the release name, Also use -n command)

Deploy Manifest

  1. kubectl apply -k .

Clear Elasticsearch index

curl -X DELETE 'http://localhost:9200/_all'

Elastic operator CRD

https://www.elastic.co/guide/en/cloud-on-k8s/2.5/k8s-deploy-eck.html https://github.com/elastic/cloud-on-k8s/tree/2.5/config/samples https://www.elastic.co/guide/en/cloud-on-k8s/2.5/k8s-deploy-eck.html

Restart fluentbit daemonsets

curl -X DELETE 'http://localhost:9200/_all'

Quick Guide everything

  1. https://banzaicloud.com/docs/one-eye/logging-operator/quickstarts/es-nginx/

Useful Command

  1. kubectl get secret quickstart-es-elastic-user -n logging -o go-template='{{.data.elastic | base64decode}}'
  2. kubectl port-forward service/quickstart-es-http -n logging-test 9200
  3. kubectl port-forward service/quickstart-kb-http -n logging-test 5601

Custom Resource Definition

You can define outputs (destinations where you want to send your log messages, for example, Elasticsearch, or and Amazon S3 bucket), and flows that use filters and selectors to route log messages to the appropriate outputs. You can also define cluster-wide outputs and flows, for example, to use a centralized output that namespaced users cannot modify.

You can configure the Logging operator using the following Custom Resource Definitions.

  • logging - Represents a logging system. Includes Fluentd and Fluent-bit configuration. Specifies the controlNamespace. Fluentd and Fluent-bit will be deployed in the controlNamespace
  • output - Defines an Output for a logging flow. This is a namespaced resource. See also clusteroutput.
  • flow - Defines a logging flow with filters and outputs. You can specify selectors to filter logs by labels. Outputs can be output or clusteroutput. This is a namespaced resource. See also clusterflow.
  • clusteroutput - Defines an output without namespace restriction. Only effective in controlNamespace.
  • clusterflow - Defines a logging flow without namespace restriction.

Enriching the logs

For extra filters and log formatting please use below filters

  1. https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/filters/

Troubleshooting fluentd

  1. https://banzaicloud.com/docs/one-eye/logging-operator/operation/troubleshooting/fluentd/

Troubleshooting fluentbit

  1. https://banzaicloud.com/docs/one-eye/logging-operator/operation/troubleshooting/fluentbit/

Interacting with elasticsearch

https://logs.sariska.io/app/dev_tools#/console

Important resources