/docker-logging

Primary LanguagePython

docker-logging

This repository contains examples of various concepts when logging from containers.

Build the docker container

docker build -t logging-101:v1 .

Example 1

In the example, the application is writing to STDOUT and STDERR. We can capture the messages logged inside the container to syslog running on the host machine using the Syslog logging driver.

Setup:

  • docker running on single linux host
  • rsyslog is also running on the host

docker run --log-driver syslog --log-opt syslog-address=unixgram:///dev/log logging-101:v1 example-1/example_1.py

  • specify the logging driver as Syslog
  • specify the protocol as UDP via unixgram
  • the socket is /dev/log

Example 2

In this example, the application is logging to the Unix syslog using the Python SysLogHandler. It is sending to the domain socket /dev/log. By default, the path does not exists inside the container. We will mount the path on the host machine to the container to allow the host machine to capture the messages.

Setup:

  • docker running on single linux host
  • rsyslog running on the host

docker run --mount type=bind,source=/dev/log,target=/dev/log logging-101:v1 example-2/example_2.py

  • this expose the /dev/log address to the container
  • I used the bind mount option instead of a volume because the address is not maintained by docker

Example 3

In this example, the application is logging to STDOUT and STDERR. Instead of relying on the host machine to provide the rsyslog service, we can created a dedicated logging container. The logging container will expose the listening port and log data to a location that is mounted on the host.

Setup:

  • docker running on the single linux host
  • rsyslog container running on the host

docker run -d --mount type=bind,source=/tmp/clogs,target=/var/log -p 127.0.0.1:5514:514/udp --name rsyslog voxxit/rsyslog

  • /tmp/clogs is a directory on the local host. It is mounted on the container as /var/log. Rsyslog writes data to /var/log/messages
  • the container listens on port 5514 of the local host and routes to port 514 on internal network

docker run --log-driver syslog --log-opt syslog-address=udp://127.0.0.1:5514 --log-opt syslog-facility=daemon --log-opt tag=app101 --name=logging-ex3 logging-101:v1 example-1/example_1.py

  • specify the logging driver as Syslog
  • specify the networking address of the Rsyslog process

You can run multiple instances of the application and all of them will log to the same rsyslog container. Tail the log to see all the messages.

Example 4

In this example, we will create a user-defined network. This will provide us with a DNS server, so we can resolve IP addresses by container name. We will have a rsyslog service and bind mount a directory to store the logs. Our application will log to syslog. And because both containers are attached to the same network, they can talk to each other. handler = SysLogHandler(address=('rsyslog', 514))

Setup:

  • docker running on the single linux host
  • rsyslog container running on the host
  • application running on host
  • a user-defined network

docker network create ex4_network

mkdir /tmp/clogs

docker run -d --network=ex4_network --mount type=bind,source=/tmp/clogs,target=/var/log --name rsyslog voxxit/rsyslog

  • we mount a data storage
  • and specify that the container runs in the ex4_network

docker run --network=ex4_network --name logging-ex4 logging-101:v1 example-4/example_4.py

  • run our application and attached it to the ex4_network

The logs generated by our application are sent to the Rsyslog server. We can see them by tailing /tmp/clogs/messages

Example 5

In this example, we will log to Fluentd. We will bind mount a directory to the container running Fluentd to permenently store the logs. See https://hub.docker.com/r/fluent/fluentd/

Setup:

  • docker running on the host
  • Fluentd docker container
  • application logging to STDOUT and STDERR.

mkdir /data

docker run -d -p 24224:24224 -p 24224:24224/udp --mount type=bind,source=/data,target=/fluentd/log fluent/fluentd

  • publish port 24224, both tcp and udp

docker run --log-driver fluentd --name=logging-ex5 logging-101:v1 example-1/example_1.py

  • we're using a lot of default values here. Protcol is TCP. Fluentd logging driver address is localhost:24224.

We can go to /data on the host and find the log files.

Example 6

In this example, we will use Kubernetes. This example is similar to example 1 where there is a single logging agent running on the node.

Setup:

  • kubernetes (minikube)
  • 2 app pods that log to STDOUT
  • a fluentd pod

The two apps are writing to stdout. Kubernetes capture the data and append them to log files. The log files are stored in /var/log/containers. We use fluentd to capture all the logs and output them (stdout in our example) We do this by mounting the volume where the log files are output to, then use the fluentd tail plugin.

example-6/alpine-sysout contains the default fluentd.conf file, and the Dockerfile. This is for our custom fluentd docker image. Note: In this example, the fluentd had to be running as root user in order to have permission to access the log directories.

Create configmap which contains my custom configuration for fluentd

kubectl create configmap my-fluentd-config --from-file=my_fluentd.conf

Start the apps.

kubectl create -f counter-odd-pod.yaml kubectl create -f counter-even-pod.yaml

Start the logging agent. In this case I am starting it as pod for simplicity. A better approach is to start it as a daemonset.

kubectl create -f fluentd-agent.yaml

Check that the logging agent is collecting log data from both running apps.

kubectl logs fluentd-agent-pod

Example 7

In this example, we will use Kubernetes and try out the concept of a sidecar container with logging agent. This pattern is needed in the situation where the app container does not log to STDOUT or STDERR. In this case, the logging agent in the sidecar can be configured to read logs based on how the app is logging. In this example 7, the app is logging to syslog, so we create a fluentd agent that listens SYSLOG protocol.

Setup:

  • kubernetes (minikube)
  • a POD with
    • an app logging to syslog
    • fluentd listening to syslog protcol and output to stdout (sidecar)

kubectl create configmap my-fluentd-syslog-config --from-file=example-7/my_fluentd.conf

kubectl create -f example-7/app-syslog-sidecar.yaml

kubectl logs app-syslog fluentd-syslog-agent

You should be able to see the logs from the app inside the fluentd-syslog-agent container.