Blog On Logging in a docker container.

[rhatdan]

Playing with Logging in an docker container.

syslog and journal log messages silently dropped by default

One big problem with standard docker containers is that any service that writes messages to syslog or directly to the journal get dropped by default. docker does not record any logs unless the messages are written to STDIN/STDERR. If you want proper logging setup, I would suggest that you investigate running
systemd inside of a container.

Alternatively you could leak the hosts sockets into your container.

STDIN/STDERR messages in journal

I recently received a bugzilla complaining about logging inside of a docker container.

First the user complained about all of STDOUT/STDERR showing up in the journal. This can actually be configured in the docker daemon using the --log-driver

man dockerd
...

       --log-driver="json-file|syslog|journald|gelf|flu‐
       entd|awslogs|splunk|etwlogs|gcplogs|none"
         Default driver for container logs. Default is json-file.
         Warning: docker logs command works only for json-file logging driver.

Red Hat based Operating Systems use --log-driver=journald by default, because we want to log files to be permanent on the system. The upstream docker default is json-file, but these logs are removed when the user does a docker rm of the container. Also we have had problems with containers logs filling up the system. If you don't want the STDOUT/STDERR going into the journal, you can edit /etc/sysconfig/docker and change the log-driver.

Getting messages out of the container to the host logging system.

The bug reporter went on to show that volume mounting the /dev/log from the host into the container did not successfully get log messages from the container to the host journal. They did now get messages sent to syslog.

# docker run -ti -v /dev/log:/dev/log fedora sh
container# dnf -y install systemd-python
...
container# python <<< "from systemd import journal; journal.send('journald Hello')"
container# logger "logger Hello"
container# exit

# journalctl -b | grep Hello
Oct 19 09:53:28 dhcp-10-19-62-196.boston.devel.redhat.com root[16787]: logger Hello

Notice the message to the journal does not show up but the logger message does. The difference is syslog messages sent from the logger command write to /dev/log. journald on the host is listening for syslog messages in the hosts /dev/log. When it sees the message sent to the bind mounted /dev/log it logs the message in the journal.

The message sent directly to the journal is writes to a socket in /run/systemd/journald/socket, which does not exists inside of the container. The journald message is silently dropped.

This works for me.

# docker run -ti -v /dev/log:/dev/log -v /var/run/systemd/journal/socket:/var/run/systemd/journal/socket fedora sh
container# dnf -y install systemd-python
...
container# python <<< "from systemd import journal; journal.send('journald Hello')"
container# logger "logger Hello"
container# exit

# journalctl -b | grep Hello
Oct 19 09:57:51 dhcp-10-19-62-196.boston.devel.redhat.com python[17523]: journald Hello
Oct 19 09:57:53 dhcp-10-19-62-196.boston.devel.redhat.com root[16787]: logger Hello

The journal.Send call above connects to /run/systemd/journal/socket and since we leaked it into the container, the messages gets to the hosts journal.

Note: SELinux was in enforcing mode for all of these tests. We allow container processes to communicate with the journal/syslog sockets on the host by default.

[jberkus]

I've started formattting this, but it really needs an intro paragraph. Do you want to write one, or should I?

Or, I can just remove the first subtitle ... it seems to work pretty well that way. But see line edits.

[jberkus]

Alternatively, you could leak the hosts sockets into your container.

What? Can you explain this? Is this what the rest of the blog post is about? If so, needs some transitional text ...

Also, we have had problems with containers' logs filling up the system.

So ... is that a recommendation for journald, or against it? Not clear.

Notice the message to the journal does not show up but the logger message does. The difference is syslog messages sent from the logger command write to /dev/log. journald on the host is listening for syslog messages in the hosts /dev/log. When it sees the message sent to the bind mounted /dev/log it logs the message in the journal.

I feel like this explanation could be clearer, somehow; I had to read it three times. This version might be a bit better, do you agree?

Notice the "journald Hello" message to the journal does not show up, but the logger message does. The difference is syslog messages sent from the logger command write to /dev/log, and journald on the host is listening for syslog messages there on the host. When it sees the message sent to the bind mounted /dev/log, it logs the message in the journal.

Better?

[jberkus]

How about changing the title of the blog to this?

Improving docker logging by leaking host sockets

Also, how about this concluding para?

So, if you're looking for an alternative to channeling all logging through container STDIN/STDERR, consider bind-mounting the journal socket.

[rhatdan]

Lets move the blog to a google doc so we can collaborate.

[jberkus]

Posted