Docker temporary files are never removed

[sgrimm-sg]

On all my Docker hosts, I'm accumulating a perpetually-growing set of files in /var/log/scalyr-agent-2, two for each Docker container that has ever been launched on the host (one for stdout, one for stderr).

My agent config for Docker monitoring (we use the docker_api log mode rather than syslog because we want to be able to run docker logs locally on the hosts in addition to viewing logs on Scalyr):

// Send Docker logs to Scalyr.
{
  monitors: [
    {
      // Log each Docker container's output in a separate Scalyr logfile
      module: "scalyr_agent.builtin_monitors.docker_monitor",
      api_socket: "/var/run/docker.sock",
      log_mode: "docker_api",
      // Container metrics are pretty noisy and we already have system metrics
      report_container_metrics: false
    }
  ]
}

Example of the problem (this is from one of our CI servers where containers get launched constantly):

ubuntu@buildkite-1:~$ ls -lt /var/log/scalyr-agent-2 | tail -5
-rw-rw-rw- 1 root root        0 Sep 21 18:59 docker-buildkited6412f29a7474ee9bc89c89d4d0bd1e1_app_build_18-stderr.log
-rw-rw-rw- 1 root root        0 Sep 21 18:59 docker-buildkited6412f29a7474ee9bc89c89d4d0bd1e1_app_build_18-stdout.log
-rw-rw-rw- 1 root root        0 Sep 21 18:42 docker-buildkitefe9713dc33b741a0885e9f486c71960c_app_build_17-stderr.log
-rw-rw-rw- 1 root root        0 Sep 21 18:42 docker-buildkitefe9713dc33b741a0885e9f486c71960c_app_build_17-stdout.log
-rw-rw-rw- 1 root root        0 Sep 21 18:21 docker_monitor.log

Expected behavior would be for the agent to delete the files as soon as it has successfully sent the logs to the Scalyr servers.

[imron]

Thanks for reporting this issue. I'm currently looking in to it, but in the meantime, there is a workaround where you can use the config option docker_raw_logs: true in the docker_monitor config e.g.

// Send Docker logs to Scalyr.
{
  monitors: [
    {
      // Log each Docker container's output in a separate Scalyr logfile
      module: "scalyr_agent.builtin_monitors.docker_monitor",
      api_socket: "/var/run/docker.sock",
      log_mode: "docker_api",
      docker_raw_logs: true,
      // Container metrics are pretty noisy and we already have system metrics
      report_container_metrics: false
    }
  ]
}

You'll also need to map the hosts docker log directory to the same path on the container. This is typically /var/lib/docker/containers which should be mapped in with something like:

docker run ... -v /var/lib/docker/containers:/var/lib/docker/containers

Once you've done this, then the agent won't create its own files on disk, instead it will read directly from the container log files.