core dumps not being stored

[olafbuitelaar]

with the introduction of the security update to run haproxy as the haproxy user instead of root, haproxy cannot generate core dumps anymore when it crashes. The default core location is the root, which it cannot write to, it seems it tries to pipe the dump through abrt (|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %e %i), but the dump is not stored anywhere.
I've created a workaround, by pre-creating the file and change the ownership;

RUN touch /core.8 && chown haproxy.haproxy /core.8

see; haproxy/haproxy#1966 (comment)

But maybe changing the /proc/sys/kernel/core_pattern to a location where haproxy can write to at run time (which would require root and than later dropping this privilege), or alternative abrt configuration to write the dump to would a better default setup.

[tianon]

Unfortunately, /proc/sys/kernel/core_pattern is a setting from the host (or the container orchestrator), not something we can change or control from the container image. 😞

[olafbuitelaar]

maybe it's an option to let haproxy always create the core dumps in it's working directory?

[tianon]

I think the primary problem with that idea is that it's not actually haproxy itself making the core dumps, right? (Aren't those generated by the kernel, which is why where to put them is configured at the kernel level?)

[TimWolla]

Aren't those generated by the kernel

Indeed. But with the default settings the core is generated in the working directory, so setting an (empty) WORKDIR that is writable to the haproxy user might be sufficient here.

[TimWolla]

I'm seeing that the /var/lib/haproxy directory is already set as the home for the haproxy and also already exists with the correct permissions, so just setting WORKDIR /var/lib/haproxy should hopefully work.

[tianon]

If someone can test it out and verify, I'd be happy to see such a change. 😄

[TimWolla]

@tianon Can confirm it works:

$ cat /proc/sys/kernel/core_pattern
core.%p

$ cat haproxy.cfg
global
	set-dumpable

listen foo
	bind *:8080
	mode http

	server example example.com:80
$ cat Dockerfile
FROM haproxy

WORKDIR /var/lib/haproxy

COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg

$ docker run --restart=always --name haproxy-test -it haproxy:test
[NOTICE]   (7) : haproxy version is 2.7.0-437fd28
[NOTICE]   (7) : path to executable is /usr/local/sbin/haproxy
[WARNING]  (7) : config : missing timeouts for proxy 'foo'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[NOTICE]   (7) : New worker (9) forked
[NOTICE]   (7) : Loading success.

$ sudo kill -11 219604

[NOTICE]   (7) : haproxy version is 2.7.0-437fd28
[NOTICE]   (7) : path to executable is /usr/local/sbin/haproxy
[ALERT]    (7) : Current worker (9) exited with code 139 (Segmentation fault)
[ALERT]    (7) : exit-on-failure: killing every processes with SIGTERM
[WARNING]  (7) : All workers exited. Exiting... (139)

$ docker exec -it haproxy-test ls -alh
total 3.0M
drwxr-xr-x 2 haproxy haproxy    3 Dec 16 22:33 .
drwxr-xr-x 8 root    root       8 Dec  6 04:27 ..
-rw------- 1 haproxy haproxy 208M Dec 16 22:33 core.9