Do we need a init system?
gtcno opened this issue · 9 comments
Maybe, probably. Otherwise we likely need some different way of starting apps.
In this current baseimage draft and our current app we use CMD or ENTRYPOINT to start our apps with sh.
According to this: https://pracucci.com/graceful-shutdown-of-kubernetes-pods.html the alpine linux shell, /bin/ash does not propagate signals to its children. So the java app does not see the SIGTERM and thus frameworks like spring boot and camel do not gracefully shut down.
We need to verify this behaviour and find a solution, where dumb-init could be one.
We use the exec form to start Java apps, which means they get pid 1. So there shouldn't be any issues regarding signals, but I'll test anyway.
We are having problems taking heap dumps.
Might be related to this:
docker-library/openjdk#76.
Verified that heap dump is possible when running the java process with PID > 1.
Fails with PID 1. Docker > 1.13 has the --init flag which can be set on the the docker hosts
which will run the container with https://github.com/krallin/tini.
However we still need to include jmap, jstack etc to this baseimage, I think.
Yeah we need this then. Maybe we should set HeapDumpOnOutOfMemoryError and HeapDumpPath as default flags as well. That way NAIS can automate retrieval of these?
Regarding the jmap, jstack etc utils, they are part of the JDK and not the JRE. So we either need to base the baseimages of the JDK (increasing layer size) or include them through some other mechanism.
@gtcno are you planning to add the --init flag when starting docker containers in NAIS? I made a branch adding dumb-init to java-8, however I guess this is not needed if you guys add the tinit functionality.
Then we only need to add the openjdk:8-jdk-alpine to java-8. The difference in layer is 22MB which is not huge. However, being able to make heapdumps is useful. @davidsteinsland thoughts?
I must admit, I have completely forgotten about this issue.
If we're not using the --init feature, we could have a bash entrypoint which uses trap to receive SIGTERM, which it could forward to java (if it's required to have java running on pid > 1).
Changing the base image to java-8 sounds good to me, a few extra megabytes doesn't kill anyone. 👍
First cut adding the flag to the daemon failed. Daemonsets pods failed. I did not try out why. I'll look into it. However, can't see why we should not add it to the baseimage. It will work fine with the flag det globally as well
It wasn't possible to add the --init flag globally in NAIS, so we add the dumb-init to our baseimages for now.