docker-compose stop exits with 137 code and take long time

[petr-ujezdsky]

When shutting down activemq docker container using docker-compose stop the docker-compose waits for a few seconds and then finally kills the container.

The activemq should listen to shutdown signal and gracefully shutdown itself.
I do not know if this is problem with this docker image or the ActiveMQ does not support it at all.
Eg. the mongo container supports this exiting much more quickly and with exit code 0.

Simple docker-compose:

$ cat docker-compose.yml
version: '3'
services:
  activemq:
    image: rmohr/activemq
    ports:
      - "61616:61616" # TCP
      - "8161:8161"   # UI

Run it and stop it

$ docker-compose up -d activemq
Starting docker_activemq_1 ... done

$ docker-compose stop
Stopping docker_activemq_1 ... done

$ docker-compose ps
      Name                     Command                State     Ports
---------------------------------------------------------------------
docker_activemq_1   /bin/sh -c bin/activemq co ...   Exit 137

[rmohr]

Hi,

I think the issue is that activemq is not pid 1. That should be fixably by replacing

CMD ["/bin/sh", "-c", "bin/activemq console"]

with

CMD [ "bin/activemq", "console"]

You can immediately try it out if you start the container like this:

docker run rmohr/activemq bin/activemq console

If you then stop the container with docker stop the exit code will hopefully look fine.

[rmohr]

Ok, does not help. A wrapping bash script is needed, which traps signals and then explicitly runs bin/activemq stop.

[Mati20041]

Instead of using /bin/activemq console I've started using command that runs activemq directly -> java $JAVA_OPTS -jar /activemq-home/activemq/bin/run.jar start . Now I don't have any problem with this stopping and starting.
Example from dockerfile

ENV JAVA_OPTS=" \
    -Xms256M \
    -Xmx256M \
    -Dorg.apache.activemq.UseDedicatedTaskRunner=true \
    -Djava.util.logging.config.file=logging.properties \
    -Dcom.sun.management.jmxremote \
    -Dactivemq.classpath=/activemq-home/activemq/conf; \
    -Dactivemq.home=/activemq-home/activemq \
    -Dactivemq.base=/activemq-home/activemq \
"
COPY startamq.sh startamq.sh
CMD ["./startamq.sh"]

startamq.sh:
exec java $JAVA_OPTS -jar /activemq-home/activemq/bin/run.jar start

[petr-ujezdsky]

Nice solution, but that bypasses a whole lot of logic in the activemq script.
I am afraid that the activemq script itself has to be modified to support the exec option. Which means contacting activemq developers :-/

[nagper]

Using "tini" solves the problem.
Search for the "--init" flag here:
https://docs.docker.com/engine/reference/run/
More on this:
https://github.com/krallin/tini

[petr-ujezdsky]

Thanks for the tini info, I did not know about it :)
However this also does not work

$ cat docker-compose.yml
version: '3'
services:
  activemq:
    image: rmohr/activemq
    init: true
    ports:
      - "61616:61616" # TCP
      - "8161:8161"   # UI

It stops the container very quickly, however the KILL signal is not properly forwarded to the ActiveMQ, so it stops abruptly with exit code 143.

I have found the problem - the java process is invoked as exec sh -c 'java some.jar args'. This makes the shell running under PID 1, but the java itself will run under new PID. (we are using custom java installation so its path here is different)

activemq@a9d909468041:/opt/apache-activemq-5.15.10$ ps axu
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
activemq     1  0.0  0.0   4296   708 ?        Ss   12:05   0:00 sh -c "/opt/java/bin/java" -Xms64M -Xm
activemq    23 29.3  1.5 5481824 251840 ?      Sl   12:05   0:12 /opt/java/bin/java -Xms64M -Xmx1G -Dja
activemq   108  0.1  0.0  18144  3172 pts/0    Ss   12:06   0:00 /bin/bash
activemq   164  0.0  0.0  36644  2832 pts/0    R+   12:06   0:00 ps axu

The proper invocation should be exec sh -c 'exec java some.jar args'. Then the java itself is using the PID 1 and the KILL signal is properly forwarded to AMQ which then shuts down gracefully.

activemq@a9d909468041:/opt/apache-activemq-5.15.10$ ps axu
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
activemq     1  161  1.6 5282132 275824 ?      Ssl  12:08   0:11 /opt/java/bin/java -Xms64M -Xmx1G -Dja
activemq    61  0.5  0.0  18144  3220 pts/0    Ss   12:08   0:00 /bin/bash
activemq    82  0.0  0.0  36644  2788 pts/0    R+   12:08   0:00 ps axu

I will try to file a bug the upstream ActiveMQ -> https://issues.apache.org/jira/browse/AMQ-8364

[nagper]

Hi Petr,

Nice catch, although i am not sure exit code 143 is as bad as you think.
Have a look a these conversations:
neo4j/docker-neo4j#30
docker-library/tomcat#57

Besides I have a container in which java is the main process (pid 1) and it also stops with exit code 143 (without tini)

I am not pretty sure that i am right, please let me know what you think of it.

Regards,
Peter

[rmohr]

Thanks you for your contribution.

As you probably noticed, I am not using this repo myself anymore for a long time and fail to find the time to maintain it for quite some time now too. Please consider using a fork. This repo is unmaintained and I am going to archive it soon.