tiangolo/full-stack

How to troubleshoot a service when it fails to run [workflow for docker swarm deploy]

AYEG opened this issue · 3 comments

AYEG commented

Hi!

I'm trying to deploy a cluster using these instructions .
https://github.com/tiangolo/full-stack/blob/master/docker-swarm-cluster-deploy.md

It seem like the DB is not coming online and causing the backend to also fail after many attempts from tenacity.

$ docker service logs db tells me "no such task or service: db", but I can see it is listed with
$ docker service ls, with replicas 0/1

I don't have previous experience with swarm, is there a way to access failed service logs?
I tried following advice like print journalctl, but that doesnt seem very useful in this case. Should I include something extra to output logs?

AYEG commented

One more thing I tried is edit docker-stack.yml to include

logging:
driver: journald

Then run this command:
$ journalctl -u docker CONTAINER_NAME=db

As described here:
https://stackoverflow.com/questions/43663013/access-logs-of-a-killed-docker-container/43663253

It tells me:
-- Logs begin at Mon 2018-11-12 21:37:13 CET, end at Thu 2018-12-27 14:15:03 CET. --
-- No entries --

AYEG commented

Found a way to print errors here is a possible solution for anyone else wondering how to do this:

docker service ls // list the services
docker service ps <id> // shows container id and a truncated error message in second to last column
docker inspect <container id from first column> | grep Err // print out the error message belonging to the service container

"Err": "no suitable node (scheduling constraints not satisfied on 1 node)"

Running these commands no longer requires enabling experimental docker features, it works by default

Sorry for the delay.

When you create a Docker Swarm mode stack, all the "services" get a name prefixed by the stack name.

So, if you create a stack my-super-app-com, your backend service will be named my-super-app-com_backend.

Then you can see the logs with:

docker service logs my-super-app-com_backend

To see only the last 100 logs:

docker service logs my-super-app-com_backend --tail=100

To see only the last 100 logs and keep showing any new logs live, "following" the logs:

docker service logs my-super-app-com_backend --tail=100 -f

(This last one is what I use the most).