Task missing at service update
dongluochen opened this issue · 7 comments
When I update image of a service, the tasks may go down and come up. But in most cases some tasks disappear in this period. I'll have to refresh visualizer to get them back.
In this example, 4 tasks from service test are running but only 2 are shown.
ubuntu@ip-172-19-241-144:~$ docker service ls
ID NAME REPLICAS IMAGE COMMAND
ikbylntzf1a8 test 5/5 dongluochen/simpleweb:2.0
itaigkgu9o4h viz 1/1 manomarks/visualizer
ubuntu@ip-172-19-241-144:~$ docker service ps test | grep -i running
test.1.aepqd3xweezqtgbcqtwrq26f4 dongluochen/simpleweb:2.0 ip-172-19-241-144 Running Running 35 seconds ago
test.2.maqa2woorz8g7erg0nuok31wl dongluochen/simpleweb:2.0 ip-172-19-241-145 Running Running 12 seconds ago
test.3.t3e0npr9f9mimrqaa5dfrby86 dongluochen/simpleweb:1.0 ip-172-19-241-146 Running Running 4 minutes ago
\_ test.4.shrsqrcnosli2cjo0ek21frop dongluochen/simpleweb:1.0 ip-172-19-241-146 Shutdown Running 3 minutes ago
test.5.z4uclx3cc9c8x98d2fvfacf3y dongluochen/simpleweb:1.0 ip-172-19-241-145 Running Running 4 minutes ago
This project is awesome!
@dongluochen Can you give me some steps to recreate? I'm not showing this problem.
@ManoMarks I have viz setup in AWS with 3 machines cluster. I access viz thru public IP of the manager:8080 with port 8080 allowed to my IP. Here are the steps to repro.
$ uname -a
Linux ip-172-19-241-144 4.2.0-22-generic #27-Ubuntu SMP Thu Dec 17 22:57:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
4ianz3jxl171eqh137b2ic11k ip-172-19-241-146 Ready Active
5x3kjyttddq67kln5pvocaclj ip-172-19-241-145 Ready Active
8gpmadjgkqbpuanctojbfkosp * ip-172-19-241-144 Ready Active Leader
# create viz
$ docker service create --name=viz --publish=8080:8080/tcp --constraint=node.role==manager --mount=type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock manomarks/visualizer
# create a service
$ docker service create --name test --replicas 3 dongluochen/simpleweb:1.0
$ docker service ls
ID NAME REPLICAS IMAGE COMMAND
i3d05j4973ao test 3/3 dongluochen/simpleweb:1.0
qrruwcvm69nq viz 1/1 manomarks/visualizer
# validate container counts are correctly shown in viz interface
# scale the service
$docker service scale test=10
# the container counts are not properly shown in viz interface
$ docker version
Client:
Version: 1.13.0-dev
API version: 1.25
Go version: go1.7.3
Git commit: 89ef0f2
Built: Mon Nov 7 18:06:13 2016
OS/Arch: linux/amd64
Server:
Version: 1.13.0-dev
API version: 1.25
Go version: go1.7.3
Git commit: 89ef0f2
Built: Mon Nov 7 18:06:13 2016
OS/Arch: linux/amd64
Experimental: false
If I do a refresh on the viz interface, it'll update to show the correct containers in each machine.
Same problem with image manomarks/visualizer:beta. It looks like automatic refresh doesn't go thru. Force a refresh on chrome would fix the problem.
Here is info of visualizer.
ubuntu@ip-172-19-241-144:~$ docker service inspect viz --pretty
ID: qrruwcvm69nq9f33vpz1to18l
Name: viz
Service Mode: Replicated
Replicas: 1
UpdateStatus:
State: completed
Started: 2 hours
Completed: 2 hours
Message: update completed
Placement:Contraints: [node.role==manager]
UpdateConfig:
Parallelism: 1
On failure: pause
Max failure ratio: 0
ContainerSpec:
Image: manomarks/visualizer:beta@sha256:3a9fcc9d1f905d384b8b15f47d99aa1eeacbf65426f2c467d2abbb4f0a6c1378
Mounts:
Target = /var/run/docker.sock
Source = /var/run/docker.sock
ReadOnly = false
Type = bind
Resources:
Endpoint Mode: vip
Ports:
PublishedPort 8080
Protocol = tcp
TargetPort = 8080
I am able to replicate the problem using Docker for AWS. What seems to happen is that the node stack doesn't show always show up if it involves changing a node that already has a service running on it. I'll dig deeper into this.
Any update on this? I ran into this on my swarm setup on DigitalOcean. @ManoMarks I realize this is not production, but this is such an awesome tool when someone wants to see how the deployment works :-)
I have taken some time to look at it, but it's pretty complex. The original code wasn't mine so I'm having to track the lifecycle of a task through the swarm, it's a little complicated :-)