How does reconciliation works?

[lenfree]

I recently experienced an issue with the state of my deploy/task. I use Docker container by the way. To reproduce this scenario,

Singularity version: 0.18.1
Mesos version: 1.0.0

1. Pick an existing deploy ( service type ) host ip
2. On this host ip, stop Docker daemon
3. Wait after 5 seconds start Docker daemon
4. Singularity spins up a new task on another mesos-slave which is the behaviour we expect. However, from above host ip, docker ps shows that earlier container running. However, Singularity thought it has successfully removed this task but this is not the case.

There are logs from mesos-slave:

Handling status update TASK_FAILED (UUID: 12202362-f37a-4e50-aecc-f355671853db)
Checkpointing UPDATE for status update TASK_FAILED (UUID: 12202362-f37a-4e50-aecc-f355671853db) 
Received status update TASK_FAILED (UUID: 12202362-f37a-4e50-aecc-f355671853db)
Updating the state of <task_name> of framework Singularity (latest state: TASK_FAILED, status update state: TASK_FAILED)
Forwarding status update TASK_FAILED (UUID: 12202362-f37a-4e50-aecc-f355671853db) 
Status update TASK_FAILED (UUID: 12202362-f37a-4e50-aecc-f355671853db) 
Forwarding the update TASK_FAILED (UUID: 12202362-f37a-4e50-aecc-f355671853db)
Sending acknowledgement for status update TASK_FAILED (UUID: 12202362-f37a-4e50-aecc-f355671853db)
SingularityTaskStatusHolder{taskStatus=Optional.of(SingularityMesosTaskStatusObject{agentId=Optional.of(SingularityMesosIdObject{value='4fd57a45-39a9-48af-aca7-0faf19ea7d32-S192'}), slaveId=Optional.of(SingularityMesosIdObject{value='4fd57a45-39a9-48af-aca7-0faf19ea7d32-S192'}), healthy=Optional.absent(), message=Optional.of(Container exited with status 125), reason=Optional.absent(), state=Optional.of(TASK_FAILED), taskId=Optional.of(SingularityMesosIdObject{value='<TASK>'}), timestamp=Optional.of(1.528172015761821E9)}), taskId=<TASK>, serverTimestamp=1528172016136, serverId='6290294e-61f8-48e4-a8ae-2f3bd3408c76', slaveId=Optional.absent()}
Processed status update for <TASK> (TASK_FAILED) in 00:00.024 (waited 00:00.000 for lock)
INFO  [2018-06-05 04:13:36,160] com.hubspot.singularity.mesos.SingularityMesosStatusUpdateHandler: Processed status update for <TASK> (TASK_FAILED) in 00:00.024 (waited 00:00.000 for lock)
Removing task <TASK> with resources ports(*):[10377-10377]; cpus(*):1; mem(*):512 of framework Singularity on agent 4fd57a45-39a9-48af-aca7-0faf19ea7d32-S192 at slave(1)@<IP>:5051 (>SLAVE_IP>)
Processing ACKNOWLEDGE call 12202362-f37a-4e50-aecc-f355671853db
Received status update acknowledgement (UUID: 12202362-f37a-4e50-aecc-f355671853db) 
Checkpointing ACK for status update TASK_FAILED (UUID: 12202362-f37a-4e50-aecc-f355671853db)
Failed to get resource statistics for executor ... exited with status 1; stderr='Error response from daemon: no such image: mesos-4fd57a45-39a9-48af-aca7-0faf19ea7d32-S190.6d747a2c-3a52-4af6-b472-45e7603c0978: invalid reference format: repository name must be lowercase
I0605 04:13:35.762698 31256 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 12202362-f37a-4e50-aecc-f355671853db) 
I0605 04:13:35.790557 31256 status_update_manager.cpp:825] Checkpointing UPDATE for status update TASK_FAILED (UUID: 12202362-f37a-4e50-aecc-f355671853db) 
I0605 04:13:35.790520 31256 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: 12202362-f37a-4e50-aecc-f355671853db)
I0605 04:13:36.170769 31256 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 12202362-f37a-4e50-aecc-f355671853db)
I0605 04:13:36.170814 31256 status_update_manager.cpp:825] Checkpointing ACK for status update TASK_FAILED (UUID: 12202362-f37a-4e50-aecc-f355671853db) 
I0605 04:13:36.787250 31255 slave.cpp:4193] Cleaning up executor
Executor '<TASK_HOST>' of framework Singularity exited with status 0



Should Singularity's reconciliation fix this kind of incorrect state between mesos-slave/s and Singularity/Mesos master?

[ssalinas]

Singularity's reconciliation (and that of most mesos frameworks that I know of) is a reconciliation with the master. Singularity periodically checks with the master to be sure that its internal state matches the mesos master's state of the world in terms of task statuses.

For the mesos slave side of the world, it can recover when the slave itself is restarted, but the task pids must keep running for that to happen. For example, if your docker container were to be running, but the mesos slave process itself restarted, it should be able to recover the fact that the docker container is still running. However, if the docker container exits, it is considered failed/killed by the mesos slave. There is no reconciliation done between the mesos slave and docker daemon that I know of. If you are interested in requesting a feature like that from mesos, the apache mesos JIRA is probably the place to turn to