Running state [bug]

[onyxg]

Hi,

we had a bug the other day where a job was running and the docker container was taken down.
The job remained in "running" state after 'docker-compose up' and we were unable to flush it (probably would have needed to manually fix it in the database)

I know this a rare case, but ideally you should flush that state when the go app starts up, or provide a means to force stop an running job whether actually running and taking really long or a job just in a running state (as per the database)

[akornatskyy]

Probability of this bug depends whenever a pending job is currently in retry phase (which could be quite long).

Flushing a status during startup might not be a reliable solution since you might have more than one scheduler running. That is only possible for the running tasks that exceed deadline.