deprecated processes leave garbage in the logs
pbellec opened this issue · 1 comments
@amanbadhwar had a case where she cancelled a pipeline and restarted in the same folder, while all submitted jobs from the first attempt were idle. The jobs eventually started, recognized that a more recent version of the pipeline had been activated (so that feature seems to work!) and exited. But that left a whole bunch of eqsub/oqsub/exit files. Not sure if anything can be done about it, but maybe the garbage collector could archive those.
Progress report: the pipeline fully recovered and is processing full steam. Success! One easy fix would be for the pipeline manager/garbage collector and workers to do some clean up themselves when they start. They could do a short verbose so that the trace is not lost. Would actually be a way to handle proper logging for restarts (right now all files are flushed by the deamon).