parmentelat/asynciojobs

remove auto calls to co_shutdown(); expose an explicit shutdown() method on schedulers

Closed this issue · 1 comments

Before we introduced nested schedulers, the logic of co_shutdown() was that (Pure)Scheduler.co_shutdown() would primarily send co_shutdown() on all component jobs.
In a flat world, the intention was that to notify all job instances at the end of the main scheduler.

With nested schedulers OTOH, there is a need to alter this protocol, because it would now mean that jobs receive the co_shutdown method when the nested-most scheduler ends its work, i.e. way before the toplevel scheduler is done.

In the context of apssh where co_shutdown() is used to close ssh connections to the nodes, this has devastating effects; users currently need to set keep_connection=True on their SshJob instance to work around this problem.


This being said: maybe this automatic housecleaning business is more harmful than helpful.
One option is to simply remove the automatic calling to scheduler.co_shutdown() at the end of the scheduler, and to expose this as a separate method that must be explicitly called by the library user.

As the title reflects, we're going to go for the latter option; this means

  • removing automatic invokations of (co_)shutdown in (co_)run
  • adding a proper shutdown() method on schedulers - a synchronous wrapper just like run() - returns nothing
  • tweaking the documentation in apssh accordingly;
  • adding a note on that in the R2lab tuto ?