nils-braun/b2luigi

Automatic batch-system discovery

Opened this issue · 0 comments

When re-reading the b2luigi documentation, I noticed that @nils-braun had written:

In the future, there will even be an automatic discovery of the batch system to use.

The current state is LSF as a default batch system. A very simple approach would be to define a priority order as an iterable, and then if no batch system is given via setting, go through the default batch systems in the list and choose the first one that is available. This check could simply be if the submission executable exists or maybe also if some status command shows that the system is alive.

For a start I would use ("lsf", "htcondor") as defaults, maybe fall back to "local". The only thing that I'm worried with "local" in the default listis a user accidentally sending a job with 800 workers to the login node when the remote batch is down if `"local"``is down and issues like that, so maybe better leave it out. In any case it should be made very obvious to the user which batch system is chosen.