Interest in adding Flux Framework
vsoch opened this issue · 3 comments
Hiya!
I'm interested in adding FluxFramework as a connector: https://flux-framework.readthedocs.io/en/latest/quickstart.html and I noticed that for other connectors with queues, the interaction is always an ssh command -> to a command line. Flux has an official Python SDK, however I'm assuming your preference is to not add additional dependencies and rely on command line stuffs? This is definitely do-able, I just want to check first.
I also have a design question about the Kubernetes connector (or how it works in general) but I'll open a separate issue for that. Thanks!
Hi @vsoch,
I know Flux, and a connector for Flux would be highly appreciated :)
You are right that the main interaction with queue managers is through SSH. This is because on many HPC systems you are not allowed to run a job for more than X minutes (with X quite low) on the login node. Before yesterday, remote connection through SSH was the only available path.
This changed yesterday, with the introduction of the ConnectorWrapper class. Now the QueueManagerConnector can wrap an internal connector (e.g. an SSHConnector, but also a LocalConnector) and use it to move data from/to the HPC facility and to run SLURM/PBS commands. The old format, which is now deprecated, automatically generates an internal SSHConnector and falls back to the old behavour. This stuff is still totally undocumented, but I will add examples and documentation in the very next days (or at least, this is the plan XD).
You're totally right when you say that queue managers rely on command line rather than Python clients. This is to allow them to be wrapped inside SSH (or different) commands. However, I am not against Python clients for remote interaction with APIs. The only thing to bear in mind is that StreamFlow is mostly a single-threaded application based on the async/await paradigm. Therefore, API calls should either be declared async or wrapped into external runners to avoid blocking the main thread (and the whole workflow execution).
That sounds fantastic! Could you ping me here when you add the examples / docs? And please no rush - take the time you need!