/dask-remote-jobqueue

A custom dask remote jobqueue for HTCondor.

Primary LanguagePythonMIT LicenseMIT

dask-remote-jobqueue

A custom Dask remote jobqueue for HTCondor, using ssh forwarding via ssh-jhub-forwarder to enable remote worker nodes (running on a private network) to expose Dask scheduler services outside the network boundary.

The custom class spawn a Dask Scheduler inside the HTCondor cluster with a companion HTTP Controller to interact with the Dask Cluster, for example, with actions like:

  • start
  • stop
  • scale

The service connections are forwarded to the Dask Client and the Dask Scheduler interact directly with the Dask Worker Nodes.

                     SSH Forward                  HTCondor Cluster
                       Service               ┌─────────────────┬──────────────┐
┌────────┐       ┌───────┬──┬───────┐        │                 │              │
│        │       ├───────┤  ├───────┤        │ Dask Scheduler  │◄──┐  ┌────┐  │
│        │       │       │  │       │◄───────┤                 │   ├─►│ WN │  │
│ Dask   │       │ssh    │  │   ssh │        │      ┌──────────┤   │  └────┘  │
│ Client │◄──────┤listen │  │   fwd │◄───────┼──────┤          │   │          │
│        │       │       │  │       │        │      │Dashboard │   │  ┌────┐  │
│        │       │       │  │       │◄──┐    ├──────┴──────────┤   ├─►│ WN │  │
└────────┘       ├───────┤  ├───────┤   │    │   ▲             │   │  └────┘  │
                 └───────┴──┴───────┘   │    │   │             │   │          │
                                        │    │   │             │   │  ┌────┐  │
                                        │    │   ▼             │   └─►│ WN │  │
                                        │    ├─────────────────┤      └────┘  │
                                        └────┤ HTTP Controller │              │
                                             └─────────────────┴──────────────┘