Kluster

Description

Kluster is a simple, lightweight, and easy to use dask-cluster management tool. It combines the efforts of dask-gateway with the ease of use of the Arkitekt stack.

Features

Easy to use Cluster management on a per User basis
- Create clusters
- Scale clusters
- Delete clusters
Integrates tightly with Arkitekt
- Use Lok to manage users, and permissions
- Integrate easily into your existing Arkitekt stack and applicaions
Easy transition from local testing development to production
- Use the same code to managed clusters locally and in production
- Just swap out the deployment backend, and you are good to go

Standalone Installation

Even though Kluster is designed to be used with Arkitekt, it can be used as a standalone tool. In this configuration, Kluster will use docker to spin up a dask-gateway container on your local machine, and then use that gateway to spin up dask-clusters on your local machine.

Currently, Kluster only support single machine clusters, but (of course) support for multi-machine clusters is planned, and will be added soon.

Requirements

Kluster requires that you have docker installed on your machine. If you do not have docker installed, you can find instructions for installing it here. Also we require you to have at least Python3.9.

pip install kluster

Usage

While you can use docker to spin up the gateway container yourself, it is easier to use the integrated python api.

from kluster import deployed 
from kluster.api.schema import create_dask_cluster

with deployed():

    cluster = create_dask_cluster(name="test")

    # Do stuff with your cluster
    gateway = cluster.get_gateway()

    # Scale up your cluster
    gateway.adapt(minimum=1, maximum=10)

    # Do stuff with your gateway
    client = gateway.get_client()

    #// Do stuff with your client

In the above example, we use the deployed context manager to spin up a dask-gateway container on your local machine. On entering the context manager, we create a docker-compose deployment and download the necessary containers. On exit, we stop and remove the containers, all clusters, and all gateways.