KubeJobs

A Python package for creating and running Kubernetes Jobs, Persistent Volumes (PVs), and Persistent Volume Claims (PVCs). This package simplifies the process of deploying and managing jobs, PVs, and PVCs on Kubernetes, making it easier for users without Kubernetes experience to get started.

Features

Easy creation and management of Kubernetes Jobs through Python code.
Supports GPU selection, shared memory size configuration, environment variables, and volume mounts.
Initialize the KubernetesJob class using command-line arguments with Google Fire.
Automatically generates YAML files for Kubernetes Jobs.

Installation

Install KubeJobs using pip:

pip install git+https://github.com/AntreasAntoniou/kubejobs.git

Usage

KubernetesJob

The KubernetesJob class helps you create a Kubernetes Job, generate its YAML configuration, and run the job. Kubernetes Jobs are useful for running short-lived, one-off tasks in your cluster.

from kubejobs import KubernetesJob

# Create a Kubernetes Job with a name, container image, and command
job = KubernetesJob(
    name="my-job",
    image="ubuntu:20.04",
    command=["/bin/bash", "-c", "echo 'Hello, World!'"],
)

# Generate the YAML configuration for the Job
print(job.generate_yaml())

# Run the Job on the Kubernetes cluster
job.run()

create_jobs_for_experiments

The create_jobs_for_experiments function allows you to create and run a series of Kubernetes Jobs for a list of commands. This can be helpful for running multiple experiments or tasks with different parameters in parallel.

from kubejobs import create_jobs_for_experiments

# List of commands to run as separate Kubernetes Jobs
commands = [
    "python experiment.py --param1 value1",
    "python experiment.py --param1 value2",
    "python experiment.py --param1 value3",
]

# Create and run Kubernetes Jobs for each command in the list
create_jobs_for_experiments(
    commands,
    image="nvcr.io/nvidia/cuda:12.0.0-cudnn8-devel-ubuntu22.04",
    gpu_type="nvidia.com/gpu",  # Request GPU resources
    gpu_limit=1,                # Number of GPU resources to allocate
    backoff_limit=4,            # Maximum number of retries before marking job as failed
)

create_pvc

The create_pvc function helps you create a Persistent Volume Claim (PVC) in your Kubernetes cluster. PVCs are used to request storage resources from your cluster, allowing your applications to store and retrieve data.

from kubejobs import create_pvc

# Create a PVC with a name, storage size, and access mode
create_pvc("my-pvc", "5Ti", access_modes=["ReadWriteOnce"])

create_pv

The create_pv function helps you create a Persistent Volume (PV) in your Kubernetes cluster. PVs represent physical storage resources in a cluster, which can be consumed by PVCs. This allows you to manage storage resources independently from applications that use them.

from kubejobs import create_pv

# Create a PV with a name, storage size, storage class, access mode, and other properties
create_pv(
    pv_name="my-pv",
    storage="1500Gi",
    storage_class_name="my-storage-class",
    access_modes=["ReadOnlyMany"],
    pv_type="local",          # Type of Persistent Volume, either 'local' or 'gcePersistentDisk'
    claim_name="my-pvc",      # Name of the PVC to bind to the PV
    local_path="/mnt/data",   # Path on the host for a local PV (required when pv_type is 'local')
)

For more detailed examples and usage information, please refer to the official documentation.

Contributing

Contributions are welcome! If you'd like to contribute, please:

Fork the repository
Create a new branch for your feature or bugfix
Commit your changes and push them to your fork
Open a pull request

License

KubeJobs is released under the MIT License.

aryopg/kubejobs