The Kubernetes Bridge
operator is able to submit and monitor long running processes on external systems which have their own cluster/resource manager, including SLURM, LSF and Ray. The implementation is agnostic to the choice of resource manager but assumes the system exposes a HTTP/HTTPS API for its control and management.
This operator can be deployed to submit traditional HPC cluster jobs but also to submit jobs to external systems such as: AI accelerators on dedicated clusters, heterogeneous HPC platforms, or Quantum systems.
We provide specific example Pod
images for
LSF
: aPod
managing jobs inside an IBM Spectrum LSF ClusterQuantum
: aPod
managing jobs inside the IBM Quantum ServiceSLURM
: aPod
managing jobs inside a Slurm clusterRay
: aPod
managing jobs inside a Ray cluster
The Bridge Operator acts as a simple bridge between Kubernetes and the external system where visibility into the external system is limited to the system's HTTP(S) API requests. As a result there is no interaction between the Kubernetes scheduler and the workload manager on the external system.
The operator consists of a Kubernetes controller with Custom Resource Definitions
(CRDs), a Pod
and ConfigMap
.
Once the CR is deployed, the operator is responsible for:
- Validating input parameters - checking if
secret
s and other necessary Kubernetes resources specified in the CR exist. - Creating the ConfigMap with application specific data from the CR Spec. The data is later utilized by the Pod when submitting the job.
- Creating the Pod that submits the HTTP/HTTPS requests to send the job to the external system.
- Reading the job's current state from the ConfigMap (updated by the Pod) and updating the CR status accordingly.
- Monitoring the Pod and other Kubernetes resources.
- Responding to reported errors in the external system.
- Cleaning up resources when the job is deleted.
The operator implementation depends on the Operator SDK which should be installed on your machine. Installation prerequisites are here and include:
- git
- go version 1.18
- docker version 17.03+
- kubectl and access to a Kubernetes cluster of a compatible version. Installation procedures are described here For Mac users, we recommend installing with brew. Tested with version v1.22.2
The Operator's implementation presents a general way of managing Kubernetes resources to create a bridge between an external system and Kubernetes.
The code crucial to submitting and monitoring any actual jobs on the external system is located inside the Pod's
image.
A specific example image for a Pod
managing jobs inside the IBM Spectrum LSF workload management package;
is given in lsf-pod
.
The Operator defines a CRD BridgeJob
which gathers information (required and optional) about the external job.
The operator's logic is implemented in the controller's reconcile()
function and is used to create, and subsequently control,
the ConfigMap
and the Pod
. Both of which are created directly by the operator.
Additionally the operator leverages the following:
- A
Secret
with credentials needed to access the external system from thePod
. Assuming the external system requires authentication, the credentials are expected to exist in advance. - An optional
Secret
with credentials needed to access an S3 object storage bucket used to store input or output files.
Here is a general overview of the CRD YAML with some defaults filled in. All fields can hold custom values.
kind: BridgeJob
apiVersion: bridgejob.ibm.com/v1alpha1
metadata:
name: mybridgejob
spec:
resourceURL: http://mycluster.ibm.com:8080/platform/
resourcesecret: mysecret
imagepullpolicy: Always
updateinterval: 30
jobproperties:
runlimit: 20 # mins
jobdata:
jobScript: /home/batch.sh
scriptlocation: remote
s3storage:
s3secret: mysecret-s3
endpoint: minio-endpoint.us-south.containers.appdomain.cloud
secure: false
s3upload:
bucket: mybucket
files: 'output.txt'
Field | Short description |
---|---|
status.jobstatus |
Status of CR, should reflect status of job in external system |
status.message |
Message providing additional information for DONE/KILLED/FAILED/UNKNOWN jobs |
status.starttime |
Start time from external system filled when job is in finished state |
status.completiontime |
Completion time from external system filled when job is in finished state |
spec.jobproperties
is a map struct of common job properties which can be selected for the job in external system.
Possible job statuses :
PENDING
RUNNING
SUSPENDED
DONE
FAILED
KILLED
UNKNOWN
BridgeJob
is in a finished state when thejobstatus
is in one of the DONE
, KILLED
, FAILED
, or UNKNOWN
states.
When BridgeJob
fails because of missing Kuberentes resources (or data in them), status.message
is filled with a brief explanation.
The controller's reconcile logic can be compared to a finite state machine, where every BridgeJob
CR specifies a state and
actions specific to that state are performed. The possible states can be grouped into two categories as follows:
- finished: the job's state in the
ConfigMap
updated by the Pod isFailed', 'Done', 'Killed', or 'Unknown
- running: the job's state in the
ConfigMap
updated by the Pod isPending', 'Running', or 'Suspended
At the beginning of reconciliation, the controller checks if BridgeJob
is in a finished or running state. At the end of reconciliation,
BridgeJob
's state is updated according to the state in the shared ConfigMap
.
If the S3 secret name is not specified in BridgeJob
yaml, no information regarding S3 will be propagated to the shared ConfigMap
.
If the S3 secret name is specified but it does not exist in the BridgeJob
namespace or information is missing
(either in Secret
or in BridgeJob
such as the S3 endpoint), then the CR will fail.
The Pod
performs other checks, such as checking for valid credentials, bucket existence.
-
Clone this repository
git clone https://github.com/IBM/Bridge-Operator.git
-
cd operator
-
To run:
a) Locally outside the cluster
- Locally from binaries
- Deploy the CRD:
kubectl apply -f config/crd/bases/bridgejob.ibm.com_bridgejobs.yaml
- Edit the file
main.go
, for theflag.BoolVar
as follows: (that is changetrue
tofalse
)
flag.BoolVar(&enableLeaderElection, "leader-elect", false, "Enable leader election for controller manager. "+ "Enabling this will ensure there is only one active controller manager.")
- Then run the follwing command:
make install run
b) Run as a Deployment inside cluster
- Deploy the CRD:
kubectl apply -f config/crd/bases/bridgejob.ibm.com_bridgejobs.yaml
make deploy
- From pre-built resources
- Deploy all resources at once:
kubectl apply -f all.yaml
- Deploy all resources at once:
NOTE: for running on a local Kind cluster comment out all lines in operator/config/prometheus/monitor.yaml before deploying the operator.
For valid examples of Secrets
, see lsfsecret.yaml and s3secret.yaml
in samples/core/secrets
.
For CR examples, see samples/core/operator
and for tutorials see samples/tutorials
.
Before building, make sure kustomize
is installed.
Clone this repository git clone https://github.com/IBM/Bridge-Operator.git
. Then run the following commands:
cd operator
cd config/manager
kustomize edit set image controller=${IMG}:[NEW_VERSION]
cd ../../ # Back to the repo root directory
make docker-build docker-push IMG=${IMG}:[NEW_VERSION]
make deploy IMG=${IMG}:[VERSION] # Deploys controller to K8s cluster specified in ~/.kube/config
kustomize build config/default > all.yaml # Generates all resources that have been deployed