Scale complex AI/ML pipelines anywhere

CodeFlare is a framework to simplify the integration, scaling and acceleration of complex multi-step analytics and machine learning pipelines on the cloud.

Its main features are:

Simplified user experience: Interactive and rich command line interface and live dashboards enabling automation to deploy, run and monitor end-to-end pipelines, significantly minimizing the effort and skills needed to scale AI and ML workflows.
Pipeline execution and scaling: Integrated with Ray Workflows, CodeFlare Pipelines faciltates the definition and parallel execution of pipelines. It unifies pipeline workflows across multiple frameworks while providing nearly optimal scale-out parallelism on pipelined computations.

Deploy and integrate anywhere: CodeFlare simplifies deployment and integration by enabling a serverless user experience with the integration with Red Hat OpenShift and IBM Cloud Code Engine and providing adapters and connectors to make it simple to load data and connect to data services.

Quick start

Run on your laptop

Installing CodeFlare client

See instructions here for installing CodeFlare CLI and Dashboard.

Instaling locally

CodeFlare can be installed from PyPI.

Prerequisites:

Python 3.7 or 3.8
JupyterLab (to run examples)

We recommend installing Python 3.8.6 using pyenv. You can find here recommended steps to set up the Python environment.

Install from PyPI:

pip3 install --upgrade pip          # CodeFlare requires pip >21.0
pip3 install --upgrade codeflare

Alternatively, you can also build locally with:

git clone https://github.com/project-codeflare/codeflare.git
cd codeflare
pip3 install --upgrade pip
pip3 install .

Pipeline execution and scaling

⚠ UPDATE
As of January 2022, this feature is now built on Ray Workflows with parts of it in Ray core and the rest in a DAG contribution repository. Please follow these links to contribute to CodeFlare Pipelines.

CodeFlare Pipelines reimagined pipelines to provide a more intuitive API for the data scientist to create AI/ML pipelines, data workflows, pre-processing, post-processing tasks, and many more which can scale from a laptop to a cluster seamlessly.

See the API documentation here, and reference use case documentation in the Examples section.

A set of reference examples are provided as executable notebooks.

To run examples, if you haven't done so yet, clone the CodeFlare project with:

git clone https://github.com/project-codeflare/codeflare.git

Example notebooks require JupyterLab, which can be installed with:

pip3 install --upgrade jupyterlab

Use the command below to run locally:

jupyter-lab codeflare/notebooks/<example_notebook>

The step above should automatically open a browser window and connect to a running Jupyter server.

If you are using any one of the recommended cloud based deployments (see below), examples are found in the codeflare/notebooks directory in the container image. The examples can be executed directly from the Jupyter environment.

As a first example of the API usage, see the sample pipeline.

For an example of how CodeFlare Pipelines can be used to scale out common machine learning problems, see the grid search example. It shows how hyperparameter optimization for a reference pipeline can be scaled and accelerated with both task and data parallelism.

Deploy and integrate anywhere

CodeFlare is built on Red Hat OpenShift Container Platform and can be deployed anywhere, from on-prem to cloud, and integrate easily with other cloud-native ecosystems.

See Running with Red Hat OpenShift for detailed instructions on how to run CodeFlare on OpenShift Container Platform.

Contributing

Join us in making CodeFlare Better! We encourage you to take a look at our Contributing page.

Blog

CodeFlare related blogs are published on our Medium publication.

License

CodeFlare is an open-source project with an Apache 2.0 license.

csgorham/codeflare