Harsh-1309/ploomber

Write maintainable, production-ready pipelines using Jupyter or your favorite text editor. Develop locally, deploy to the cloud. ☁️

Join our community | Newsletter | Contact us | Docs | Blog

Notebooks are hard to maintain. Teams often prototype projects in notebooks, but maintaining them is an error-prone process that slows progress down. Ploomber overcomes the challenges of working with .ipynb files allowing teams to develop collaborative, production-ready pipelines using JupyterLab or any text editor.

Main Features

1. Scripts as notebooks. Open .py files as notebooks, then execute them from the terminal and generate an output notebook to review results.
2. Dependency resolution. Quickly build a DAG by referring to previous tasks in your code; Ploomber infers execution order and orchestrates execution.
3. Incremental builds. Speed up iterations by skipping tasks whose source code hasn't changed since the last execution.
4. Production-ready. Deploy to Kubernetes (via Argo Workflows), Airflow, and AWS Batch without code changes.
5. Parallelization. Run independent tasks in parallel.
6. Testing. Import pipelines in any testing frameworks and test them with any CI service (e.g. GitHub Actions).
7. Flexible. Use Jupyter notebooks, Python scripts, R scripts, SQL scripts, Python functions, or a combination of them as pipeline tasks. Write pipelines using a pipeline.yaml file or with Python.

Community

• Join us on Slack
• Newsletter
• Contact the development team

Resources

• Documentation
• Develop and deploy an ML pipeline in 30 minutes (EuroPython 2021)
• Guest blog post on the official Jupyter blog
• PyData Chicago talk (covers motivation and demo)
• Examples (Machine Learning pipeline, ETL, among others)
• Blog
• Comparison with other tools
• More videos

Installation

Compatible with Python 3.6 and higher.

Install with pip:

pip install ploomber

Or with conda:

conda install ploomber -c conda-forge

Getting started

Use Binder to try out Ploomber without setting up an environment:

Or run an example locally:

# ML pipeline example
ploomber examples -n templates/ml-basic -o ml-basic
cd ml-basic

# if using pip
pip install -r requirements.txt

# if using conda
conda env create --file environment.yml
conda activate ml-basic

# run pipeline
ploomber build

Pipeline output saved in the output/ folder. Check out the pipeline definition in the pipeline.yaml file.

To get a list of examples, run ploomber examples.

Click here to go to our examples repository.