Run your workflow on JupyterHub!
Run your ML job right away on Kubernetes with jupyterflow.
- No container image build & push
- No Kubernetes manifest (YAML)
- Just simply run your ML code with single command
jupyterflow.
jupyterflow is a ML tool for Data Scientist to help run ML code on Kubernetes without any containerization process.
- Launch your jupyter notebook
- Write your ML code
- Run your ML model on Kubernetes through
jupyterflow
# Write your code.
echo "print('hello')" > hello.py
echo "print('world')" > world.py
# Install jupyterflow.
pip install jupyterflow
# in jupyterflow `>>` directive expresses container dependencies similar to Airflow.
jupyterflow run -c "python hello.py >> python world.py"
- I wanted to run ML models across multiple training server efficiently.
- Using Kubernetes was a good idea, since
- it is easy to run jobs distributedly.
- it is easy to schedule jobs on multiple training server.
- it has native resource management mechanism.
- it has good monitoring system.
- But there were some drawbacks.
- I needed to re-build & re-push image everytime I updated my model. This was painful.
- People who were not familiar with k8s had a hard time writing K8s manifest file.
JupyterFlow aims to solve this problem. Run your workflow on JupyterHub with single command without containerization & k8s troublesome task. For more details, read this article.
JupyterFlow only works on following platforms:
Therefore, although using JupyterFlow does not require Kubernetes manifest, setting up JupyterFlow requires Kubernetes understandings(YAML, helm, Service). If you're familiar with Kubernetes, it will not be too hard.
If you want to know why there is such limitation, refer to How it works guide.
To set up jupyterflow and start running your first workflow, follow the Getting Started guide.
To learn how it works, go to How it works guide.
For examples how to use, please see Examples page.
To find out more configuration, take a look at Configuration page.
For more detail usage of jupyterflow command line interface, find out more at CLI Reference page.