This example is using TensorFlow, Horovod, and Nuclio demonstrating end to end solution for image classification, it consists of 4 MLRun and Nuclio functions and Kubeflow Pipelines Orchastration:
- Download: import an image archive from S3 to the cluster file system
- Label: Tag the images based on their name structure
- Traing: Distrubuted training using TF, Keras and Horovod
- Inference: Automated deployment of Nuclio model serving function (form Notebook and from Dockerfile)
The demo is written as an MLRun Project, allowing you to use the load project to run the project through the main workflow as specified in the project.yaml
The demo applications are tested on the Iguazio's Data Science PaaS, and use Iguazio's shared data fabric (v3io), and can be modified to work with any shared file storage by replacing the
apply(v3io_mount())calls with other KubeFlow volume modifiers (e.g.apply(mlrun.platforms.mount_pvc())) . You can request a free trial of Iguazio PaaS.
Pre-requisites:
- A Kubernetes cluster with pre-installed KubeFlow, Nuclio.
- MLRun Service and UI installed, see MLRun readme.
Deployment Steps
- Clone this repo to your own Git.
- in a client or notebook properly configured with MLRun and KubeFlow run:
mlrun project my-proj/ -u git://github.com/<your-fork>/demo-image-classification.git
-
Run the local playground notebook to build, test, and run functions.
-
Open the project notebook and follow the instructions to run an automated ML Pipeline and source control.
Note: alternatively you can run the
mainpipeline from the CLI and specify artifacts path using:
mlrun project my-proj/ -r main -p "v3io:///users/admin/kfp/{{workflow.uid}}/"