/20220726_Databricks_Demo_Transfer_Learning_with_MLflow

We will go hands-on with an image classification demo using transfer learning, while leveraging MLflow to track our model experiments on Databricks

Primary LanguageJupyter Notebook

Databricks Demo: Transfer Learning with MLflow

with FourthBrain

We will go hands-on with an image classification of Cats vs Dogs demo using transfer learning, while leveraging MLflow to track our model experiments on Databricks.

You may find more information and a recording of this demo on the FourthBrain website.

If you would like run this demo, please download the two notebook files inside the notebooks folder on Github:

1. Run them on Databricks Community Edition (free).

Note that Community Edition is a free offering of Databricks and there are a few features that are not available on Community Edition, such as the Model Registry (optional code at the end of "Transfer Learning Demo Part 2.ipynb").

(a) First sign up for a free Community Edition account, please refer to this doc if you have questions. You can use this link to sign in if you have an existing account. image

(b) Download this file "Transfer Learning Demo Part 1.ipynb" inside of the notebooks folder on this repo, and import this .ipynb file into your user account using Databricks Community Edition. Refer to this doc under "import a notebook" section.

image

(c) Create a cluster under the "Compute" tab on the left, select Databricks runtime version 10.4 LTS ML, and name the cluster test10.4ML. Please ensure you are using 10.4 LTS ML and not 10.4 LTS as the ML runtime pre-installs many useful libraries, such as TensorFlow & MLflow. Refer to this doc about how to create a cluster. It may take a few minutes to start a cluster.

image

(d) Open the imported notebooks on Databricks from the workspace tab, open "Transfer Learning Demo Part 1" notebook, attach the test10.4ML cluster to your notebook, then click "Run All" on the top to run the notebook. Refer to this doc under the section "Attach a notebook to a cluster".

image

image

(e) You can also try to import and open "Transfer Learning Demo Part 2" notebook similarly, attach it to the cluster test10.4ML, and click "Run All". It may take several minutes to run this notebook.

(f) Important note: you must attach your notebook to a cluster before you can run it. If your cluster is terminated, you need to restart it or re-create a new cluster.

(g) You may check your MLflow experiments by clicking "Experiments" on the top.

2. Or locally (for example, Jupyter notebook):

  • Please follow this link for tensorflow transfer learning tutorial.
  • To use MLflow locally, you may need to add additional code and steps for configuration. These steps may include pip install mlflow to install MLflow locally, !mlflow ui to view the MLflow UI, and !pkill -f gunicorn to stop the UI. Note: it is an exercise for the users, so the full code is not provided here.