Introduction

The following is a Movie Recommendation System Data pipeline implemented within Azure Databricks.

Architecture

Movie ratings data is generated via a simple .NET core application running in an Azure Container instance which sends this data into an Azure Event Hub. The movie ratings data is then consumed and processed by a Spark Structured Streaming (Scala) job within Azure Databricks. The recommendation system makes use of a collaborative filtering model, specifically the Alternating Least Squares (ALS) algorithm implemented in Spark ML and pySpark (Python). The solution also contains two scheduled jobs that demonstrates how one might productionize the fitted model. The first job creates daily top 10 movie recommendations for all users while the second job retrains the model with the newly received ratings data. The solution also demonstrates Sparks Model Persistence in which one can load a model in a different language (Scala) from what it was originally saved as (Python). Finally, the data is visualized with a parameterize Notebook / Dashboard using Databricks Widgets.

DISCLAIMER: Code is not designed for Production and is only for demonstration purposes.

Deployment

Ensure you are in the root of the repository and logged in to the Azure cli by running az login.

Requirements

Azure CLI 2.0
Python virtualenv
jq tool
Check the requirements.txt for list of necessary Python packages. (will be installed by make requirements)

Development environment

The following works with Windows Subsystem for Linux
Clone this repository
cd azure-databricks-recommendation
virtualenv . This creates a python virtual environment to work in.
source bin/activate This activates the virtual environment.
make requirements. This installs python dependencies in the virtual environment.
WARNING: The line endings of the two shell scripts deploy.sh and configure_databricks.sh may cause errors in your interpreter. You can change the line endings by opening the files in VS Code, and changing in the botton right of the editor.

Deploy Entire Solution

To deploy the solution, simply run make deploy and fill in the prompts.
When prompted for a Databricks Host, enter the full name of your databricks workspace host, e.g. https://southeastasia.azuredatabricks.net
When prompted for a token, you can generate a new token in the databricks workspace.
To view additional make commands run make

Data

This solutions makes use of the MovieLens Dataset*

Project Organization

├── LICENSE
├── Makefile           <- Makefile with commands like `make data` or `make deploy`
├── README.md          <- The top-level README for developers using this project.
├── data
│   │
│   └── raw            <- The original, immutable data dump.
├── deploy             <- Deployment artifacts
│   │
│   └── databricks     <- Deployment artifacts in relation to the Databricks workspace
│   │
│   └── deploy.sh      <- Deployment script to deploy all Azure Resources
│
├── notebooks          <- Azure Databricks Jupyter notebooks. 
│
├── references         <- Contains the powerpoint presentation, and other reference materials.
│
├── requirements.txt   <- The requirements file for reproducing the analysis environment, e.g.
│                         generated with `pip freeze > requirements.txt`
│
├── src                <- Source code for use in this project.
    ├── __init__.py    <- Makes src a Python module
    │
    ├── data           <- Scripts to download or generate data
    │
    └── EventHubGenerator  <- Visual Studio solution EventHub Data Generator (Ratings)

Project based on the cookiecutter data science project template. #cookiecutterdatascience

*F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4, Article 19 (December 2015), 19 pages. DOI=http://dx.doi.org/10.1145/2827872

morsh/azure-databricks-recommendation