markusweimer/azureml-and-dask

Shows how to spin up a DASK cluster on AML Compute

Running Dask on AzureML

This repository shows how to run a Dask cluster on an AzureML Compute cluster. It is designed to run on an AzureML Notebook VM (created after 8/15/2019), but it should work on your local computer, too.

Please follow these setup instructions and then start:

• here for plain DASK interactive scenarios interactive/StartDask.ipynb.
• here for DASK with NVIDIA RAPIDS interactive scenarios rapids_interactive/start_cluster.ipynb.

Setting up the Python Environment

The environment you are running should have the latest version of dask and distributed installed -- run this code in the terminal to make sure:

    conda activate py36  # assuming AzureML Notebook VM
    pip install --upgrade dask distributed

Or, if you want to be on the safe side, create a new conda environment using this environment.yml file like so:

    conda env create -f dask/environment.yml  
    conda activate dask
    python -m ipykernel install --user --name dask --display-name "Python (dask)"