Running Dask on AzureML

This repository is no longer maintained. For a simple way of running Dask on an AzureML cluster, please check out the AzureML CLI v2 DASK samples here: https://github.com/Azure/azureml-examples/tree/main/cli

This repository shows how to run a Dask cluster on an AzureML Compute cluster. It is designed to run on an AzureML Notebook VM (created after 8/15/2019), but it should work on your local computer, too.

Please follow these setup instructions and then start:

here for plain DASK interactive scenarios interactive/StartDask.ipynb.
here for DASK with NVIDIA RAPIDS interactive scenarios rapids_interactive/start_cluster.ipynb.

Setting up the Python Environment

The environment you are running should have the latest version of dask and distributed installed -- run this code in the terminal to make sure:

    conda activate py36  # assuming AzureML Notebook VM
    pip install --upgrade dask distributed

Or, if you want to be on the safe side, create a new conda environment using this environment.yml file like so:

    conda env create -f dask/environment.yml  
    conda activate dask
    python -m ipykernel install --user --name dask --display-name "Python (dask)"

danielsc/azureml-and-dask

Running Dask on AzureML

Setting up the Python Environment