Dask Tutorial

This tutorial was last given at SciPy 2017 in Austin Texas. A video is available online.

Dask provides multi-core execution on larger-than-memory datasets.

We can think of dask at a high and a low level

High level collections: Dask provides high-level Array, Bag, and DataFrame collections that mimic NumPy, lists, and Pandas but can operate in parallel on datasets that don't fit into main memory. Dask's high-level collections are alternatives to NumPy and Pandas for large datasets.
Low Level schedulers: Dask provides dynamic task schedulers that execute task graphs in parallel. These execution engines power the high-level collections mentioned above but can also power custom, user-defined workloads. These schedulers are low-latency (around 1ms) and work hard to run computations in a small memory footprint. Dask's schedulers are an alternative to direct use of threading or multiprocessing libraries in complex cases or other task scheduling systems like Luigi or IPython parallel.

Different users operate at different levels but it is useful to understand both. This tutorial will interleave between high-level use of dask.array and dask.dataframe (even sections) and low-level use of dask graphs and schedulers (odd sections.)

Prepare

You should clone this repository

git clone http://github.com/dask/dask-tutorial

and then install necessary packages.

a) Install into an existing environment

You will need the following core libraries

conda install numpy pandas h5py Pillow matplotlib scipy toolz pytables snakeviz dask distributed

You may find the following libraries helpful for some exercises

pip install graphviz cachey

b) Create a new environment

In the repo directory

conda env create -f environment.yml

and then on osx/linux

source activate dask-tutorial

on windows

activate dask-tutorial

c) Use Dockerfile

You can build a docker image out of the provided Dockerfile.

Graphviz on Windows

Windows users can install graphviz as follows

Install Graphviz from http://www.graphviz.org/Download_windows.php
Add C:\Program Files (x86)\Graphviz2.38\bin to the PATH

Alternatively one can use the following conda commands (one installs graphviz and one installs python-bindings for graphviz):

conda install -c conda-forge graphviz
conda install -c conda-forge python-graphviz

Prepare artificial data.

From the repo directory

python prep.py

Launch notebook