This webinar provides an introduction to distributed computing with Python, we will show how to modify a standard Python script to use multiple CPU cores using the concurrent.futures module from the Python standard library and then the dask package. Then we will leverage dask workers running on other machines to distribute a data processing task and monitor its execution through the live dask dashboard. You will understand the difference between threads and processes, how the Global Interpreter Lock works and principles of distributed computing. All material will be available as Jupyter Notebooks.
notebook_singularity.slrm: SLURM script to launch a Jupyter Notebook job through Singularitydigits_of_pi/: example of parallel programming estimating the digits of pi, usingconcurrent.futuresanddaskdaskscripts:launch_workers.shandlaunch_scheduler.shanddask_workers.slrmlaunch all the components ofdaskdistributed- Slides on Google Docs