How to use dask on Hobart?

[matt-long]

Spinning up a dask cluster on Hobart requires specifying a resource_spec argument to PBSCluster.

cluster = dask_jobqueue.PBSCluster(cores=48,  
                                   processes=16,                                                                                                                                                                                                                                                          
                                   queue='medium',
                                   resource_spec='nodes=1:ppn=48')

cc @andersy005, @jkshuman

[jkshuman]

Thanks @matt-long I met with @andersy005 and he helped me get this up and running. It took under 3 minutes to process 66 years across two cases, whereas with open_mfdataset took over 10 minutes to open 5 years. code was updated to specify memory in Hobart

from dask_jobqueue import PBSCluster
#cores=16 processes = 16: 1 thread per process
cluster = PBSCluster(cores=16,
processes=16,
queue='verylong',
resource_spec='nodes=1:ppn=48',
memory='90Gb')