Sticky : Dask
Opened this issue · 0 comments
asmodehn commented
https://www.youtube.com/watch?v=mjQ7tCQxYFQ
https://github.com/dask/dask
Dask seems quite interesting...
Pros - Python Distributed Task scheduler :
- Pythonic, minimal dependencies
- Fine grained task distribution
Cons - Dask does parallelism, not distribution :
- Central Scheduler
- Focus on performance, not on reliability ( what if a server crashes ? )
- Distribution is implicit, hidden from the user
So Dask can be useful for intuitively distributing code, inside one of zmp Node. The distribution is transparent for the user.
This helps define pyzmp targets :
- pythonic
- fully distributed
- reliability focused
- explicit distribution
Dask seems therefore complementary, in the sense that it could fit in one zmp "coprocess" to compute heavy tasks. On the other hand pyzmp could eventually be used to distribute the dask scheduler.
However it remains to be seen if the way dask defines and handle compute task cannot help us define some API for pyzmp usage...