rapidsai/dask-cuda

`dask cuda` command raises errors with the latest alpha release

vijayaditya opened this issue · 3 comments

vijayaditya commented 
python -m pip install --extra-index-url https://pypi.nvidia.com  dask==2023.8.1 dask_cuda-23.10.0a230829
 dask --version
  dask, version 2023.8.1

 dask cuda
Usage: dask [OPTIONS] COMMAND [ARGS]...
Try 'dask -h' for help.

Error: No such command 'cuda'.

NOT A CONTRIBUTION

pentschev commented

There are 2 things here:

  1. You have a type in your command;
  2. We do not guarantee backwards compatibility and latest nightly Dask-CUDA changes depend on upstream Dask/Distributed.

Due to the typos in your command, with the above you should have seen an error:

ERROR: Could not find a version that satisfies the requirement dask_cuda-23.10.0a230829 (from versions: none)
ERROR: No matching distribution found for dask_cuda-23.10.0a230829

Here's the correct command, fixing your typo and installing Dask/Distributed from upstream:

python -m pip install --extra-index-url https://pypi.nvidia.com  [dask==2023.8.1](git+https://github.com/dask/dask@main git+https://github.com/dask/distributed@main) dask-cuda==23.10.0a230829
vijayaditya commented

Sorry for the typo 🙈 it was an error during formatting of the markdown.
Thanks for clarifying that

latest nightly Dask-CUDA changes depend on upstream Dask/Distributed.

On a more general note I frequently rely on pre-release dask-cuda as it unpins the dask versions which enables co-existence with other dask packages. e.g. a few months back we had

dask-cuda 23.6.0 depends on distributed==2023.3.2.1
dask[complete] 2023.3.2 depends on distributed==2023.3.2; extra == "complete"

Could you please let me know if it is possible to have more generous version bounds in the release versions of dask-cuda ?

pentschev commented

Could you please let me know if it is possible to have more generous version bounds in the release versions of dask-cuda ?

Unfortunately this is not currently possible. We've discussed this matter extensively, #848 is one such discussion, but the problem is Dask and Distributed constantly introduce breaking changes making it impossible for us to guarantee forward-compatibility, and backwards-compatibility would increase the RAPIDS test matrix exponentially and make it time and cost-prohibitive, thus we've decided to only guarantee compatibility with to one Dask release, which is hard-pinned at release time. We constantly discuss alternatives, but so far we couldn't find any solutions that are both reliable and cost-effective.

  • 👍1