Cuda issues

Question

Cuda issues

lianabagh opened this issue 9 months ago · 8 comments

lianabagh commented 9 months ago

Hi,

I am facing with cuda driver issues. Can you specify your nvcc cuda version, torch cuda version and gcc version?

Answer 1 · 2024-02-07T20:49:20.000Z

And also GPU.

Answer 2 · 2024-02-08T13:48:02.000Z

And also GPU driver cuda version ))

Answer 3 · 2024-02-09T10:42:16.000Z

Hi @lianabagh , the code for the ppaer ran on torch==1.10+cu11.6 with CUDA version 11.6
But definitely it should also work with more recent CUDA/torch configurations, as soon as they are aligned. I think the stable build is currently torch==2.2.0 with CUDA version 12.1

The GPU model should not have any importance, as soon as your CUDA driver supports it.

Answer 4 · 2024-02-09T10:47:37.000Z

Also, it can be that ninja poses problems with torch. If the error is caused by ninja, I'd recommend uninstalling both ninja torch and their dependencies, and reinstall them.
Also, don't forget to clean your torch-cuda caches in ~/.cache/torch_extensions after changing your torch / CUDA / ninja versions.

Answer 5 · 2024-02-09T13:06:02.000Z

Thanks a lot for the answers. Yes the problem is with ninja. I will try to reinstall all.

…

On Fri, Feb 9, 2024 at 2:47 PM Jean-Marie Lemercier < ***@***.***> wrote: Also, it can be that ninja poses problems with torch. If the error is caused by ninja, I'd recommend uninstalling both ninja torch and their dependencies, and reinstall them. Also, don't forget to clean your torch-cuda caches in ~/.cache/torch_extensions after changing your torch / CUDA / ninja versions. — Reply to this email directly, view it on GitHub <#15 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIYSWQSBVAPKXF6GY3UHWN3YSX5FJAVCNFSM6AAAAABC55Z47SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZVG4YDANZSG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Answer 6 · 2024-02-10T20:44:40.000Z

The problem is with ninja and it doesn't disappear after doing the changes you suggest. What is your ninja version? On Fri, Feb 9, 2024 at 5:05 PM Liana Baghdasaryan < ***@***.***> wrote:

…

Thanks a lot for the answers. Yes the problem is with ninja. I will try to reinstall all. On Fri, Feb 9, 2024 at 2:47 PM Jean-Marie Lemercier < ***@***.***> wrote: > Also, it can be that ninja poses problems with torch. If the error is > caused by ninja, I'd recommend uninstalling both ninja torch and their > dependencies, and reinstall them. > Also, don't forget to clean your torch-cuda caches in > ~/.cache/torch_extensions after changing your torch / CUDA / ninja > versions. > > — > Reply to this email directly, view it on GitHub > <#15 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AIYSWQSBVAPKXF6GY3UHWN3YSX5FJAVCNFSM6AAAAABC55Z47SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZVG4YDANZSG4> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

Answer 7 · 2024-02-20T08:49:42.000Z

I notice we actually removed the calls to ninja in the current implementation (since it was always causing problems), so I am not sure what your problem is.

Answer 8 · 2024-02-20T08:51:15.000Z

If you 1- clean your torch caches as mentioned 2- restart your machine to reinit your CUDA kernels 3- create a new environment and install the packages using the suggested requirements.txt (and optionally 4- re-restart your machine): does this still happen? and if yes, please share a MWE