PyTorch (GPU) not working on Win 11 box

Question

PyTorch (GPU) not working on Win 11 box

tomvaiuw opened this issue 6 months ago · 4 comments

Hi,
I am trying to get Casanovo running on a Windows 11 box with GEForce RTX 3070 and following the instructions I could not get it work. I have Casanovo running but without using the GPU.
I first tried to install it several ways (mostly probably wrong), installing NVIDIA driver directly from NVIDIA website. Not sure if I got all appropriate drivers installed. But I believe I did. Then tried following the instruction in Casanovo FAQ page, uninstalling torch, then following installation as described on Pytorch website (and using the conda command it provided), still Casanovo would not see GPU.
Then I tried the >conda install -c nvidia cuda-toolkit and got error at the very end:
C:\Windows\System32>CALL C:\ProgramData\Anaconda3\etc\conda\activate.d\vs2017_get_vsinstall_dir.bat
Did not find VSINSTALLDIR
File Not Found
Windows SDK version found as: ""
The system cannot find the path specified.
Did not find VSINSTALLDIR

C:\Windows\System32>if defined INCLUDE (set "INCLUDE_CONDA_NVCC_BACKUP=" )

C:\Windows\System32>set "INCLUDE=;\targets\x64"

(base) C:\Windows\System32>

Any suggestion and advice would be greatly appreciated.

Thanks,

Tomas Vaisar

Answer 1 · 2024-07-09T17:09:24.000Z

Hi Tomas,

Could you try opening a terminal window and running the command nvidia-smi? This should tell you if CUDA is installed, and if it is installed what version is installed. The output should look something like this:

You should see the CUDA version in the top right corner of the output. Sometimes if there's a mismatch between the CUDA version specified in the PyTorch installation and the actual CUDA version installed on your system PyTorch won't be able to find your GPU.

As far as the specific error message your seeing, it looks like the error originates from a package that simply locates the installation directory of vscode. You might find some answers about resolving this specific error message here: https://stackoverflow.com/questions/68780561/long-error-messages-about-vsinstalldir-and-vs2019-whenever-i-start-anaconda-prom

Answer 2 · 2024-07-09T17:43:06.000Z

Thanks a lot for the response. The CUDA is installed (v 12.5) - the command you sent worked. Regarding the other issue - that post did not help unfortunately. Is specific version of Visual Studio required? It appears that that it is looking for VS 2017 and I have VS 2022 installed. Thanks, Tomas

…

________________________________ From: Gwenneth Straub ***@***.***> Sent: Tuesday, July 9, 2024 10:09 AM To: Noble-Lab/casanovo ***@***.***> Cc: Tomas Vaisar ***@***.***>; Author ***@***.***> Subject: Re: [Noble-Lab/casanovo] PyTorch (GPU) not working on Win 11 box (Issue #353) Hi Tomas, Could you try opening a terminal window and running the command nvidia-smi? This should tell you if CUDA is installed, and if it is installed what version is installed. The output should look something like this: image.png (view on web)<https://urldefense.com/v3/__https://github.com/Noble-Lab/casanovo/assets/16980750/81d7dace-d1d9-4fe8-ba6b-a2ac28509c53__;!!K-Hz7m0Vt54!nv0sIfGEiMtsPQ70ghpvv7paCVuRoHTY1gFlH6oRSMrsY1y6loeuByWJWsjGHQQz5D1fAmxdf_codEg9pklho9c$> You should see the CUDA version in the top right corner of the output. Sometimes if there's a mismatch between the CUDA version specified in the PyTorch installation and the actual CUDA version installed on your system. As far as the specific error message your seeing, it looks like the error originates from a package that simply locates the installation directory of vscode. You might find some answers about resolving this specific error message here: https://stackoverflow.com/questions/68780561/long-error-messages-about-vsinstalldir-and-vs2019-whenever-i-start-anaconda-prom<https://urldefense.com/v3/__https://stackoverflow.com/questions/68780561/long-error-messages-about-vsinstalldir-and-vs2019-whenever-i-start-anaconda-prom__;!!K-Hz7m0Vt54!nv0sIfGEiMtsPQ70ghpvv7paCVuRoHTY1gFlH6oRSMrsY1y6loeuByWJWsjGHQQz5D1fAmxdf_codEg90_NU2zo$> — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/Noble-Lab/casanovo/issues/353*issuecomment-2218252239__;Iw!!K-Hz7m0Vt54!nv0sIfGEiMtsPQ70ghpvv7paCVuRoHTY1gFlH6oRSMrsY1y6loeuByWJWsjGHQQz5D1fAmxdf_codEg9XGF7uDc$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AYEGTBFEJMU2L2ATMWOS3MTZLQKNXAVCNFSM6AAAAABKN3JOYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJYGI2TEMRTHE__;!!K-Hz7m0Vt54!nv0sIfGEiMtsPQ70ghpvv7paCVuRoHTY1gFlH6oRSMrsY1y6loeuByWJWsjGHQQz5D1fAmxdf_codEg9DqDEObY$>. You are receiving this because you authored the thread.Message ID: ***@***.***>

Answer 3 · 2024-07-09T23:30:51.000Z

A specific version of Visual Studio is not required, nor is Visual Studio required at all actually. It looks like something went wrong when installing one of the dependencies for nvidia cuda-toolkit, which AFIK is not required to be installed explicitly in order to use Casanovo with a GPU.

In order to get a better understanding of what's going on, try opening a new terminal window and activating the Casanovo conda environment with conda activate casanovo_env. Then, start an interactive python session by typing the command python. Inside of the interactive python session, first import Pytorch with import torch, and then check to see if PyTorch can see the GPU by typing torch.cuda.is_available(). This should output true. The entire process should look something like this:

If you see False instead of True for the output of torch.cuda.is_available(), try reinstalling PyTorch with:

pip3 install torch torchvision torchaudio --force-reinstall --index-url https://download.pytorch.org/whl/cu121

The --force-reinstall will ensure that the correct PyTorch build that supports CUDA 12.5 is installed.

Answer 4 · 2024-07-10T17:43:45.000Z

Thanks a lot. I ended up uninstalling all the NVIDA junk I installed, uninstalled Anaconda (and reinstalled, had to downgrade python for Fragpipe to work) and then I ran the (base) C:\Windows\System32>conda install -c nvidia cuda-toolkit from this I ended up getting the errors about visual studio I was seeing before. I uninstalled the cuda-toolkit - and the error disappeared. Then I followed your instructions - on the first "force-reinstall" of the pytorch it did not seem to work, but I repeated it making sure I do it within the casanovo environment and this time it works. I can see that the GPU is available by the > torch.cuda.is_available() And when I run casanovo I see it is now using the GPU. Although I got some warnings: (casanovo_env) c:\Users\vaisarlab\Documents\HPPG2023\24Jun_mAb>casanovo sequence -o results.mztab *.mgf WARNING: Dataloader multiprocessing is currently not supported on Windows or MacOS; using only a single thread. Seed set to 454 INFO: Model weights file C:\Users\vaisarlab\AppData\Local\casanovo\casanovo_nontryptic_v4_2_0.ckpt retrieved from local cache INFO: Casanovo version 4.2.1 INFO: Sequencing peptides from: INFO: sample_preprocessed_spectra.mgf GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores HPU available: False, using: 0 HPUs INFO: Reading 1 files... sample_preprocessed_spectra.mgf: 128spectra [00:00, 4046.27spectra/s] You are using a CUDA device ('NVIDIA GeForce RTX 3070') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] Predicting DataLoader 0: 0%| | 0/1 [00:00<?, ?it/s]WARNING: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:455.) WARNING: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:455.) Predicting DataLoader 0: 100%|███████████████████████████████████████████████████████████| 1/1 [00:02<00:00, 0.47it/s] INFO: DONE! Thank you, Tomas

…

________________________________ From: Gwenneth Straub ***@***.***> Sent: Tuesday, July 9, 2024 4:31 PM To: Noble-Lab/casanovo ***@***.***> Cc: Tomas Vaisar ***@***.***>; Author ***@***.***> Subject: Re: [Noble-Lab/casanovo] PyTorch (GPU) not working on Win 11 box (Issue #353) A specific version of Visual Studio is not required, nor is Visual Studio required at all actually. It looks like something went wrong when installing one of the dependencies for nvidia cuda-toolkit, which AFIK is not required to be installed explicitly in order to use Casanovo with a GPU. In order to get a better understanding of what's going on, try opening a new terminal window and activating the Casanovo conda environment with conda activate casanovo_env. Then, start an interactive python session by typing the command python. Inside of the interactive python session, first import Pytorch with import torch, and then check to see if PyTorch can see the GPU by typing torch.cuda.is_available(). This should output true. The entire process should look something like this: image.png (view on web)<https://urldefense.com/v3/__https://github.com/Noble-Lab/casanovo/assets/16980750/4d2ec495-b433-4f02-99ed-78b70d84c03b__;!!K-Hz7m0Vt54!lktk8UGG8XHAL03AHDqshE8TO3QLnyQNHORKd5jcUKMI_5oAhAvATlkmlqc_rw5UGHBMbZCkpgLL2wPbAgcHHxM$> If you see False instead of True for the output of torch.cuda.is_available(), try reinstalling PyTorch with: pip3 install torch torchvision torchaudio --force-reinstall --index-url https://download.pytorch.org/whl/cu121<https://urldefense.com/v3/__https://download.pytorch.org/whl/cu121__;!!K-Hz7m0Vt54!lktk8UGG8XHAL03AHDqshE8TO3QLnyQNHORKd5jcUKMI_5oAhAvATlkmlqc_rw5UGHBMbZCkpgLL2wPbVbW6s7c$> The --force-reinstall will ensure that the correct PyTorch build that supports CUDA 12.5 is installed. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/Noble-Lab/casanovo/issues/353*issuecomment-2218978080__;Iw!!K-Hz7m0Vt54!lktk8UGG8XHAL03AHDqshE8TO3QLnyQNHORKd5jcUKMI_5oAhAvATlkmlqc_rw5UGHBMbZCkpgLL2wPbqMpXaic$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AYEGTBDSKQE545LJF4KDMYLZLRXEDAVCNFSM6AAAAABKN3JOYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJYHE3TQMBYGA__;!!K-Hz7m0Vt54!lktk8UGG8XHAL03AHDqshE8TO3QLnyQNHORKd5jcUKMI_5oAhAvATlkmlqc_rw5UGHBMbZCkpgLL2wPbsj51JPE$>. You are receiving this because you authored the thread.Message ID: ***@***.***>