Adding CUDA SDK Toolkit ?
adrastee opened this issue · 26 comments
Hi,
Using dizcza/docker-hashcat (latest
tag), and starting hashcat -b,
it outputs :
* Device #1: CUDA SDK Toolkit installation NOT detected.
CUDA SDK Toolkit installation required for proper device support and utilization
Falling back to OpenCL Runtime
While it's just a warning, it seems to have bad effects on performance/stability.
Could you kindly add CUDA SDK Toolkit installation to the docker file?
Thanks!
Yeap. I was thinking of adding CUDA support. But I don't have a GPU anymore to test it.
Would you be willing to test the docker build? This will probably require changing the dockerfile once I push a draft - the way I think it should work - and building it manually if the installation fails.
Ok for testing! Let me know which docker I should load.
BTW CUDA is quicker than OpenCL according to the last version of hashcat.
Actually, this might be easier than I thought. I just replaced manual OpenCL driver installation instructions with
FROM nvidia/cuda:10.2-runtime-ubuntu18.04
as you can find in the "cuda" branch. Check it out and give me the feedback whether it works or not:
dizcza/docker-hashcat:cuda
Sadly it does not work.
Using dizcza/docker-hashcat:cuda on a 1080 Ti:
$ hashcat -I
hashcat (v6.0.0-3-g53fb4050) starting...
clGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR
ATTENTION! No OpenCL-compatible or CUDA-compatible platform found.
You are probably missing the OpenCL or CUDA runtime installation.
* NVIDIA GPUs require this runtime and/or driver (both):
"NVIDIA Driver" (440.64 or later)
"CUDA Toolkit" (9.0 or later)
$ nvidia-smi
Fri Jun 19 18:05:31 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... On | 00000000:08:00.0 Off | N/A |
| 23% 56C P8 10W / 125W | 1MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
Damn. Nvidia team cannot even ship a proper docker image for their products - the same situation happened with their opencl docker image, as you mentioned in your one-year-old last issue.
Please do the steps described in this post. After you remove all cuda-related drivers, the official website proposes the next installation steps:
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget http://developer.download.nvidia.com/compute/cuda/11.0.1/local_installers/cuda-repo-ubuntu1804-11-0-local_11.0.1-450.36.06-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-11-0-local_11.0.1-450.36.06-1_amd64.deb
sudo apt-key add /var/cuda-repo-ubuntu1804-11-0-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda
which you can easily find by following https://developer.nvidia.com/cuda-downloads.
Once you're done, please report.
I think your should load nvidia/cuda:10.2-devel-ubuntu18.04
instead of nvidia/cuda:10.2-runtime-ubuntu18.04
runtime
docker does NOT include CUDA-toolkit, while devel
does => no need to uninstall/install new things.
devel: Builds on the runtime and includes headers, development tools for building CUDA images
Then simply download hashcat:
$ hashcat -I
hashcat (v6.0.0-3-g53fb4050) starting...
clGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR
CUDA Info:
==========
CUDA.Version.: 10.2
Backend Device ID #1
Name...........: GeForce GTX 1080 Ti
Processor(s)...: 28
Clock..........: 1582
Memory.Total...: 11178 MB
Memory.Free....: 11039 MB
$ hashcat -b -m 2500 -w 4
hashcat (v6.0.0-3-g53fb4050) starting in benchmark mode...
clGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR
CUDA API (CUDA 10.2)
====================
* Device #1: GeForce GTX 1080 Ti, 11039/11178 MB, 28MCU
Benchmark relevant options:
===========================
* --workload-profile=4
Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4095)
Speed.#1.........: 603.5 kH/s (357.10ms) @ Accel:64 Loops:512 Thr:1024 Vec:1
Issue:
We notice the error CL_PLATFORM_NOT_FOUND_KHR. Maybe because OpenCL is not properly installed? clinfo
returns 0 device, so it confirms OpenCL platforms is not installed.
I don't really know if OpenCL is mandatory when CUDA is here.
Side note , using dizcza/docker-hashcat:latest (=> using OpenCL only, no CUDA), performance on WPA (mode 2500) are lower:
$ hashcat -b -w4 -m2500
hashcat (v6.0.0) starting in benchmark mode...
* Device #1: CUDA SDK Toolkit installation NOT detected.
CUDA SDK Toolkit installation required for proper device support and utilization
Falling back to OpenCL Runtime
OpenCL API (OpenCL 1.2 CUDA 10.0.206) - Platform #1 [NVIDIA Corporation]
========================================================================
* Device #1: GeForce GTX 1080 Ti, 11008/11178 MB (2794 MB allocatable), 28MCU
Benchmark relevant options:
===========================
* --workload-profile=4
Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4095)
Speed.#1.........: 578.0 kH/s (395.67ms) @ Accel:64 Loops:512 Thr:1024 Vec:1
Roughly +5% with CUDA 10.2..
Cool, thank you! Then I'll switch to devel, as you suggest.
I don't really know if OpenCL is mandatory when CUDA is here.
According to the description opencl is not mandatory if cuda is installed. Moreover, I get the impression that having cuda only is recommended if your device supports it because hashcat priorities cuda (if it's available) over opencl and thus installing opencl-related software will only contaminate your system unless you use them elsewhere.
runtime docker does NOT include CUDA-toolkit, while devel does
Hm, this is strange. Did I misinterpret their nvidia docker description...
runtime: Builds on the base and includes the CUDA math libraries, and NCCL. A runtime image that also includes cuDNN is available.
devel: Builds on the runtime and includes headers, development tools for building CUDA images. These images are particularly useful for multi-stage builds.
I guess the warning "clGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR" tells not only that ocl-icd-libopencl1
(requires by clinfo
) is missing system-wide but also that hashcat does not see anything in /etc/OpenCL/vendors/
folder. I think this warning can be ignored.
Here is what I did:
nvidia/cuda:10.2-devel-ubuntu18.04
for thecuda
tag.- latest = cuda + opencl, meaning that apart from
nvidia/cuda:10.2-devel-ubuntu18.04
I kept opencl-related installation.
I'd be nice if you try :latest
tag (update it beforehand) and give me the feedback if it works for you. It should also get rid of the warning you see with cuda-only backend.
Something strange has changed here, I've been unable to load the :latest
tag with Vast.ai sessions, :pocl
loads correctly and I can drop in to an attached shell, but neither :latest
nor :cuda
seem to start and run correctly.
Using cuda
tag (from nvidia/cuda:10.2-devel-ubuntu18.04) : works awesome!
And weird thing : it has nvidia version 450
+ CUDA 11.0
- not consistent with the name..
$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.36.06 Driver Version: 450.36.06 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+=================
$ hashcat -I
hashcat (v6.0.0-4-g5628317d) starting...
clGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR
CUDA Info:
==========
CUDA.Version.: 11.0
Backend Device ID #1
Name...........: GeForce GTX 1080 Ti
Processor(s)...: 28
Clock..........: 1632
Memory.Total...: 11178 MB
Memory.Free....: 11039 MB
$ hashcat -b -m 2500 -w 4
...
Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4095)
Speed.#1.........: 651.1 kH/s (351.21ms) @ Accel:64 Loops:512 Thr:1024 Vec:1
As expected, CUDA 11.0 speed for this mode is +13% compared to opencl :)
@msoose : don't need OpenCL anymore, and the cuda tag works ok on Vastai as well.
@adrastee Great, thanks for the update, just confirmed :cuda
tag does work correctly on Vast instances. Excellent.
Could then anyone test :opencl-cuda
tag? I put OpenCL + CUDA back again in a separate branch:
docker pull dizcza/docker-hashcat:opencl-cuda
What's the output of hashcat -I
?
@dizcza :opencl-cuda
also works.
hashcat -I
with some snipping out of unwanted information:
CUDA Info:
==========
CUDA.Version.: 10.2
OpenCL Info:
============
OpenCL Platform ID #1
Vendor..: NVIDIA Corporation
Name....: NVIDIA CUDA
Version.: OpenCL 1.2 CUDA 10.2.131
Version........: OpenCL 1.2 CUDA
OpenCL.Version.: OpenCL C 1.2
Driver.Version.: 440.59
All right, thank you. I reverted the changes so that latest = cuda + opencl
again. I hope that the previous fail was a mere coincidence.
Talking about speed, you might squeeze an extra few percent out of your gpu capabilities if you run the computation not in a docker container. That very much depends on what you're doing on top of this image. If you're into dockering and use it primarily for wpa hashes, take a look at a repo of mine: https://github.com/dizcza/hashcat-wpa-server.
Feel free to post here cuda-related issues, if you encounter any.
Thanks a lot!
Hi again,
Using dizcza/docker-hashcat:cuda
on latest 3090 GPUs, I got the error hashcat nvrtcCompileProgram(): NVRTC_ERROR_INVALID_OPTION
It works great with docker image nvidia/cuda:11.1-devel-ubuntu18.04
Maybe you can FROM
this one?
Best.
Thanks for the heads-up!
I've pushed CUDA 11.1 devel into the cuda branch and left 10.2 in the master.
I'll keep cuda branch as experimental and master with CUDA latest_major_version-1
.
Feel welcome to post here next time the 11.1 doesn't work to bump up the version again, as I'm not using it anyway and cannot capture outdated software errors.
I am getting an error:
nvrtcCompileProgram(): NVRTC_ERROR_INVALID_OPTION
nvrtc: error: invalid value for --gpu-architecture (-arch)
* Device #1: Kernel /usr/local/share/hashcat/OpenCL/shared.cl build failed.
* Device #1: Kernel /usr/local/share/hashcat/OpenCL/shared.cl build failed
```.
This happened on many hosts that I have tried the container on.
I will try dizcza/docker-hashcat:nvidia-full next
@DenkertM you probably have an outdated GPU card. With CUDA 11 release, Kepler architecture support has been dropped. Try running the latest
branch, not cuda
. The latest
tag comes with CUDA 10.2 which supports sm_30 and sm_32 compute capabilities.
I've just tested on Nvidia GTX 760 which is a quite old card - sm_30 arch. The cuda
tag fails to run a benchmark with the error you described. The latest
tag runs flawlessly.
I am wondering if this was a driver issue? It was running on a remote RTX 3060 when the issue happened, and when I used :cuda on a 2x RTX 3080 remote host it was able to start the benchmark successfully. Tonight I can do a direct A/B, I didn’t have time last night. Is there a configuration you think I should look at?
Ah, so it's the opposite: your card is one of the newest.
Let's settle it down.
It should be a CUDA incompatibility issue (not Nvidia driver).
nvidia-full
tag is obsolete, outdated, and is about to be removed. It was built at the time when HashCat had no CUDA support whatsoever.
Which tag triggers the error you described - latest
or cuda
? Try running both. If cuda
runs and latest
fails, CUDA 11 must be used. If latest
runs and cuda
fails, CUDA 10 must be used.
If none of the tags works, you can change the base image, the CUDA version, at the first line of the Dockerfile. The valid cuda images can be found on nvidia hub docker.
@dizcza Would you consider building an arm64 version of the image? I'm trying to run this on an Nvidia Jetson Nano
@lawndoc well yes, if you test it - I don't have any arm64 hardware to test.
Let me have a look.
I should be able to test it, but I might need a bit to figure out how to upgrade the cuda version on the jetson. I tried building locally and when I run I get an error that says it's not detecting 11.8 (though I did install it). I am using an older version of Jetpack that might not work nicely with newer cuda versions, so I'm going to start with reflashing Jetson OS and starting with the newest Jetpack.
Actually nevermind, I ended up getting hashcat working locally. Sorry, and thanks for the consideration!