Feature request: GPU passthrough for LLamaGPT
highghlow opened this issue · 0 comments
highghlow commented
I upgraded my server recently and tried to run LLamaGPT on it. It bombs the CPU and doesn't touch the gpu at all. To run AI on a GPU it has to be from Nvidia and the OS has to use the proprietary nvidia driver, so it should probably be an optional feature.
Useful links:
https://wiki.debian.org/NvidiaGraphicsDrivers#Debian_12_.22Bookworm.22
https://docs.docker.com/compose/gpu-support/
https://www.tensorflow.org/guide/gpu NOTE: some distros have a separate tensorflow_gpu package