yoshoku/llama_cpp.rb

Apple Metal support

mldev94 opened this issue · 2 comments

mldev94 commented

I believe adding the metal cpp files from "LLaMACpp" repository to src and some small changes on "extconf.rb" should be sufficient enough to build the native extension with metal support in case anyone runs sidekiq queues on mac devices to generate the responses 10x faster.

https://github.com/ggerganov/llama.cpp

I couldn't manage to build the gem even without changing the code. I always get "failed build native extensions" with .o files are not found error.

I will try to create a pull request if I can figure out why I get ".o" files are not found error.

Just wanted to bring it up if it's a 5-minute thing to add from your end since you created this gem. I can do my best to help with testing or wherever you need help with.

yoshoku commented

@mldev94
I have released v0.2.0 with metal support. I do not know if it will be 10x faster, but please try it.
https://github.com/yoshoku/llama_cpp.rb/blob/main/CHANGELOG.md#020---2023-06-11

mldev94 commented

Hi yoshuku, it works great. Thank you, it's not 10x but it's definitely 3x faster. It went down from 200-240 seconds of processing time for my long prompts to 40-75 seconds.