persimmon-ai-labs/adept-inference

Llama.cpp Support

Closed this issue · 2 comments

loretoparisi commented

Exploring possibilities to support GGML / GGUF formats to run with Llama.cpp

VincentJGeisler commented

the model is missing some keys and count be converted to GGUF format

'rms_norm_eps'

maddes8cht commented

A full set of Llama.cpp compatible .gguf files is available at
https://huggingface.co/maddes8cht/adept-persimmon-8b-base-gguf
and
https://huggingface.co/maddes8cht/adept-persimmon-8b-chat-gguf
For the moment, cuda accelleration seems not to work, so you need to use -ngl 0 with the cublas versions.

  • 🚀2