Atome-FE/llama-node

Ggml v3 support in Llama.cpp

Closed this issue · 4 comments

synw commented

Hi, thanks for this nice package. LLama.cpp made a new breaking change in their quantization methods recently (PR ref).

Would it be possible to get an update for the llama-node package to be able to use the ggml v3 models? Actually the new ggml models that come out are all using this format

hlhr202 commented

will update in the following few days.

  • 🚀1
hlhr202 commented

resolved in v0.1.6

synw commented

I've just installed 0.1.6 to test it but it refuses to compile typescript code with this error:

$ tsc -p .
node_modules/@llama-node/llama-cpp/index.d.ts:137:31 - error TS2304: Cannot find name 'LoadModel'.

137   static load(params: Partial<LoadModel>, enableLogger: boolean): Promise<LLama>
synw commented

If I change in llama-cpp/index.d.ts line 137 static load(params: Partial<LoadModel> by static load(params: Partial<ModelLoad> it works. And I confirm that the compiled code can run ggml v3 models: nice 👍