Simplified llama2.c.dll
JohnClaw opened this issue · 4 comments
Llama.dll that can be downloaded from llama.cpp repo is suitable mostly for programming languages that have abilities to work with rather difficult (for novice coder) concepts like pointers, structures etc. The same thing applies to this repo. So the question is: does anybody know any simplified implementation of llama2.c.dll or llama.cpp.dll? For example, it would be wonderful if it will be possible to work with this dll in such way:
CONSOLE
Dim LLama_DLL as Integer
Load "llama.dll" as LLama_DLL
Call function Load_Local_Gguf ("Llama-2-7b-q8.0.gguf") from LLama_DLL
Call function Set_Context_Size_For_Loaded_Gguf (4096) from LLama_DLL
Dim Answer$ As String
Dim Question$ As String
Label #Begin
Input Question$
Answer$ = Call function SentPromptToLoadedGguf (Question$) from LLama_DLL
Print Answer$
Goto #Begin
I have done this to call from PHP for a web app but I haven't shared it on github yet. It's not a big modification, especially since the refactor on this repo. It consists of passing a pointer to the main function with what would be arguments for a CLI, putting the pointer in transformer.data, replacing the fread with memcpy in the read_configuration, and I think that's it. I can upload if there's nothing out there right now.
https://github.com/jameswdelancey/llama3.c/blob/master/runqdll.c#L1054
If you look at the bottom, you can see how quick and dirty replicated the CLI args with the build_main call. This works with the quantized version of LLaMA 3B base or instruct when exported by export.py as a version-2 (quantized ak42 magic)
It doesn't work with LLaMA2 or tinyshakespear or tinystories though, because the overhaul to Llama3 has taken place, but can be backported.
It doesn't work with LLaMA2 or tinyshakespear or tinystories though, because the overhaul to Llama3 has taken place, but can be backported.
I found dll that can chat with llm-s using one simple function without pointers and structures: https://github.com/tinyBigGAMES/Dllama
It's ideal for novice coders like me and can be used from any programming language. For example, here's AutoIt's (https://www.autoitscript.com/site/) implementation:
Local $answer = DllCall($hDLL, "str:cdecl", "Dllama_Simple_Inference", "str", "config.json", "str", "llama3", "str", $question, "uint", 1024)
Could you add such simple function to your dll, please?