yoshoku/llama_cpp.rb

Expose more llama.cpp options

technicalpickles opened this issue · 2 comments

I've been testing https://huggingface.co/TheBloke/wizardLM-7B-GGML , and the example code is:

./main -t 10 -ngl 32 -m wizardLM-7B.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"

A few of these options (and others) would be useful to expose:

  • temp
  • repeat_penalty
  • top-k
  • top-n

@technicalpickles
Thank you for your interesting proposal. In version 0.3.2, I made the sampling options the arguments of the generate module function in LLaMACpp.
https://yoshoku.github.io/llama_cpp.rb/doc/LLaMACpp.html#generate-class_method
https://github.com/yoshoku/llama_cpp.rb/blob/main/CHANGELOG.md#032---2023-07-08

Thank you!