555cider/Rust-Candle-LLM

Studying LLMs with Rust

What this is

• These are my records of testing LLMs locally Using Rust with Candle.
  I just modified the sample code to fit my needs, and currently Mistral-7B-Instruct-v0.2-GGUF was tested.

How to run

1. Download the model file.
   : mistral-7b-instruct-v0.2.Q4_K_M.gguf (TheBloke/Mistral-7B-Instruct-v0.2-GGUF)

2. Download the tokenizer file.
   : tokenizer.json (mistralai/Mistral-7B-Instruct-v0.2)

3. Place them somewhere in root project.
   : /model/Mistral-7B-Instruct-v0.2/

4. Run.

cargo run --bin mistral --release -- -p "chat"

About the argument of "-p"

1. interactive: multiple prompts in an interactive way
2. chat: multiple prompts in an interactive way where history is preserved
3. else: a single prompt