/vanilla-llama

Plain pytorch implementation of LLaMA

Primary LanguagePythonGNU General Public License v3.0GPL-3.0

vanilla-llama πŸ¦™

πŸ“’ vanilla-llama is a plain-pytorch implementation of LLaMA with minimal differences with respect to the original Facebook's implementation. You can run vanilla-llama on 1, 2, 4, 8 or 100 GPUs

Couldn't be more easy to use πŸ”₯

Comes with an inference server included πŸ”‹

from inference import LLaMAInference

llama = LLaMAInference(llama_path, "65B")
print(llama.generate(["My name is Federico"]))

Features πŸ†

  • Easy to use and fine-tune πŸ”₯
  • Uses πŸ€— accelerate to distribute the model on all available GPUs
  • Comes with batteries includedπŸ”‹
  • Nice one line loading and generation 😎

Examples πŸ€”

Stop generation on specific tokens (13 is the new-line token)

llama.generate(["Chat:\nHuman: Hi i am an human\nAI:"], stop_ids=[13])

Stop generation on specific texts

llama.generate(["Question: is the sky blue?\nAnswer:"], stop_words=["Question"])

Batch generation

llama.generate(["My name is Federico", "My name is Zuck"])

Inference server 🌐

Install server requirements

pip install -r server_requirements.txt

Run the server

python server.py --llama-path <CONVERTED-WEIGHTS-PATH> --model <MODEL>

Test it!

curl -X GET http://localhost:3000/generate -H "Content-Type: application/json" -d '{"prompt": "REST server are very useful becouse"}'

Installation βš™οΈ

Clone this repository

git clone https://github.com/galatolofederico/vanilla-llama.git
cd vanilla-llama

Install the requirements

python3 -m venv env
. ./env/bin/activate
pip install -r requirements.txt

Convert LLaMA weights 🏭

To convert LLaMA weights to a plain pytorch state-dict run

python convert.py --llama-path <ORIGINAL-LLAMA-WEIGHTS> --model <MODEL> --output-path <CONVERTED-WEIGHTS-PATH>

Run example ✨

Run the provided example

python example.py --llama-path <CONVERTED-WEIGHTS-PATH> --model <MODEL>

Many thanks to these amazing projects ❀️