/llm_with_cpp

LLM inference with cpp.

Primary LanguageC++MIT LicenseMIT

llm_with_cpp

LLM inference with cpp.

Only for fun.

Referenced from llama2.c

Platform

  • Ubuntu 22.04.3
  • GCC 17

Usage

  1. Clone this project.
git clone https://github.com/SuperJokerayo/llm_with_cpp.git
  1. Install sentencepiece tokenizer.
sudo apt-get install cmake build-essential pkg-config libgoogle-perftools-dev

git clone https://github.com/google/sentencepiece.git ./third_party/sentencepiece/
  1. Download LLM checkpoint from llama2.c repo. For example:
# download OG model which has 15M parameters
wget -P ./checkpoints/  https://huggingface.co/karpathy/tinyllamas/resolve/main/stories15M.bin
  1. Compile this project with cmake tool and run.
mkdir build
cd build
cmake
make -j $nproc

then, the executable file is put in ./bin

and you can run with:

./bin/run

  1. The config parameters are written in config.ini, and custom config is supported.

  2. You can also run the shell script if dependencies are installed manually.

bash ./run.sh

License

Have a look at the LICENSE file for details.