abhishekrana/llm-service

RESTful service with LLMs (Large Language Models) running locally

llm-service

RESTful service with LLMs (Large Language Models) running locally.

Local setup

Bootstrap

./bootstrap

Download LLaMA 2

cd llama
task download-llama2-code
task download-llama2-model

Create virtual environment and install dependencies

task setup-venv
source .venv/bin/activate
poetry install

Run RESTful service

task run

http://127.0.0.1:8000/docs

Resources

• https://github.com/ggerganov/llama.cpp