Simply python server to host Large Language Models on a restful API.
- Configure a prompt template
- OpenAI compatible rest api
- Config APIs provided to load and unload models
make setup
make start
curl --request POST \
--url http://localhost:8000/config/v1/load-model \
--header 'content-type: application/json' \
--data '{
"path": "./mistral-7b-openorca.Q5_K_M.gguf",
"options": {
"prompt_tmpl": "chatml"
}
}'
> Make sure you have downloaded a quantized ggufv2 model. Example: https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF
curl --request POST \
--url http://localhost:8000/config/v1/unload-model
- Open http://localhost:8000/docs to open swagger ui
- Open http://localhost:8000/openapi.json to access the raw openapi spec