yubuyuabc/llm_template

Imitate OpenAI with Local Models

Imitater

Usage

Create a .env file in the root directory:

.
├── src
└── .env

# imitater
AGENT_TYPE=react

CHAT_MODEL_PATH=Qwen/Qwen-14B-Chat
CHAT_MODEL_DEVICE=0
CHAT_TEMPLATE_PATH=templates/qwen.jinja
GENERATION_CONFIG_PATH=generation_config/qwen

EMBED_MODEL_PATH=BAAI/bge-small-zh-v1.5
EMBED_MODEL_DEVICE=1
EMBED_BATCH_SIZE=16

SERVICE_PORT=8010

# tests
OPENAI_BASE_URL=http://192.168.0.1:8010/v1
OPENAI_API_KEY=0

Note

Chat template is required for the chat models.

Launch Server

python src/launch.py

Test Server

python tests/test_openai.py