/imitater

Imitate OpenAI with Local Models

Primary LanguagePythonApache License 2.0Apache-2.0

Imitater

GitHub Code License PyPI

A unified language model server built upon vllm and infinity.

Usage

Install

pip install -U imitater

Launch Server

python -m imitater.service.app -c config/example.yaml

Note

Chat template is required for the chat models.

Use export USE_MODELSCOPE_HUB=1 to download model from modelscope.

Test Server

python tests/test_openai.py -c config/example.yaml

Roadmap

  • Response choices.
  • Rerank model support.