tensorchord/modelz-llm

feat: provide instructions on how community members can wrap models for this project

Opened this issue · 1 comments

as more and more models come out, would be great to see instructions on how we can go about wrapping the model to work with this framework and adding them back

The main LLM inference code is in https://github.com/tensorchord/modelz-llm/blob/main/src/modelz_llm/model.py. To add a new model, you need to check https://github.com/tensorchord/llmspec/blob/main/llmspec/model_info.py and add the corresponding docker image in this repo.