feat: provide instructions on how community members can wrap models for this project
Opened this issue · 1 comments
PaulConyngham commented
as more and more models come out, would be great to see instructions on how we can go about wrapping the model to work with this framework and adding them back
kemingy commented
The main LLM inference code is in https://github.com/tensorchord/modelz-llm/blob/main/src/modelz_llm/model.py. To add a new model, you need to check https://github.com/tensorchord/llmspec/blob/main/llmspec/model_info.py and add the corresponding docker image in this repo.