Open AI API Maturity
Jack-Khuu opened this issue · 4 comments
🚀 The feature, motivation and pitch
The OpenAI API support in torchchat is actively in development and will be one of the main entry point for interacting with torchchat
https://github.com/pytorch/torchchat?tab=readme-ov-file#server
Alternatives
No response
Additional context
No response
RFC (Optional)
No response
PR 995 addresses some initial concerns. Responses should now be formatted in JSON using the API dataclasses. Here are our functional gaps so far:
- system_fingerprint/seed implementation is incomplete. We need to figure out a good way to create and consume fingerprints from seeds. For now, the client cannot control reproducibility of the responses.
- model cannot be selected by the client. The model is selected when starting the server and cannot be changed without exiting the process. The server does not check to ensure the request/response models match
- Agentic behavior/toolcall functionality is unimplemented. We will focus on getting chat right first, then create entrypoints for tool usecases.
Behavior Issues
- Model responses may not respond in the expected manner. See #989
Hello there! I'm curious if supporting OpenAI Vision API is also something you're interested in since it looks like the team does plan to support vision language models from the discussion in #988
If you do, happy to help!
Hey Roger, thanks for reaching out! We do plan on adding LLaVA support in the coming weeks, and we'd appreciate your help on the API/server components.
I'm still working on bringing the initial version of the API and server up to spec, but I'll keep this issue updated with my progress.