Cloud Native LLM runtime proposal

Question

Cloud Native LLM runtime proposal

Opened this issue 3 months ago · 2 comments

For python developers, litellm maybe a good choice. That would be great we can do it in a runtime and any developers can use LLMs in own language way rather than packaging HTTP/gRPC calls by themselves.

Now we have many LLM APIs like OpenAI, Azure AI, Cohere, LLaMA, AWS bedrock, Kserve, OpenVINO and so on, migrate from one to the other still need many code changes.

I propose that we can do it in a Cloud Native LLM runtime, then developers can migrate from one to the other only by config.

Answer 1 · 2024-06-26T10:37:32.000Z

Related doc here: https://docs.google.com/document/d/1FQN_hGhTNeoTgV5Jj16ialzaSiAxC0ozxH1D9ngCVew/edit

Answer 2 · 2024-07-02T03:22:08.000Z

The whole proposal is here, and point out difference between gateway and runtime.