sgl-project/sglang

Development Roadmap (Deprecated)

Ying1123 opened this issue · 17 comments

Ying1123 commented

Function Calling

High-level Pythonic Interface

Inference Optimizations

  • Speculative decoding for local models
  • Speculative execution for OpenAI Chat API

Structured Decoding

Compiler

  • Support tracing and compiling sgl.fork
  • Support sending a full serialized SGL program to the server

LoRA Support

  • Port multi-LoRA batching and unified memory from S-LoRA

Model Coverage

Device Coverage

  • AMD support. Investigate AMD support in Trion and FlashInfer.
  • CPU support. This is better done by adding a llama.cpp backend.
AriMKatz commented

Are there still plans for a high level pythonic interface? #39 (comment)

Ying1123 commented

Are there still plans for a high level pythonic interface? #39 (comment)

Hi @AriMKatz, thanks for the reference. This is very important, I just added it.

  • ❤️1
nivibilla commented

For the vision models support, is it possible to align with the openai gpt4v API?
https://platform.openai.com/docs/guides/vision

  • 👍1
aliencaocao commented

Are there plans for loading models in 8bit or 4bit?

Ying1123 commented

For the vision models support, is it possible to align with the openai gpt4v API? https://platform.openai.com/docs/guides/vision

@nivibilla Yes, it is already aligned with the openai gpt4v API, see here.
You can also find a runnable example of serving it with Sky Serve here.

  • 👍2
  • ❤️2
Ying1123 commented

Are there plans for loading models in 8bit or 4bit?

@aliencaocao Thanks for the question! The AWQ and GPTQ are already supported. But we do not support an automatic dtype translation yet. You are welcome to submit a PR for that.

aliencaocao commented

Are there plans for loading models in 8bit or 4bit?

@aliencaocao Thanks for the question! The AWQ and GPTQ are already supported. But we do not support an automatic dtype translation yet. You are welcome to submit a PR for that.

I'm looking to load llava 1.6 in 8bit, but it does not seem that llava series has AWQ or GPTQ quants, or did I miss out anything here?

EDIT: I saw 1.5 has but not 1.6 yet. Perhaps its just too new and no one did a calibration yet.

qeternity commented

Hi all - is anyone working on the S-LoRA integration currently? I see the branch, but it looks a few months old.

Would love to see this, happy to pick up from existing work or start fresh.

Ying1123 commented

Hi all - is anyone working on the S-LoRA integration currently? I see the branch, but it looks a few months old.

Would love to see this, happy to pick up from existing work or start fresh.

Hi @qeternity, I was working on it but have been blocked by other affairs. You are welcome to contribute, either continue on the branch or start fresh! I'll be happy to review and collaborate.

Bit0r commented

Tools support is very important, which is necessary for many use cases.

  • 👍5
omri-sap commented

Is TinyLlama supported? TinyLlama/TinyLlama-1.1B-Chat-v1.0
generation seems a bit slow...

  • 👀1
wille-42 commented

I see llama.cpp integration is on the roadmap. When will this feature be delivered? It would be very nice to have it , since it will support running local LLMs, such as llama models, on Mac computers and experiment them with the powerful and expressive SGLang.

Gintasz commented

I'd request to include support for Phi-3-mini

  • 👍6
binarycrayon commented

Hi all - is anyone working on the S-LoRA integration currently? I see the branch, but it looks a few months old.
Would love to see this, happy to pick up from existing work or start fresh.

Hi @qeternity, I was working on it but have been blocked by other affairs. You are welcome to contribute, either continue on the branch or start fresh! I'll be happy to review and collaborate.

Hi which branch is it? looks like better start fresh

zhyncs commented

I can help by getting rid of the vLLM in the dependencies.

  • 👍1
  • ❤️1
Ying1123 commented

Moved to #634

CedricHwong commented

Is the “tools” ok now？