bentoml/BentoVLLM

Issues

• awq quantization is not fully optimized yet. The speed can be slower than non-quantized models.

  #57 opened 2 months ago by jackNhat
  0

• CUDA out of memory

  #42 opened 4 months ago by Chasapas
  1

• VLLM is stuck on Outlines 0.0.34 and this sample requires 0.0.37

  #28 opened 3 months ago by ProVega
  1

• Upgrading to VLLM 0.4.1 - TypeError

  #38 opened 3 months ago by ProVega
  4

• Does BentoVLLM support multi LoRA adapters functionality of vLLM?

  #36 opened 5 months ago by heartcored98
  1

• Using Bentoml v 1.2.10, cant connect to host network and fails to download the debian

  #30 opened 5 months ago by OriAlpha
  1

• No Swagger UI on localhost:3000

  #22 opened 5 months ago by FBR65
  6

• BentoVLLM Service Fails to Start on Linux Server Due to Pydantic Related Errors

  #14 opened 6 months ago by Juhong-Namgung
  2