LLMServe/DistServe

Issues

• How to calculate the memory read and write for attention calculation in decoding phase?

  #49 opened 5 months ago by azamikram
  1

• Error in building SwiftTransformers with torch 2.2.4

  #44 opened 7 months ago by gursimar
  1

• 求助，运行一直异常退出，llama-7b-hf模型，A100, worker 一直出现异常退出

  #29 opened 8 months ago by dingzhiqiang
  2

• Error in building SwiftTransformers (error: more than one conversion function from "half" to a built-in type applies)

  #46 opened 6 months ago by lylcyl
  3

• Decode Wrong Token

  #16 opened 10 months ago by sitabulaixizawaluduo
  8

• Generating max_num_tokens.csv for Different Hardware Environments

  #30 opened 8 months ago by village-way
  1

• offline/online serving is stuck at fetching files in paraworkers

  #48 opened 6 months ago by hyuenmin-choi
  0

• Can distserve change block_size ?

  #47 opened 6 months ago by WANGHanshuo1220
  0

• How to use tensorrt-llm as the inference backend for DistServe

  #45 opened 7 months ago by GGBond8488
  0

• 底层跨group的kv cache传输用的是什么库呢？

  #22 opened 7 months ago by CSEEduanyu
  11

• 模型推理结果混乱，怎么解决。

  #40 opened 8 months ago by liweiqing1997
  2

• Adding a new model?

  #43 opened 7 months ago by gursimar
  0

• not distserve.simulator.utils

  #42 opened 8 months ago by TZHelloWorld
  1

• decoder.embed_tokens.weight.pt not found

  #10 opened 10 months ago by llx-08
  12

• Why appear 0 unaccepted, 0 waiting, 0 processing？

  #26 opened 9 months ago by LeSoleilGo
  5

• What does pp_cross mean in the simulator output?

  #41 opened 8 months ago by xshqhua
  0

• 编译SwiftTransformer失败

  #37 opened 8 months ago by FredHuang99
  5

• fail to run examples/offline.py , unable to download the model to reproduce

  #35 opened 8 months ago by William12github
  13

• fail to run examples/offline.py, load model weight error.

  #38 opened 8 months ago by ddqspace-xyz
  3

• 分离部署多个prefill实例与多个decode实例支持问题

  #27 opened 8 months ago by Lin-Qingyang-Alec
  1

• Question on original DistServe paper - communication overhead

  #33 opened 8 months ago by lxldavid91
  2

• Error: Peer-to-peer access is unsupported on this platform

  #39 opened 8 months ago by Youhe-Jiang
  2

• [Profile] How to use nsight to profile CUDA execution information

  #36 opened 8 months ago by vincentccc
  2

• Support autoscaled prefill/decode servers

  #28 opened 8 months ago by liurupeng
  1

• DistServe是否支持异构推理？

  #34 opened 8 months ago by RobertLou
  4

• Cmake build fail

  #31 opened 8 months ago by hyuenmin-choi
  0

• An Trick of Enhancing LLM Serving Throughput with KV Cache Pipeline Strategy

  #3 opened 8 months ago by chenzhuofu
  5

• codellama34b ttft延迟问题

  #19 opened 9 months ago by sitabulaixizawaluduo
  7

• Swift transformers cmak build 一直循序

  #25 opened 9 months ago by lcvcl
  2

• Great work!

  #20 opened 9 months ago by irasin
  2

• Model not loaded error

  #24 opened 9 months ago by melissadu-db
  0

• load weight error but no other information when llama-2-7b-hf

  #23 opened 9 months ago by wangguanggg
  0

• SwitfTransformer compilation fails with ambiguous conversion error at PyTorch 24.05 container.

  #21 opened 9 months ago by piotrm-nvidia
  2

• Offline.py LLMEngine.__init__() missing 1 required positional argument: 'simulator_config'

  #15 opened 10 months ago by fivebamboo694
  5

• 一些关于TTFT的问题

  #18 opened 9 months ago by YLSnowy
  6

• 安装环境为什么要conda和pip混着用呢 不能全用pip吗

  #17 opened 9 months ago by CSEEduanyu
  0

• How difficult will adding Llama 3 support be?

  #12 opened 10 months ago by kalradivyanshu
  8

• How to profile

  #13 opened 10 months ago by YLSnowy
  7

• missing SimulatorConfig.profiler_data_path when runing offline.py

  #7 opened 10 months ago by KylinC
  3

• offline.py need simulator_config

  #5 opened 10 months ago by YLSnowy
  1

• cmake error when running "cmake -B build" in swiftTransformer

  #6 opened 10 months ago by KylinC
  1

• hotcrp

  #9 opened 10 months ago by YLSnowy
  1