EricLBuehler/candle-vllm

Pipeline batching tracking issue

EricLBuehler opened this issue · 3 comments

This is the pipeline batching tracking issue.

@EricLBuehler are you working on this yet? would be interested in collaborating on it

@sigma-andex, thanks for offering - that sounds great! Perhaps you can open a PR to track development?

I will be probably making some major changes in the architecture of the ModulePipeline in the coming days, but once they are made I will let you know.

Closing this pending a rewrite of the scheduler and cache, see the scheduler branch. This task is encompassed by the scheduler, which manages the batching. A new tracking issue will be opened.