vllm-project/production-stack

chore: Make router a python package

gaocegege opened this issue · 4 comments

gaocegege commented

Making the router a Python package offers several benefits:

  • The package can be installed with pip, simplifying dependency management and deployment.
  • Tests can be run more easily, and the package can be integrated into CI/CD pipelines for automated testing.
  • The package can be versioned, allowing for controlled updates and backward compatibility.

Besides, we should include some basic end-to-end (e2e) tests as part of the continuous integration (CI) pipeline.

gaocegege commented

Hey, I saw the roadmap mentions a "more performant router using non-Python languages." If we plan to tackle that soon, maybe we could explore rewriting the router in another language for a performance boost and ignore this. Just a thought.

ApostaC commented

Hey @gaocegege , making it a python package sounds great to me!

We are right now working on end-to-end tests and CI pipeline. And it should be available soon. Thanks for pointing that out!

gaocegege commented

Thanks for the prompt response.

Should we turn it into a Python package now, or wait until the CI setup is ready? It may impact the CI implementation.

ApostaC commented

It would be great if we could turn it into a python package now. (otherwise we need to adjust the CI again IIUC?)

  • 👍1