This repository has been archived and is no longer maintained. We have created ray.serve.llm and ray.data.llm APIs to simplify deployment of LLMs on top of Ray. These APIs are now directly integrated into Ray and managed by the Ray team. The history of this repository is moved to archived-master branch only for historical context.