Use Ray to deploy your remote services.
Source: Scaling Python with Ray
Use Ray Serve for implementing a general-purpose microservice framework and how to use this framework for model serving. Ray Serve is implemented on top of Ray with Ray actors. Three kinds of actors are created to make up a Serve instance:
| Ray Actor | Description |
|---|---|
| Controller | The controller is responsible for creating, updating, and destroying other actors. All of the Serve API calls (e.g., creating or getting a deployment) use the controller for their execution. |
| Router | There is one router per node. Each router is a HTTP server that accepts incoming requests, forwards them to replicas, and responds after they are completed. |
| Worker Replica | Worker replicas execute the user-defined code in response to a request. |