ucbrise/clipper

[Question] Scaling the Query Frontend

Opened this issue · 3 comments

rp4fx12 commented

Hi, this question is sort of a two parter. 1. Do you guys have a slack or something to post questions? 2. The other question is regarding scaling out the query-frontend container. Regarding this portion of the docs: http://clipper.ai/tutorials/container_managers/#known-limitation-query-frontend-scaleout, would a rebalancing method mean that each query frontend would need to have at least one model container behind it? I'm guessing if I spin up a new query-frontend, I would need to launch it in such a way that it either terminates the connection to one of the pre-existing model containers or it spins up a fresh version of a model container. I'm also guessing I would need something to orchestrate that, like maybe some flask server or something else that sits as a separate pod to the query-frontend (or bake it into the mgmt frontend).

rkooo567 commented

@simon-mo @RehanSD Could you guys answer this question?

withsmilo commented
  1. We have a Slack workspace, but I don't have no idea about how to invite you.
  2. We usually leave enough query-frontends already created and scale out the model containers as needed to avoid the problem.
asauray commented

I don't really understand your answer @withsmilo

(1) We agree that the query frontend is stateful right ? So do de we have a 1:1 relationship between a model and a query frontend ?

(2) If you add more instances of the query frontend, and at the same time, model instances connect only to one instance of the query frontend. How do you ensure that each query frontend has at least one connection to an instance of a model in that situation ?