enable users to override the default api endpoint path

[aniketmaurya]

customize the API endpoint path by providing LitServer(..., endpoint_path="/api/CUSTOM_PATH").

[eugsokolov]

@aniketmaurya @williamFalcon hi

i am curious what thoughts are around standardization. it would be nice to change the default /predict or to add new endpoints (doesnt seem to be possible now but I could be wrong). having something like /classify or /cluster or the like might be interesting for some users. having flexibility to change the default (via decorator) might be nice

[aniketmaurya]

hi @eugsokolov, may I know more about your use case, please? What can /classify solve that /predict is not able to? Some context will be helpful.

[eugsokolov]

hi @eugsokolov, may I know more about your use case, please? What can /classify solve that /predict is not able to? Some context will be helpful.

/predict can do everything /classify can do, but sometimes a model may not want to provide a prediction but instead do something else. there might be a model that can predict and classify (and do something else) as well and it might be nice to have multiple endpoints all under one LitAPI

to clarify - i think this would be a nice to have and was curious what the rationale in the PR around "this is breaking our standardization" were

[williamFalcon]

i’m super interested in understanding what problem this solves so we can help. just need more details 😉

the word predict is being taken too literally here. predict just means “do what you want now with your model”. if you want to classify, segment, summarize, etc… just do it.

it’s just a word.

so, i’m still having a hard time understanding what the problem is that you are trying to solve?

“with predict i cannot do XYZ and thus we need ABC to enable the behavior i am looking for”.

without a clear motivation, this feels purely aesthetic…

are you asking that the endpoint /predict be customizable?

are you asking for the literal “predict” method to be named something else? if this is the case, why does the name matter?

are you asking for a server to provide multiple functionalities? ie: predict AND classify? and if so, is that really a best practice? do you have a concrete example where this is used in a real world application (not just “nice” in theory?) and if that’s further true, can’t a simple if statement handle that logic for you in the predict step?

we aim to provide a standard that allows servers to interoperate and be legible by anyone. if every litserve implementation uses a different name it loses the ability to be standardized… ie: right now if you want to know what a server is doing, then simply look at the predict method. if we remove this standard, it will be hard to pinpoint exactly what a server is doing to someone not familiar with a code base.

not saying no to the request, just asking to understand the underlying problem to understand what a reasonable solution might be.

[eugsokolov]

yes this is aesthetic - if i am exposing an endpoint to the public i may not want it to be called /predict if it isnt providing any sort of prediction. i have a model that is classifying geospatial data and not doing any predictions but i am forced to use the /predict endpoint (openai doesnt even have a predict endpoint, just completion and embedding). in similar vein, i name my variables to something that resembles the data they are storing.

i dont feel very strongly here and thought it would be nice to name the endpoint via a decorator instead of always needing to use the default name

as far as best practice to provide multiple endpoints, maybe i want to provide a debug or data endpoint for my users. if there isnt much flexibility here then ill create a fastapi app to provide the relevant endpoints for my users

[williamFalcon]

ok. then yes.

To summarize why i think this makes sense:

An endpoint could be made public to people who didn't create the endpoint and thus the path name should match the functionality to reduce confusion.

in this case it makes sense to enable something similar to what @aniketmaurya proposed in the beginning here

# default
LitServer(..., api_path="predict")

# overwrite it (for example to be called /classify)
LitServer(..., api_path="classify")

but the method name still remains "predict" in the server API.

wdyt @aniketmaurya @lantiga

[eugsokolov]

ok. then yes.

To summarize why i think this makes sense:

An endpoint could be made public to people who didn't create the endpoint and thus the path name should match the functionality to reduce confusion.

in this case it makes sense to enable something similar to what @aniketmaurya proposed in the beginning here

# default
LitServer(..., api_path="predict")

# overwrite it (for example to be called /classify)
LitServer(..., api_path="classify")

but the method name still remains "predict" in the server API.

wdyt @aniketmaurya @lantiga

awesome!

i'll also add that a partial requirement of "renaming" would be to add new endpoints (which of course would require a different name) and hence it may make sense to force the developer to name the endpoint. this seems like a paradigm shift from the current implementation. but i do believe that taking an example from fastapi would be useful here in allowing for developers the flexibility.

the default example can include /predict. what i am envisioning would look something like:

# server.py
import litserve as ls

# STEP 1: DEFINE YOUR MODEL API
class SimpleLitAPI(ls.LitAPI):
    def setup(self, device):
        # Setup the model so it can be called in `predict`
        self.model = lambda x: x**2

    def decode_request(self, request):
        # Convert the request payload to your model input.
        return request["input"]

    @ls.endpoint("/predict")
    def predict(self, x):
        # Run the model on the input and return the output.
        return self.model(x)

    @ls.endpoint("/classify")
    def classify(self, x):
        return self.model.classify(x)

    @ls.endpoint("/model-weights")
    def model_weights(self, x):
        return self.model.weights

    def encode_response(self, output):
        # Convert the model output to a response payload.
        return {"output": output}


# STEP 2: START THE SERVER
if __name__ == "__main__":
    api = SimpleLitAPI()
    server = ls.LitServer(api, accelerator="auto")
    server.run(port=8000)

[aniketmaurya]

hi @eugsokolov, thanks for all the feedback. For manually specifying endpoint, the following looks ideal to me.

# default
LitServer(..., api_path="predict")

# overwrite it (for example to be called /classify)
LitServer(..., api_path="classify")

@eugsokolov for multiple endpoints, I would suggest to use LitSpec (note that it is not 100% documented yet), we use it to provide OpenAI compatible API endpoints here.

[williamFalcon]

yeah not sure about multiple endpoints on the same server. because now you need a different encode and decode function for each as well.

i don’t like building “nice to haves” for “theoretical scenarios”. do you have a concrete, real-world example where you must have multiple methods?

i totally get it for regular web development… but i don’t see how that applies here and i’m not even sure we would even want to encourage such a thing. i can see that being fraught with problems like out of memory, collisions, etc….

again, i don’t want to jump to an answer here, we need to understand the problem yet. i understand in “theory” it would be “nice” to do X… but how realistic is this? products get overbloated with “nice” things that could happen in “theory” and we know 99.999% of the time it never comes up.

[lantiga]

Interestingly, we already have the concept of endpoint with their own path (and method) behind the scenes, which we currently use to expose /v1/chat/completion for the OpenAI spec.

The only question is, do we really need to expose it to users? Flexibility comes with cognitive overhead so we need to be super conservative. In the future, as practical needs arise, we can expose the mechanism for advanced users, but probably not before we have a real need.

For now @aniketmaurya 's solution seems to check all boxes so it's a +1 for me.