pytorch/serve

Standardize PyTorch 2.x features config in model-config.yaml

agunapal opened this issue ยท 1 comments

๐Ÿš€ The feature

Define the following structure for a PyTorch 2.x feature config in model-config.yaml

pt2:
  <API name>:
    enable: True
    option1:  value1
    optionN:  valueN

Example:
For torch.compile, we would specify the following options

pt2:
  compile:
    enable: True
    backend: "inductor"
    mode: "max-autotune"

Motivation, pitch

When torch.compile was integrated with TorchServe, there was no torch.export and torch.ao
We had only the following config in TorchServe for torch.compile

pt2: "inductor"

This was later modified to support other torch.compile options

pt2 : {backend: inductor, mode: reduce-overhead}

With torch.export , torch.ao being included in TorchServe, there is a need to clearly define the interface to make the user experience intuitive for customers. There is also a need to make the design consistent for any new/future PyTorch 2.x feature.

Alternatives

No response

Additional context

No response

Done