Support of Multi-model Endpoint using HuggingFace inference toolkit

[Tarun02]

Hi,

Does SageMaker huggingface inference toolkit supports multi-model endpoint ?

[philschmid]

Yes, the HuggingFace Inference Toolkit uses similar mechanism than the other ones. You can find documentation here:

• https://docs.aws.amazon.com/sagemaker/latest/dg/create-multi-model-endpoint.html
• https://sagemaker.readthedocs.io/en/stable/api/inference/multi_data_model.html

[DevBey]

Hi @philschmi,

thanks for the resources !!
in this https://www.philschmid.de/sagemaker-huggingface-multi-container-endpoint you mention that multi-container endpoints are not possible on GPU ??

is this true for multi-modal endpoint also ??

[philschmid]

Yes, currently Multi-Model Endpoint also only works with CPU instances

[DevBey]

thanks @philschmid you saved me sometime

[leloykun]

Hello!

I'm interested in helping out with this. That said, what's the main blocker on implementing support for GPU MME?

Why wouldn't it suffice to just add the multi-models tag?

[ibicdev]

Second this question. It has been 1.5 years since the original question, yet is GPU still not supported?