How to enable Batch inference on AWS deployed Serverless model from Hub?

[jmparejaz]

I am using the serverless inference from Sagemaker with Huggingface Model from the hub
according this example :
https://github.com/huggingface/notebooks/blob/main/sagemaker/19_serverless_inference/sagemaker-notebook.ipynb

using the
#image uri
image_container=get_huggingface_llm_image_uri("huggingface",version="0.9.3")

I was expecting the resulting pipeline to execute as the Pipeline class from transformers for this task (text generation)
however, the input does not work with list.

Is there any approach to do batch inference on Sagemaker SDK?

[philschmid]

Hello @jmparejaz,

The input schema for the LLM container should be the same with {"inputs":"text", "parameters": {}} what issue are you seeing. The only difference here is that the LLM container has additional/different parameter, see here: https://huggingface.co/blog/sagemaker-huggingface-llm#4-run-inference-and-chat-with-our-model