aws/sagemaker-huggingface-inference-toolkit

Support for return_all_scores in pipeline

CHENHUI-XU opened this issue · 2 comments

CHENHUI-XU commented

For text_classification pipeline, the parameter return_all_scores=True is needed to get all scores of all labels. Could this be integrated into the toolkit? (or potentially for newer version, the parameters is top_k = n).

Allowing to pass additional parameters needed for the pipeline would be great.

pipe = pipeline("text-classification", model='path/to/mode', tokenizer='path/to/tokenizer', return_all_scores=True)
philschmid commented

Hello @CHENHUI-XU,

We are trying to keep the zero-code deployments as flexible and as open as possible that's why the return_all_scores is not set.
If you want to enable it for you endpoints you can easily create a inference.py to initialize the pipeline as you want. Here you can find an example https://github.com/huggingface/notebooks/blob/main/sagemaker/17_custom_inference_script/sagemaker-notebook.ipynb

  • ❤️1
philschmid commented

I just came across this forum post. You can also add the return_scores during the inference as well. https://discuss.huggingface.co/t/emotion-model-additional-inference-parameter-not-processed-in-sagemaker/18932/3

I hope this works for you.

  • ❤️1