aws/sagemaker-huggingface-inference-toolkit

Support for huggingface/peft

dbpprt opened this issue · 3 comments

dbpprt commented

After looking through the code it currently seems to not be possible to lead adapter models produced by peft. It would be a great addition to HF DLCs.

philschmid commented

Hello @dennisbappert, it should be possible to use PEFT models through a custom inference.py module. See for an example on how to create a inference.py https://www.philschmid.de/custom-inference-huggingface-sagemaker

Or have you tried this already?

dbpprt commented

Hi @philschmid, no I haven't tried it yet but I do also think that it is possible. Integrated support however would make it much easier to use out of the box without custom inference scripts.

Likely just a feature-request for the future 👍

philschmid commented

Hey @dennisbappert,

Thats a great feature request, indeed! but PEFT is still pretty new lets revisit this in a few months when we have a better understanding how it is used.