Incorporate exllama?
uogbuji opened this issue · 1 comments
uogbuji commented
Any chance you could incorporate Ooba's new exllama support? It's more than just an upstream code update. Needs additional cloning of that repo, etc. https://github.com/oobabooga/text-generation-webui/blob/main/docs/ExLlama.md
Atinoda commented
I have integrated ExLlama and tested with wizard-mega-13B-GPTQ - it runs quickly indeed! Struggled to load older models though. Docker images are being built and pushed now - let me know how it works for you!