Quantized LLMs on Hailo-8L

[bkawakami]

Hello Hailo Team,

I recently acquired the Raspberry Pi AI Kit, which includes the Hailo-8L AI Accelerator. I am exploring the potential of running quantized large language models (LLMs) on this setup. Could you provide any guidance or examples of successfully implementing quantized LLMs with the Hailo-8L on a Raspberry Pi? Your assistance and any documentation on this would be greatly appreciated.

Thank you!

[giladnah]

Hi, currently LLMs are not supported on the H8L. Performance would be significantly hindered by memory access limitations over PCIe.