Tutorial for TinyChat: Optimizing LLM on Edge Devices
This is a lab for efficientml.ai course.
Running large language models (LLMs) on the edge is of great importance. By embedding LLMs directly into real-world systems such as in-car entertainment systems or spaceship control interfaces, users can access instant responses and services without relying on a stable internet connection. Moreover, this approach alleviates the inconvenience of queuing delays often associated with cloud services. As such, running LLMs on the edge not only enhances user experience but also addresses privacy concerns, as sensitive data remains localized and reduces the risk of potential breaches.
However, despite their impressive capabilities, LLMs have traditionally been quite resource-intensive. They require considerable computational power and memory resources, which makes it challenging to run these models on edge devices with limited capabilities.
In this lab, you will learn the following:
- How to deploy an LLaMA2-7B-chat with TinyChatEngine on your computer.
- Implement different optimization techniques (loop unrolling, multithreading, and SIMD programming) for the linear kernel.
- Observe the end-to-end latency improvement achieved by each technique.
TinyChatEngine
This tutorial is based on TinyChatEngine, a powerful neural network library specifically designed for the efficient deployment of quantized large language models (LLMs) on edge devices.
Tutorial document
Please check this document and follow the instructions which will walk you through the tutorial: https://docs.google.com/document/d/13IaTfPKjp0KiSBEhPdX9IxgXMIAZfiFjor37OWQJhMM/edit?usp=sharing
Submission
- Report: Please write a report (form) that includes your code and the performance improvement for each starter code.
- Code: Use
git diffto generate a patch for your implementation. We will use this patch to test the correctness of your code. Please name your patch as{studentID}-{ISA}.patchwhere {ISA} should be one of x86 and ARM, depending on your computer.
Related Projects
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration