This is a lab for efficientml.ai course.
Running large language models (LLMs) on the edge is of great importance. By embedding LLMs directly into real-world systems such as in-car entertainment systems or spaceship control interfaces, users can access instant responses and services without relying on a stable internet connection. Moreover, this approach alleviates the inconvenience of queuing delays often associated with cloud services. As such, running LLMs on the edge not only enhances user experience but also addresses privacy concerns, as sensitive data remains localized and reduces the risk of potential breaches.
However, despite their impressive capabilities, LLMs have traditionally been quite resource-intensive. They require considerable computational power and memory resources, which makes it challenging to run these models on edge devices with limited capabilities.
In this lab, you will learn the following:
- How to deploy an LLaMA2-7B-chat with TinyChatEngine on your computer.
- Implement different optimization techniques (loop unrolling, multithreading, and SIMD programming) for the linear kernel.
- Observe the end-to-end latency improvement achieved by each technique.
This tutorial is based on TinyChatEngine, a powerful neural network library specifically designed for the efficient deployment of quantized large language models (LLMs) on edge devices.
Please check this document and follow the instructions which will walk you through the tutorial: https://docs.google.com/document/d/13IaTfPKjp0KiSBEhPdX9IxgXMIAZfiFjor37OWQJhMM/edit?usp=sharing
- Report: Please write a report (form) that includes your code and the performance improvement for each starter code.
- Code: Use
git diffto generate a patch for your implementation. We will use this patch to test the correctness of your code. Please name your patch as{studentID}-{ISA}.patchwhere {ISA} should be one of x86 and ARM, depending on your computer.
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
Test on the x86 machine.
cd transformers
$./evaluate.sh reference
-------- Sanity check of reference implementation: Passed! --------
Section, Total time(ms), Average time(ms), Count, GOPs
reference, 910.317017, 91.031006, 10, 2.879700
$./evaluate.sh loop_unrolling
-------- Sanity check of loop_unrolling implementation: Passed! --------
Section, Total time(ms), Average time(ms), Count, GOPs
loop_unrolling, 902.317017, 90.231003, 10, 2.905232
$./evaluate.sh multithreading
-------- Sanity check of multithreading implementation: Passed! --------
Section, Total time(ms), Average time(ms), Count, GOPs
multithreading, 349.601013, 34.960003, 10, 7.498377
$./evaluate.sh simd_programming
-------- Sanity check of simd_programming implementation: Passed! --------
Section, Total time(ms), Average time(ms), Count, GOPs
simd_programming, 61.665005, 6.166000, 10, 42.510988
$./evaluate.sh multithreading_loop_unrolling
-------- Sanity check of multithreading_loop_unrolling implementation: Passed! --------
Section, Total time(ms), Average time(ms), Count, GOPs
multithreading_loop_unrolling, 347.059021, 34.705002, 10, 7.553298
$./evaluate.sh all_techniques
-------- Sanity check of all_techniques implementation: Passed! --------
Section, Total time(ms), Average time(ms), Count, GOPs
all_techniques, 33.264000, 3.326000, 10, 78.807117
play with chats bot:
./chat