llm-rk3588

This repository is intend to provide a complete guide on how to run LLMs on rk3588 SBC, specifically Orange Pi 5 Plus. But other rk3588 based board should be able to run without problem.

Link

BLOG | MLC-LLM | Apache TVM

Environment setup

Download and install the Ubuntu 22.04 for your board from here

Download and install libmali-g610.so

cd /usr/lib && sudo wget https://github.com/JeffyCN/mirrors/raw/libmali/lib/aarch64-linux-gnu/libmali-valhall-g610-g6p0-x11-wayland-gbm.so

Check if file mali_csffw.bin exist under path /lib/firmware by running ls /lib/firmware

If not, then download it with command:

cd /lib/firmware && sudo wget https://github.com/JeffyCN/mirrors/raw/libmali/firmware/g610/mali_csffw.bin

Download OpenCL ICD loader and manually add libmali to ICD

sudo apt install mesa-opencl-icd
sudo mkdir -p /etc/OpenCL/vendors
echo "/usr/lib/libmali-valhall-g610-g6p0-x11-wayland-gbm.so" | sudo tee /etc/OpenCL/vendors/mali.icd

Download and install libOpenCL
```
sudo apt install ocl-icd-opencl-dev
```

Download and install dependencies for Mali OpenCL

sudo apt install libxcb-dri2-0 libxcb-dri3-0 libwayland-client0 libwayland-server0 libx11-xcb1 clinfo

Run clinfo to check if OpenCL runs successfully

$ clinfo
arm_release_ver: g13p0-01eac0, rk_so_ver: 3
Number of platforms                               2
    Platform Name                                   ARM Platform
    Platform Vendor                                 ARM
    Platform Version                                OpenCL 2.1 v1.g6p0-01eac0.2819f9d4dbe0b5a2f89c835d8484f9cd
    Platform Profile                                FULL_PROFILE
    ...

Feel free to check this article for other platform

MLC-LLM setup

Use prebuilt (recommended)

Clone mlc-llm repository

sudo apt install git git-lfs
git clone --recursive https://github.com/mlc-ai/mlc-llm.git && cd mlc-llm
mkdir -p dist/prebuilt && cd dist/prebuilt
git clone https://github.com/mlc-ai/binary-mlc-llm-libs.git lib
git clone https://huggingface.co/mlc-ai/mlc-chat-RedPajama-INCITE-Chat-3B-v1-q4f16_1
cd ../../..

Build mlc-chat-cli from source

cd mlc-llm/
mkdir -p build && cd build
python3 ../cmake/gen_cmake_config.py
cmake .. && cmake --build . --parallel $(nproc) && cd ..

Verify installation

# expect to see `mlc_chat_cli`, `libmlc_llm.so` and `libtvm_runtime.so`
ls -l ./build/

# expect to see help message
./build/mlc_chat_cli --help

Run LLMs

./build/mlc_chat_cli --local-id RedPajama-INCITE-Chat-3B-v1-q4f16_1 –device mali

Compile your own LLMs

Install mlc-llm package

git clone --recursive https://github.com/mlc-ai/mlc-llm.git && cd mlc-llm
pip install .