English| 简体中文
hobot_llm is a Node of Large Language Model (LLM) integrated into the Horizon RDK platform, allowing users to experience LLM at the edge. Currently, there are two ways to experience it: direct text chatting at the terminal and subscribing to text messages, then publishing the results in text format.
| Robot Name | Manufacturer | Reference Link |
|---|---|---|
| RDK X3 (4GB RAM) | Multiple Suppliers | Click here |
Before experiencing it, the following basic requirements need to be met:
- Confirm that the Horizon RDK is the 4GB RAM version.
- The Horizon RDK has been burned with the Ubuntu 20.04 system image provided by Horizon.
- Install transformers, using the command
pip3 install transformers -i https://pypi.tuna.tsinghua.edu.cn/simple. - Update hobot-dnn, using the command
sudo apt update; sudo apt install -y hobot-dnn.
After starting the RDK X3, connect to the robot through SSH or VNC terminal, copy and run the following commands on the RDK system to complete the installation of the corresponding Nodes.
tros foxy:
sudo apt update
sudo apt install -y tros-hobot-llmtros humble:
sudo apt update
sudo apt install -y tros-humble-hobot-llmBefore running the program, download the model files and unzip them using the following commands:
tros foxy:
# Download model files
wget http://sunrise.horizon.cc/llm-model/llm_model.tar.gz
# Unzip
sudo tar -xf llm_model.tar.gz -C /opt/tros/${TROS_DISTRO}/lib/hobot_llm/tros humble:
# Download model files
wget http://sunrise.horizon.cc/llm-model/llm_model.tar.gz
# Unzip
source /opt/tros/humble/setup.bash ##export TROS_DISTRO=humble
sudo tar -xf llm_model.tar.gz -C /opt/tros/${TROS_DISTRO}/lib/hobot_llm/At the same time, change the reserved memory size of BPU to 1.7GB. Refer to TODO for the setting method.
After rebooting, adjust the CPU's highest frequency to 1.5GHz and set the scheduling mode to performance, using the following commands:
sudo bash -c 'echo 1 > /sys/devices/system/cpu/cpufreq/boost'
sudo bash -c 'echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor'Currently, two running programs are provided, hobot_llm_chat and hobot_llm. Among them, hobot_llm_chat provides a terminal interactive experience, allowing users to directly input text to experience the large model. The hobot_llm program subscribes to text messages of type std_msgs/msg/String, sends them to the large model for processing, and finally publishes the results in the form of std_msgs/msg/String. This program can be connected with other nodes, for example, to play the output text as speech.
tros foxy:
source /opt/tros/setup.bash
ros2 run hobot_llm hobot_llm_chattros humble:
source /opt/tros/humble/setup.bash
ros2 run hobot_llm hobot_llm_chatAfter the program is launched, you can chat directly with the robot in the current terminal.
-
Start hobot_llm
tros foxy:
source /opt/tros/setup.bash ros2 run hobot_llm hobot_llmtros humble:
source /opt/tros/humble/setup.bash ros2 run hobot_llm hobot_llm -
Open a new terminal to subscribe to the output result topic
tros foxy:
source /opt/tros/setup.bash ros2 topic echo /text_result
tros humble:
source /opt/tros/humble/setup.bash ros2 topic echo /text_result
-
Open a new terminal to publish a message
tros foxy:
source /opt/tros/setup.bash ros2 topic pub --once /text_query std_msgs/msg/String "{data: ""What is the capital of China?""}"
tros humble:
source /opt/tros/humble/setup.bash ros2 topic pub --once /text_query std_msgs/msg/String "{data: ""What is the capital of China?""}"
The interface of hobot_llm program is as follows:
| Name | Message Type | Description |
|---|---|---|
| /text_query | std_msgs/msg/String | Default subscription topic |
| /text_result | std_msgs/msg/String | Default result publication topic |
| Parameter | Type | Description | Required | Supported Configuration | Default Value |
|---|---|---|---|---|---|
| topic_sub | std::string | Subscription topic name | No | Configured according to the actual deployment environment | /text_query |
| topic_pub | std::string | Publish result topic name | No | Configured according to the actual deployment environment | /text_result |
-
Model loading failure
Confirm that the development board has 4GB of memory, and modify the reserved memory size of BPU to 1.7GB.
-
Output result is garbled
Confirm that you have used the command
sudo apt update; sudo apt install hobot-dnnto update hobot-dnn. -
How to manually modify the BPU reserved memory to 1.7GB?
Refer to the modification method in Setting ion_cam size in device tree, and change the values from 0x2a000000 to 0x6a400000 in the alloc-ranges and size attributes.