AX-LLM

Platform	Build Status
AX650

简介

AX-LLM 由 爱芯元智 主导开发。该项目用于探索业界常用 LLM(Large Language Model) 在已有芯片平台上落地的可行性和相关能力边界，方便社区开发者进行快速评估和二次开发自己的 LLM 应用。

LLM Build

在线文档：

Pulsar2 LLM Build

已支持芯片

AX650A/AX650N
- SDK ≥ v1.45.0_P31
AX630C
- SDK ≥ v2.0.0_P7

已支持模型

LLM

Qwen1.5-0.5B/1.8B/4B
Qwen2-0.5B/1.5B
ChatGLM3-6B
MiniCPM-1B/2B
TinyLLaMa-1.1B
Llama2-7B
Llama3-8B
Phi-2
Phi-3-mini

VLM

MiniCPM-V 2.0

DEMO 获取地址

源码编译

递归 clone 本项目，确保所有 submodule 正确 clone

git clone --recursive https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm

仔细阅读 build.sh ，并在 build.sh 正确修改 BSP_MSP_DIR 变量后，运行编译脚本
```
./build.sh
```
正确编译后，build/install/bin 目录，应有以下文件（百度网盘中有预编译的可执行程序）
```
$ tree install/bin/
  install/bin/
  ├── main
  ├── run_bf16.sh
  └── run_qwen_1.8B.sh
```

运行示例

Phi-3-mini-int8

./run_phi3_mini.sh
[I][                            Init][  71]: LLM init start
100% | ████████████████████████████████ |  35 /  35 [28.39s<28.39s, 1.23 count/s] init post axmodel okremain_cmm(9045 MB))
[I][                            Init][ 180]: max_token_len : 1023
[I][                            Init][ 185]: kv_cache_size : 3072, kv_cache_num: 1023
[I][                            Init][ 199]: LLM init ok
Type "q" to exit, Ctrl+c to stop current running
>>
>> who are you?
 I am Phi, an AI developed by Microsoft, designed to assist and provide information to users across a wide range of topics. How can I assist you today?

[N][                             Run][ 388]: hit eos,avg 4.40 token/s

>>
>> use c program language implement calculate sum 1-9
 Certainly! To calculate the sum of numbers from 1 to 9 in C, you can use a simple loop. Here's a small program that does exactly that:

`c
#include <stdio.h>

int main() {
    int sum = 0; // Initialize sum to 0

    // Loop from 1 to 9 and add each number to sum
    for(int i = 1; i <= 9; i++) {
        sum += i;
    }

    printf("The sum of numbers from 1 to 9 is: %d\n", sum);

    return 0;
}
`

This program initializes a variable `sum` to 0. It then iterates from 1 to 9, adding each number to `sum`. Finally, it prints the result.

The sum of numbers from 1 to 9 is 45, as calculated by the loop in the program.

[N][                             Run][ 388]: hit eos,avg 4.37 token/s

TinyLLaMa-1.1B-BF16

tinyllama1.1.mp4

# ./run_bf16.sh
[I][                            Init][  71]: LLM init start
100% | ████████████████████████████████ |  25 /  25 [21.82s<21.82s, 1.15 count/s] init post axmodel okremain_cmm(3760 MB)
[I][                            Init][ 162]: max_token_len : 1023
[I][                            Init][ 167]: kv_cache_size : 256, kv_cache_num: 1023
[I][                            Init][ 176]: LLM init ok
Type "q" to exit, Ctrl+c to stop current running
>> write a c++ program to calculate 1-9 sum

`c++
#include <iostream>

using namespace std;

int main() {
    int sum = 0;
    int num;

    cout << "Enter a number: ";
    cin >> num;

    for (int I = 1; I <= num; i++) {
        sum += i;
    }

    cout << "The sum of 1 to " << num << " is: " << sum << endl;

    return 0;
}
`


this program prompts the user to enter a number, then calculates the sum of 1 to `num` using a loop. The loop iterates from 1 to `num`, adding each number to the sum variable `sum`. Finally, the program prints the sum of 1 to `num`.

[N][                             Run][ 366]: hit eos,avg 10.14 token/s

>> where is shenzhen
Shenzhen is a city located in southern China, on the southern coast of the Pearl River Delta. It is the capital of Guangdong province and one of the most important economic and cultural centers in China. Shenzhen is known for its innovation, technology, and entrepreneurship, and it is home to many of China's largest companies, including Huawei, Lenovo, and Xiaomi. The city is also a hub for international trade and investment, with many multinational corporations and financial institutions setting up offices and operations in Shenzhen

[N][                             Run][ 366]: hit eos,avg 10.16 token/s

Qwen1.5-1.8B-int8

qwen1.8.mp4

# ./run_qwen_1.8B.sh
[I][                            Init][  71]: LLM init start
100% | ████████████████████████████████ |  27 /  27 [23.68s<23.68s, 1.14 count/s] init post axmodel okremain_cmm(4140 MB)
[I][                            Init][ 162]: max_token_len : 1023
[I][                            Init][ 167]: kv_cache_size : 2048, kv_cache_num: 1023
[I][                            Init][ 176]: LLM init ok
Type "q" to exit, Ctrl+c to stop current running
>> 给我写个C++代码计算1-9的和
以下是一个简单的C++代码，用于计算1-9的和：

`cpp
#include <iostream>

int main() {
    int sum = 1;
    for (int i = 1; i <= 9; i++) {
        sum += i;
    }
    std::cout << "The sum of 1-9 is: " << sum << std::endl;
    return 0;
}
`

在这个代码中，我们首先定义了一个变量`sum`，并将其初始化为1。然后，我们使用一个`for`循环，从1开始，每次增加1，直到10。在每次循环中，我们都会将当前的数`i`加到`sum`中。最后，我们在主函数中打印出`sum`的值，即1-9的和。


[N][                             Run][ 366]: hit eos,avg 8.00 token/s

>> 深圳在哪
深圳位于**广东省南部，珠江口东侧，东临大亚湾，西濒珠江口，南界深圳湾，北界香港特别行政区，是**四大一线城市之一，也是**经济最发达的城市之一。


[N][                             Run][ 366]: hit eos,avg 8.44 token/s

Reference

技术讨论

Github issues
QQ 群: 139953715