Tlntin/Qwen-TensorRT-LLM

Qwen2 编译错误

mogoxx opened this issue · 5 comments

mogoxx commented

使用的基础镜像
nvcr.io/nvidia/tritonserver:23.12-trtllm-python-py3

Tensor-LLM 安装
pip install tensorrt_llm-0.7.0-cp310-cp310-linux_x86_64.whl

编译命令
python build.py --hf_model_dir /app/model/Qwen1.5-14B-Chat
--dtype float16
--remove_input_padding
--use_gemm_plugin float16
--use_gpt_attention_plugin float16
--use_inflight_batching
--max_batch_size 2
--max_input_len 2048
--max_output_len 2048
--output_dir /app/model/trt_engines/fp16/1-gpu/

Traceback (most recent call last):
File "/app/Qwen-TensorRT-LLM/examples/qwen2/build.py", line 13, in
from transformers.models.qwen2 import Qwen2Config, Qwen2ForCausalLM
ModuleNotFoundError: No module named 'transformers.models.qwen2'

好像是transformers 版本问题
pip list|grep tran
transformers 4.33.1
transformers-stream-generator 0.0.4

Tlntin commented

qwen2需要transformers 4.37.1以上

mogoxx commented

qwen2需要transformers 4.37.1以上

Tensor-LLM需要对应哪个版本

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorrt-llm 0.7.0 requires transformers==4.33.1, but you have transformers 4.37.2 which is incompatible.

image

Tlntin commented

忽略警告即可。

mogoxx commented

忽略警告即可。

会报别的错误，提示Tensor-LLM版本不一致

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_common.py", line 56, in _init
torch.classes.load_library(ft_decoder_lib)
File "/usr/local/lib/python3.10/dist-packages/torch/_classes.py", line 51, in load_library
torch.ops.load_library(path)
File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 933, in load_library
ctypes.CDLL(path)
File "/usr/lib/python3.10/ctypes/init.py", line 374, in init
self._handle = _dlopen(self._name, mode)
OSError: /usr/local/lib/python3.10/dist-packages/tensorrt_llm/libs/libth_common.so: undefined symbol: _ZN3c1017RegisterOperatorsD1Ev

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/app/Qwen-TensorRT-LLM/examples/qwen2/build.py", line 15, in
from model import Qwen2ForCausalLM as Qwen2ForCausalLM_TRT
File "/app/Qwen-TensorRT-LLM/examples/qwen2/model.py", line 7, in
from tensorrt_llm._common import default_net, default_trtnet, precision
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/init.py", line 61, in
_init(log_level="error")
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_common.py", line 59, in _init
raise ImportError(str(e) + msg)
ImportError: /usr/local/lib/python3.10/dist-packages/tensorrt_llm/libs/libth_common.so: undefined symbol: _ZN3c1017RegisterOperatorsD1Ev
FATAL: Decoding operators failed to load. This may be caused by the incompatibility between PyTorch and TensorRT-LLM. Please rebuild and install TensorRT-LLM.

Tlntin commented

降pytorch版本到2.1

pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121