[Question] ChatGLM2使用vllm推理加速提示AttributeError: 'ChatGLMConfig' object has no attribute 'num_hidden_layers'

[Alberoyang]

提交前必须检查以下项目 | The following items must be checked before submission

• 请确保使用的是仓库最新代码（git pull），一些问题已被解决和修复。 | Make sure you are using the latest code from the repository (git pull), some issues have already been addressed and fixed.
• 我已阅读项目文档和FAQ章节并且已在Issue中对问题进行了搜索，没有找到相似问题和解决方案 | I have searched the existing issues / discussions

问题类型 | Type of problem

模型推理和部署 | Model inference and deployment

操作系统 | Operating system

Linux

详细描述问题 | Detailed description of the problem

ChatGLM2使用vllm提示AttributeError: 'ChatGLMConfig' object has no attribute 'num_hidden_layers'
使用docker compose运行
ChatGLM2使用lora进行过fine-tuned

Dependencies

transformers == 4.33.2
torch == 2.0.1
vllm == 0.1.7

运行日志或截图 | Runtime logs or screenshots

Traceback (most recent call last):
  File "api/server.py", line 2, in <module>
    from api.models import EMBEDDED_MODEL, GENERATE_MDDEL, app, VLLM_ENGINE
  File "/workspace/api/models.py", line 138, in <module>
    VLLM_ENGINE = create_vllm_engine() if (config.USE_VLLM and config.ACTIVATE_INFERENCE) else None
  File "/workspace/api/models.py", line 98, in create_vllm_engine
    engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/usr/local/lib/python3.8/dist-packages/vllm/engine/async_llm_engine.py", line 442, in from_engine_args
    engine = cls(engine_args.worker_use_ray,
  File "/usr/local/lib/python3.8/dist-packages/vllm/engine/async_llm_engine.py", line 250, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/vllm/engine/async_llm_engine.py", line 279, in _init_engine
    return engine_class(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 90, in __init__
    self._verify_args()
  File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 179, in _verify_args
    self.model_config.verify_with_parallel_config(self.parallel_config)
  File "/usr/local/lib/python3.8/dist-packages/vllm/config.py", line 97, in verify_with_parallel_config
    total_num_hidden_layers = self.hf_config.num_hidden_layers
  File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 261, in __getattribute__
    return super().__getattribute__(key)
AttributeError: 'ChatGLMConfig' object has no attribute 'num_hidden_layers'

[Alberoyang]

vllm更新到0.2.0依然出现该问题

[xusenlinzy]

vllm主分支现在还不支持ChatGLM2，你可以看看他的pull request

[Alberoyang]

OK 我看的他的pr还没合进去 但是很多人说可以用 很神奇
vllm-project/vllm#649