grimoire/mmdetection-to-tensorrt

error when convert mmdetection-yolov3 to tensorrt

Closed this issue · 9 comments

ShaneYS commented

1. enviroment:

  • OS: ubuntu 18.04
  • python_version: Python 3.7.6
  • pytorch_version:
>>> torch.__version__
'1.7.0+cu110'
>>> torchvision.__version__
'0.8.1+cu110'
>>> mmdet.__version__
'2.5.0'
>>> mmcv.__version__
'1.2.0'
>>> tensorrt.__version__
'7.2.1.6'
  • cuda_version:
nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Thu_Jun_11_22:26:38_PDT_2020
Cuda compilation tools, release 11.0, V11.0.194
Build cuda_11.0_bu.TC445_37.28540450_0
  • cudnn_version:
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 0
#define CUDNN_PATCHLEVEL 5

  • mmdetection_version: 2.5.0

  • nvidia driver version:

NVIDIA-SMI 455.38       Driver Version: 455.38       CUDA Version: 11.1

2. error infomation

/home/yangshuai/anaconda3/envs/mmlab/bin/python /home/yangshuai/Downloads/pycharm-2019.3.3/plugins/python/helpers/pydev/pydevd.py --cmd-line --multiproc --qt-support=auto --client 127.0.0.1 --port 46145 --file /home/yangshuai/project/runtime/mmdetection-to-tensorrt/demo/inference.py
pydev debugger: process 6725 is connecting

Connected to pydev debugger (build 193.6494.30)
<class 'tensorrt.tensorrt.IPluginFactory'>
WARNING:root:module mmdet.models.VFNetHead not exist.
WARNING:root:module mmdet.models.VFNet not exist.
[TensorRT] ERROR: INVALID_ARGUMENT: getPluginCreator could not find plugin GridAnchorDynamicPluginDynamic version 1
Traceback (most recent call last):
  File "/home/yangshuai/project/runtime/mmdetection-to-tensorrt/mmdet2trt/mmdet2trt.py", line 147, in mmdet2trt
    int8_calib_algorithm=int8_calib_algorithm)
  File "/home/yangshuai/project/runtime/torch2trt_dynamic/torch2trt_dynamic/torch2trt_dynamic.py", line 513, in torch2trt_dynamic
    outputs = module(*inputs)
  File "/home/yangshuai/anaconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/yangshuai/project/runtime/mmdetection-to-tensorrt/mmdet2trt/models/detectors/two_stage.py", line 50, in forward
    rois = rpn_head(feat, x)
  File "/home/yangshuai/anaconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/yangshuai/project/runtime/mmdetection-to-tensorrt/mmdet2trt/models/dense_heads/rpn_head.py", line 31, in forward
    device=cls_scores[0].device)
  File "/home/yangshuai/anaconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/yangshuai/project/runtime/mmdetection-to-tensorrt/mmdet2trt/core/anchor/anchor_generator.py", line 61, in forward
    x, stride=self.generator.strides[index], device=device)
  File "/home/yangshuai/anaconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/yangshuai/project/runtime/torch2trt_dynamic/torch2trt_dynamic/torch2trt_dynamic.py", line 307, in wrapper
    converter['converter'](ctx)
  File "/home/yangshuai/project/runtime/mmdetection-to-tensorrt/mmdet2trt/converters/anchor_generator.py", line 28, in convert_AnchorGeneratorDynamic
    base_anchors=base_anchors)
  File "/home/yangshuai/project/runtime/mmdetection-to-tensorrt/mmdet2trt/converters/plugins/create_gridanchordynamic_plugin.py", line 69, in create_gridanchordynamic_plugin
    return creator.create_plugin(layer_name, pfc)
AttributeError: 'NoneType' object has no attribute 'create_plugin'

It seems Python can not find plugin in create_gridanchordynamic_plugin.py, creator returns None:

    creator = trt.get_plugin_registry().get_plugin_creator(
        'GridAnchorDynamicPluginDynamic', '1', '')

I had install TensorRT, torch2trt_dynamic, amirstan_plugin, mmdet, mmdetection-to-tensorrt according to instructions.

ShaneYS commented

execute in terminal:

(mmlab) $ python inference.py 
WARNING:root:module mmdet.models.VFNetHead not exist.
WARNING:root:module mmdet.models.VFNet not exist.
[TensorRT] ERROR: INVALID_ARGUMENT: getPluginCreator could not find plugin GridAnchorDynamicPluginDynamic version 1
Traceback (most recent call last):
  File "inference.py", line 77, in <module>
    main()
  File "inference.py", line 40, in main
    device=args.device)
  File "/home/yangshuai/project/runtime/mmdetection-to-tensorrt/mmdet2trt/mmdet2trt.py", line 147, in mmdet2trt
    int8_calib_algorithm=int8_calib_algorithm)
  File "/home/yangshuai/project/runtime/torch2trt_dynamic/torch2trt_dynamic/torch2trt_dynamic.py", line 513, in torch2trt_dynamic
    outputs = module(*inputs)
  File "/home/yangshuai/anaconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/yangshuai/project/runtime/mmdetection-to-tensorrt/mmdet2trt/models/detectors/single_stage.py", line 49, in forward
    result = bbox_head(feat, x)
  File "/home/yangshuai/anaconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/yangshuai/project/runtime/mmdetection-to-tensorrt/mmdet2trt/models/dense_heads/yolo_head.py", line 42, in forward
    pred_maps_list, device=pred_maps_list[0].device)
  File "/home/yangshuai/anaconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/yangshuai/project/runtime/mmdetection-to-tensorrt/mmdet2trt/core/anchor/anchor_generator.py", line 61, in forward
    x, stride=self.generator.strides[index], device=device)
  File "/home/yangshuai/anaconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/yangshuai/project/runtime/torch2trt_dynamic/torch2trt_dynamic/torch2trt_dynamic.py", line 307, in wrapper
    converter['converter'](ctx)
  File "/home/yangshuai/project/runtime/mmdetection-to-tensorrt/mmdet2trt/converters/anchor_generator.py", line 28, in convert_AnchorGeneratorDynamic
    base_anchors=base_anchors)
  File "/home/yangshuai/project/runtime/mmdetection-to-tensorrt/mmdet2trt/converters/plugins/create_gridanchordynamic_plugin.py", line 69, in create_gridanchordynamic_plugin
    return creator.create_plugin(layer_name, pfc)
AttributeError: 'NoneType' object has no attribute 'create_plugin'
double free or corruption (!prev)
已放弃 (核心已转储)
grimoire commented

Have you export the enviroment variable?

export AMIRSTAN_LIBRARY_PATH=${amirstan_plugin_root}/build/lib

And driver 455 might cause some error, please switch to driver <= 450.36

maggiez0138 commented

Thanks for the author's work. I have also tried yolov3 which was trained on coco, it does work.

ShaneYS commented

Have you export the enviroment variable?

export AMIRSTAN_LIBRARY_PATH=${amirstan_plugin_root}/build/lib

And driver 455 might cause some error, please switch to driver <= 450.36

Environment Var had already add to ~/.bashrc according to the README of amirstan_plugin

I also down upgrade driver version to 450.36.
But still can not find plugin.

grimoire commented

Errr...
please try the docker image. If it works, reinstall all repos follow the dockfile (the version of each package does not need to be the same).

ShaneYS commented

I checkout commit id = 25e3a9207dd4f1db86a7031745d0a10b4b6ff2bb, and can export mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth with python demo/inference.py successfully, and can inference with trt_model.

Weird ...

  • 👍1
  • 😄1
  • 🎉1
grimoire commented

@ShaneYS Congratulations.
I have regist all these plugins double times before this commit. Guess that is the cause of the error, my fault.
Did it work on 455?

ShaneYS commented

@ShaneYS Congratulations.
I have regist all these plugins double times before this commit. Guess that is the cause of the error, my fault.
Did it work on 455?

Yes，I test on 450.80 and 455.38 and both works well.
Thanks for your nice work.

Markdcle commented

I checkout commit id = 25e3a9207dd4f1db86a7031745d0a10b4b6ff2bb, and can export mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth with python demo/inference.py successfully, and can ‎推理‎ trt_model.

Weird ...

@ShaneYS hi,what the id is? I 'm afraid I can't understand it.

  • 😕1