grimoire/mmdetection-to-tensorrt

can't use trtexec to load the .engine file

shiyongming opened this issue · 6 comments

shiyongming commented

To Reproduce

  1. export .engine file:
  • using docker "mmdet2trt_docker:v1.0" to export xxx.engine file.
  1. using NVIDIA TensorRT image to test the .engine file:
  • "docker run --rm -it --gpus all -v [your_local_path]:/workspace/hostdir nvcr.io/nvidia/tensorrt:20.11-py3"
  • install torch
  • install torch2trt_dynamic
  • install amirstan_plugin
  • using "trtexec --loadEngine=xxx.engine --verbose" to test the .engine file

enviroment:

  • OS: [Ubuntu]
  • python_version: [3.6.9]
  • pytorch_version: [1.6.0]
  • cuda_version: [10.2]
  • mmdetection_version: [2.3.0]
  • docker: nvcr.io/nvidia/tensorrt:20.03-py3

I attached some screenshots. Could you help to check the reason?
Thanks in advance!

1
2
3

grimoire commented

Thanks for the report. I will try to fix it.
Since this seems related to trtexec, which is part of TensorRT OSS. You can also create an issue there.

shiyongming commented

Thanks for the report. I will try to fix it.
Since this seems related to trtexec, which is part of TensorRT OSS. You can also create an issue there.

Hi Grimoir. Thanks for your reply. I will check the reason at the same time. If you need any more information please let me know. Looking forward to your update.

grimoire commented

Hi
I built the trtexec from source. It works:

TensorRT-OSS/build/out/trtexec \
  --explicitBatch \
  --shapes=input_0:1x3x800x1344 \
  --loadEngine=trt.engine \
  --plugins=amirstan_plugin/build/lib/libamirstan_plugin.so \
  --verbose

It might be a bug of prebuild trtexec.

You can also test if the plugin can be loaded in c++:

#include <iostream>
#include <string>
#include <NvInferRuntimeCommon.h>
#include <dlfcn.h>

int main(){
    std::cout << "start test\n";

    // load plugin library
    std::string path = "amirstan_plugin/build/lib/libamirstan_plugin.so";
    void* handle = dlopen(path.c_str(), RTLD_LAZY);
    if (handle == nullptr)
    {
        std::cout << "Could not load plugin library: " << path << ", due to: " << dlerror() << std::endl;
    }

    // get registry and creator
    auto registry = getPluginRegistry();
    auto creator = registry->getPluginCreator("BatchedNMS_TRT_CUSTOM", "1", "");
    if (creator==nullptr){
        std::cout << "Could not read plugins\n";
    }

    std::cout << "finish\n";

    return 0;
}
Chen-cyw commented

@grimoire I use you c++ code and test all the plugins (the libamirstan_plugin.so is build with cuda10.2 tensorrt7.2.1.6)

#include <iostream>
#include <string>
#include <NvInferRuntimeCommon.h>
#include <dlfcn.h>
#include <vector>

int main(){
    std::cout << "start test"<<std::endl;

    std::string path = "/home/cyw/convert2trt/amirstan_plugin/build/lib/libamirstan_plugin.so";
    void* handle = dlopen(path.c_str(), RTLD_LAZY);
    if (handle == nullptr)
    {
        std::cout << "Could not load plugin library: " << path << ", due to: " << dlerror() << std::endl;
    }

    auto registry = getPluginRegistry();

    std::vector<std::string> liblist={"AdaptivePoolPluginDynamic","BatchedNMS_TRT_CUSTOM","CarafeFeatureReassemblePluginDynamic",
                                      "DeformableConvPluginDynamic","ModulatedDeformableConvPluginDynamic","DeformablePoolPluginDynamic",
                                     "Delta2BBoxPluginDynamic","ExViewPluginDynamic","GridAnchorDynamicPluginDynamic","GridSamplePluginDynamic",
                                     "GroupNormPluginDynamic","LayerNormPluginDynamic","MeshGridPluginDynamic","RepeatDimsPluginDynamic",
                                     "RoiExtractorPluginDynamic","RoiPoolPluginDynamic","TorchCumMaxMinPluginDynamic","TorchCumPluginDynamic",
                                     "TorchFlipPluginDynamic","TorchGatherPluginDynamic","TorchNMSPluginDynamic"};
    for(auto libname:liblist)
    {
        auto creator = registry->getPluginCreator(libname.c_str(), "1", "");

        if (creator==nullptr){
            std::cout << "Could not read plugins: "<<libname<<std::endl;
        }
        else {
            std::cout<<"read plugins success: "<<libname<<std::endl;
        }
    }

    std::cout << "finish\n";

    return 0;
}

finally find only the "BatchedNMS_TRT_CUSTOM" plugin can not be registry ,the result is:

read plugins success: AdaptivePoolPluginDynamic
Could not read plugins: BatchedNMS_TRT_CUSTOM
read plugins success: CarafeFeatureReassemblePluginDynamic
read plugins success: DeformableConvPluginDynamic
read plugins success: ModulatedDeformableConvPluginDynamic
read plugins success: DeformablePoolPluginDynamic
read plugins success: Delta2BBoxPluginDynamic
read plugins success: ExViewPluginDynamic
read plugins success: GridAnchorDynamicPluginDynamic
read plugins success: GridSamplePluginDynamic
read plugins success: GroupNormPluginDynamic
read plugins success: LayerNormPluginDynamic
read plugins success: MeshGridPluginDynamic
read plugins success: RepeatDimsPluginDynamic
read plugins success: RoiExtractorPluginDynamic
read plugins success: RoiPoolPluginDynamic
read plugins success: TorchCumMaxMinPluginDynamic
read plugins success: TorchCumPluginDynamic
read plugins success: TorchFlipPluginDynamic
read plugins success: TorchGatherPluginDynamic
read plugins success: TorchNMSPluginDynamic
grimoire commented

Errr...I don't know.
Batched nms can be readed on my device.
Are you using the latest plugin code? Is there any error log in conversion and inference?

update:
There might be conflict between my Batched NMS and the official one. I have update the plugin code. Please clone the latest repo and try again.

Chen-cyw commented

Both "trtexec" and the test code can load all plugins correctly.
Thanks for all your hard work!