do you have a plan for windows
lqyp0420 opened this issue · 8 comments
the newest tensorrt 8.2 now has a python api for windows, do you have a plan for windows in future. I have made a trt model in ubuntu, then copy it to windows, there has some error like this:
[10/03/2021-13:05:04] [TRT] [W] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
Traceback (most recent call last):
File "E:/ai/mmdetection-2.17.0/demo/mmdet2trt_demo.py", line 55, in
main()
File "E:/ai/mmdetection-2.17.0/demo/mmdet2trt_demo.py", line 35, in main
trt_detector = create_wrap_detector(args.save_path, cfg_path, args.device)
File "E:\ai\mmdetection-2.17.0\demo\mmdet2trt_inference.py", line 190, in create_wrap_detector
trt_model = init_trt_model(trt_model)
File "E:\ai\mmdetection-2.17.0\demo\mmdet2trt_inference.py", line 13, in init_trt_model
model_trt.load_state_dict(torch.load(trt_model_path))
File "C:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1392, in load_state_dict
load(self)
File "C:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1387, in load
state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
File "E:\ai\mmdetection-2.17.0\demo\trt_model.py", line 78, in _load_from_state_dict
self.context = self.engine.create_execution_context()
AttributeError: 'NoneType' object has no attribute 'create_execution_context'
[10/03/2021-13:05:04] [TRT] [E] 1: [pluginV2Runner.cpp::nvinfer1::rt::load::290] Error Code 1: Serialization (Serialization assertion creator failed.Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)
[10/03/2021-13:05:04] [TRT] [E] 4: [runtime.cpp::nvinfer1::Runtime::deserializeCudaEngine::50] Error Code 4: Internal Error (Engine deserialization failed.)
thanks
I have not tested on TensorRT 8.2 yet.
On TensorRT7.x, you can convert the model on linux and inference on windows. You should make sure all the environment are same on different system, such as device, cuda version, cudnn version etc.
I have not tested on TensorRT 8.2 yet. On TensorRT7.x, you can convert the model on linux and inference on windows. You should make sure all the environment are same on different system, such as device, cuda version, cudnn version etc.
I have successful created amirstan_plugin dll file in windows 10 (by vs2019 nmake and cl) and it worked 4X speed up for cascade_rcnn model from mmdetection. thanks a lot. but when i used it for my own model based on cascade_rcnn in ubuntu, it has some error.
i open a new issue and can you give some help?
I have not tested on TensorRT 8.2 yet. On TensorRT7.x, you can convert the model on linux and inference on windows. You should make sure all the environment are same on different system, such as device, cuda version, cudnn version etc.
I have successful created amirstan_plugin dll file in windows 10 (by vs2019 nmake and cl) and it worked 4X speed up for cascade_rcnn model from mmdetection. thanks a lot. but when i used it for my own model based on cascade_rcnn in ubuntu, it has some error. i open a new issue and can you give some help?
@lqyp0420
I'm interested in how to modify amirstan_plugin for Windows compilation. I noticed that you own a repository named amirstan_plugin_windows, but it is empty.
I built the amirstan_plugin by Visual Studio 2017 in Windows11 successfully, but I can't build my C++ application by amirstan_plugin.lib. The compilation error roughly means "unresolved external symbol" at the "initLibAmirstanInferPlugins" function.
I commented on the "initLibAmirstanInferPlugins" function thus my code can be compiled successfully, but it failed to create a CUDA engine.
I also tried to use DLL, but my code crashed directly at the "initLibAmirstanInferPlugins" function without any output information.
My model file is exported from Windows Python API, and it can inference as I expect with Python.
What I want to do is to deploy my model with C++ instead of Python.
I would appreciate any help.
What command you used for creating amirstan_plugin dll file in windows 10?
I used :
cd build
cmake -DTENSORRT_DIR=$Env:TENSORRT_DIR ..
msbuild .\ALL_BUILD.vcxproj
but failed.
I have not tested on TensorRT 8.2 yet. On TensorRT7.x, you can convert the model on linux and inference on windows. You should make sure all the environment are same on different system, such as device, cuda version, cudnn version etc.
I have successful created amirstan_plugin dll file in windows 10 (by vs2019 nmake and cl) and it worked 4X speed up for cascade_rcnn model from mmdetection. thanks a lot. but when i used it for my own model based on cascade_rcnn in ubuntu, it has some error. i open a new issue and can you give some help?
@lqyp0420 I'm interested in how to modify amirstan_plugin for Windows compilation. I noticed that you own a repository named amirstan_plugin_windows, but it is empty.
I built the amirstan_plugin by Visual Studio 2017 in Windows11 successfully, but I can't build my C++ application by amirstan_plugin.lib. The compilation error roughly means "unresolved external symbol" at the "initLibAmirstanInferPlugins" function.
I commented on the "initLibAmirstanInferPlugins" function thus my code can be compiled successfully, but it failed to create a CUDA engine.
I also tried to use DLL, but my code crashed directly at the "initLibAmirstanInferPlugins" function without any output information.
My model file is exported from Windows Python API, and it can inference as I expect with Python.
What I want to do is to deploy my model with C++ instead of Python.
I would appreciate any help.
How did you built the amirstan_plugin by Visual Studio 2017 in Windows11 successfully?
I failed with:
TensorRT 8.1 ; cuda 11.4
cmd:
cmake -DTENSORRT_DIR=$Env:TENSORRT_DIR ..
msbuild .\ALL_BUILD.vcxproj
Could you please provide the cmd and versions?
I have not tested on TensorRT 8.2 yet. On TensorRT7.x, you can convert the model on linux and inference on windows. You should make sure all the environment are same on different system, such as device, cuda version, cudnn version etc.
I have successful created amirstan_plugin dll file in windows 10 (by vs2019 nmake and cl) and it worked 4X speed up for cascade_rcnn model from mmdetection. thanks a lot. but when i used it for my own model based on cascade_rcnn in ubuntu, it has some error. i open a new issue and can you give some help?
@lqyp0420 I'm interested in how to modify amirstan_plugin for Windows compilation. I noticed that you own a repository named amirstan_plugin_windows, but it is empty.
I built the amirstan_plugin by Visual Studio 2017 in Windows11 successfully, but I can't build my C++ application by amirstan_plugin.lib. The compilation error roughly means "unresolved external symbol" at the "initLibAmirstanInferPlugins" function.
I commented on the "initLibAmirstanInferPlugins" function thus my code can be compiled successfully, but it failed to create a CUDA engine.
I also tried to use DLL, but my code crashed directly at the "initLibAmirstanInferPlugins" function without any output information.
My model file is exported from Windows Python API, and it can inference as I expect with Python.
What I want to do is to deploy my model with C++ instead of Python.
I would appreciate any help.How did you built the amirstan_plugin by Visual Studio 2017 in Windows11 successfully?
I failed with: TensorRT 8.1 ; cuda 11.4 cmd: cmake -DTENSORRT_DIR=$Env:TENSORRT_DIR .. msbuild .\ALL_BUILD.vcxproj
Could you please provide the cmd and versions?
My version are TensorRT-8.2.1.8, CUDA10.2, msvc19.16.27045.
To notice that, Before TensorRT8.2, the official did not provide prebuilt Python packages. This means that you can only use the C++ version of TensorRT, which cannot build the amirstan_plugin.
Due to the differences between Linux and Windows, you should modify the CMakeLists.txt and other files manually, make the plugin can be compiled successfully.
I have not tested on TensorRT 8.2 yet. On TensorRT7.x, you can convert the model on linux and inference on windows. You should make sure all the environment are same on different system, such as device, cuda version, cudnn version etc.
I have successful created amirstan_plugin dll file in windows 10 (by vs2019 nmake and cl) and it worked 4X speed up for cascade_rcnn model from mmdetection. thanks a lot. but when i used it for my own model based on cascade_rcnn in ubuntu, it has some error. i open a new issue and can you give some help?
@lqyp0420 I'm interested in how to modify amirstan_plugin for Windows compilation. I noticed that you own a repository named amirstan_plugin_windows, but it is empty.
I built the amirstan_plugin by Visual Studio 2017 in Windows11 successfully, but I can't build my C++ application by amirstan_plugin.lib. The compilation error roughly means "unresolved external symbol" at the "initLibAmirstanInferPlugins" function.
I commented on the "initLibAmirstanInferPlugins" function thus my code can be compiled successfully, but it failed to create a CUDA engine.
I also tried to use DLL, but my code crashed directly at the "initLibAmirstanInferPlugins" function without any output information.
My model file is exported from Windows Python API, and it can inference as I expect with Python.
What I want to do is to deploy my model with C++ instead of Python.
I would appreciate any help.How did you built the amirstan_plugin by Visual Studio 2017 in Windows11 successfully?
I failed with: TensorRT 8.1 ; cuda 11.4 cmd: cmake -DTENSORRT_DIR=$Env:TENSORRT_DIR .. msbuild .\ALL_BUILD.vcxproj
Could you please provide the cmd and versions?My version are TensorRT-8.2.1.8, CUDA10.2, msvc19.16.27045.
To notice that, Before TensorRT8.2, the official did not provide prebuilt Python packages. This means that you can only use the C++ version of TensorRT, which cannot build the amirstan_plugin.
Due to the differences between Linux and Windows, you should modify the CMakeLists.txt and other files manually, make the plugin can be compiled successfully.
I deleted some args from CMakeLists.txt and create the ALL_BUILD.vcxproj successfully, but encounter the errors while execute the cmd "msbuild .\ALL_BUILD.vcxproj" that I can't fix:
"D:\workspace\deep_learning\open\amirstan_plugin\build\ALL_BUILD.vcxproj" (default target) (1) ->
"D:\workspace\deep_learning\open\amirstan_plugin\build\src\plugin\amirstan_plugin.vcxproj" (default target) (5) ->
(Link target) ->
batchedNMSPlugin_static.lib(batchedNMSPlugin.obj) : error LNK2019: unresolved external symbol "enum pluginStatus_t __cdecl nmsInference(struct CUstream_st *,int,int,int,bool,int,int,int,int,int,float,float,e
num nvinfer1::DataType,void const *,enum nvinfer1::DataType,void const *,void *,void *,void *,void *,void *,bool,bool,bool)" (?nmsInference@@ya?AW4pluginStatus_t@@PEAUCUstream_st@@HHH_NHHHHHMMW4DataType@nvinfe
r1@@PEBX23PEAX4444111@Z) referenced in function "public: virtual int __cdecl amirstan::plugin::BatchedNMSPluginCustom::enqueue(struct nvinfer1::PluginTensorDesc const *,struct nvinfer1::PluginTensorDesc const
*,void const * const *,void * const *,void *,struct CUstream_st *)" (?enqueue@BatchedNMSPluginCustom@plugin@amirstan@@UEAAHPEBUPluginTensorDesc@nvinfer1@@0PEBQEBXPEBQEAXPEAXPEAUCUstream_st@@@z) [D:\workspace\d
eep_learning\open\amirstan_plugin\build\src\plugin\amirstan_plugin.vcxproj]
D:\workspace\deep_learning\open\amirstan_plugin\build\bin\Debug\amirstan_plugin.dll : fatal error LNK1120: 1 unresolved externals [D:\workspace\deep_learning\open\amirstan_plugin\build\src\plugin\amirstan_pl
ugin.vcxproj]
It seems need to modify .cpp or .hpp like you said. Could you please attach the modified files by dragging & dropping, or any repo or blog that recorded the changes.
@DeepBlue97
Try the following modifications
- Add #include "batchedNMSInference.h" to the file .\src\plugin\batchedNMSPlugin\batchedNMSInference.cpp
- Modify the keyword or to || at .\src\plugin\torchCumPlugin\torchCumPlugin.cpp, where may be around line 58.
- Add #include "common.h" to the file .\src\plugin\batchedNMSPlugin\kernel.h