Unable to convert tensorflow saved model to tensorrt

Question

Unable to convert tensorflow saved model to tensorrt

greatsharma opened this issue 3 years ago · 1 comments

Description

Unable to convert tensorflow saved model to tensorrt.

Environment

TensorRT Version: 7.0
GPU Type: Quadro 5000
Nvidia Driver Version: 440.33.01
CUDA Version: 10.2
CUDNN Version: 7.6.5
Operating System + Version: ubuntu 18.04 LTS
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable): 2.3.2

I am using the below code for the conversion :

import numpy as np
import tensorflow as tf
from tensorflow.python.compiler.tensorrt import trt_convert as trt

conversion_params = trt.DEFAULT_TRT_CONVERSION_PARAMS
conversion_params = conversion_params._replace(
    max_workspace_size_bytes=(1<<32))
conversion_params = conversion_params._replace(precision_mode="FP16")
conversion_params = conversion_params._replace(
    maximum_cached_engines=1)

converter = trt.TrtGraphConverterV2(input_saved_model_dir="checkpoints/yolov4",conversion_params=conversion_params)
converter.convert()

def my_input_fn():
  inp1 = np.random.normal(size=(1, 608, 608, 3)).astype(np.float32)
  yield [inp1]

output_saved_model_dir = "checkpoints/yolo4-tftrt-fp16/"

converter.build(input_fn=my_input_fn)
converter.save(output_saved_model_dir)

The above code is taken from this nvidia's code snippet.

and I am getting the following error :

2021-02-05 15:59:08.932793: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2021-02-05 15:59:09.560548: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libnvinfer.so.7
2021-02-05 15:59:10.059435: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2021-02-05 15:59:10.163049: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:10.163575: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: Quadro RTX 5000 computeCapability: 7.5
coreClock: 1.815GHz coreCount: 48 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 417.29GiB/s
2021-02-05 15:59:10.163589: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2021-02-05 15:59:10.163613: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-02-05 15:59:10.164531: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-02-05 15:59:10.164672: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-02-05 15:59:10.165684: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-02-05 15:59:10.166273: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-02-05 15:59:10.166295: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-02-05 15:59:10.166333: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:10.166875: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:10.167359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-02-05 15:59:10.190205: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3000000000 Hz
2021-02-05 15:59:10.190450: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xd0accc0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-02-05 15:59:10.190464: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-02-05 15:59:10.300647: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:10.301093: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xd637260 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-02-05 15:59:10.301106: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Quadro RTX 5000, Compute Capability 7.5
2021-02-05 15:59:10.301200: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:10.301547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: Quadro RTX 5000 computeCapability: 7.5
coreClock: 1.815GHz coreCount: 48 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 417.29GiB/s
2021-02-05 15:59:10.301564: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2021-02-05 15:59:10.301575: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-02-05 15:59:10.301582: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-02-05 15:59:10.301591: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-02-05 15:59:10.301598: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-02-05 15:59:10.301606: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-02-05 15:59:10.301612: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-02-05 15:59:10.301637: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:10.302032: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:10.302367: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-02-05 15:59:10.302383: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2021-02-05 15:59:10.799835: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-05 15:59:10.799860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2021-02-05 15:59:10.799864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2021-02-05 15:59:10.799978: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:10.800349: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:10.800692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14523 MB memory) -> physical GPU (device: 0, name: Quadro RTX 5000, pci bus id: 0000:01:00.0, compute capability: 7.5)
2021-02-05 15:59:21.261223: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:21.261441: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2021-02-05 15:59:21.261499: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2021-02-05 15:59:21.261847: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:21.262047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: Quadro RTX 5000 computeCapability: 7.5
coreClock: 1.815GHz coreCount: 48 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 417.29GiB/s
2021-02-05 15:59:21.262064: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2021-02-05 15:59:21.262076: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-02-05 15:59:21.262083: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-02-05 15:59:21.262092: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-02-05 15:59:21.262099: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-02-05 15:59:21.262105: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-02-05 15:59:21.262111: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-02-05 15:59:21.262136: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:21.262319: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:21.262486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-02-05 15:59:21.262505: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-05 15:59:21.262509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2021-02-05 15:59:21.262512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2021-02-05 15:59:21.262553: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:21.262738: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:21.262911: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14523 MB memory) -> physical GPU (device: 0, name: Quadro RTX 5000, pci bus id: 0000:01:00.0, compute capability: 7.5)
2021-02-05 15:59:21.336841: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: graph_to_optimize
2021-02-05 15:59:21.336868: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   function_optimizer: Graph size after: 2752 nodes (2208), 2791 edges (2247), time = 41.78ms.
2021-02-05 15:59:21.336872: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   function_optimizer: function_optimizer did nothing. time = 0.889ms.
2021-02-05 15:59:24.771038: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:24.771248: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2021-02-05 15:59:24.771331: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2021-02-05 15:59:24.771580: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:24.771758: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: Quadro RTX 5000 computeCapability: 7.5
coreClock: 1.815GHz coreCount: 48 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 417.29GiB/s
2021-02-05 15:59:24.771776: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2021-02-05 15:59:24.771787: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-02-05 15:59:24.771794: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-02-05 15:59:24.771800: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-02-05 15:59:24.771807: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-02-05 15:59:24.771814: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-02-05 15:59:24.771820: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-02-05 15:59:24.771844: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:24.772022: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:24.772185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-02-05 15:59:24.772204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-05 15:59:24.772208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2021-02-05 15:59:24.772211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2021-02-05 15:59:24.772251: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:24.772432: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-05 15:59:24.772600: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14523 MB memory) -> physical GPU (device: 0, name: Quadro RTX 5000, pci bus id: 0000:01:00.0, compute capability: 7.5)
2021-02-05 15:59:26.728169: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:772] There are 92 ops of 18 different types in the graph that are not converted to TensorRT: Identity, Cast, StridedSlice, Shape, Pack, ConcatV2, DataFormatVecPermute, Placeholder, Reshape, SplitV, Tile, GreaterEqual, ExpandDims, Where, Squeeze, NoOp, Prod, GatherV2, (For more information see https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#supported-ops).
2021-02-05 15:59:26.868491: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:751] Number of TensorRT candidate segments: 10
2021-02-05 15:59:27.136114: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:854] Replaced segment 0 consisting of 5 nodes by TRTEngineOp_0_0.
2021-02-05 15:59:27.136178: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:854] Replaced segment 1 consisting of 5 nodes by TRTEngineOp_0_1.
2021-02-05 15:59:27.136220: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:854] Replaced segment 2 consisting of 1231 nodes by TRTEngineOp_0_2.
2021-02-05 15:59:27.137388: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:854] Replaced segment 3 consisting of 4 nodes by StatefulPartitionedCall/functional_1/TRTEngineOp_0_3.
2021-02-05 15:59:27.137430: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:854] Replaced segment 4 consisting of 14 nodes by StatefulPartitionedCall/functional_1/TRTEngineOp_0_4.
2021-02-05 15:59:27.137471: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:854] Replaced segment 5 consisting of 14 nodes by StatefulPartitionedCall/functional_1/TRTEngineOp_0_5.
2021-02-05 15:59:27.137510: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:854] Replaced segment 6 consisting of 14 nodes by StatefulPartitionedCall/functional_1/TRTEngineOp_0_6.
2021-02-05 15:59:27.137548: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:854] Replaced segment 7 consisting of 3 nodes by StatefulPartitionedCall/functional_1/TRTEngineOp_0_7.
2021-02-05 15:59:27.137610: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:854] Replaced segment 8 consisting of 3 nodes by StatefulPartitionedCall/functional_1/TRTEngineOp_0_8.
2021-02-05 15:59:27.137645: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:854] Replaced segment 9 consisting of 3 nodes by StatefulPartitionedCall/functional_1/TRTEngineOp_0_9.
2021-02-05 15:59:28.079870: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: tf_graph
2021-02-05 15:59:28.079898: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 1349 nodes (-1082), 1497 edges (-1080), time = 938.152ms.
2021-02-05 15:59:28.079902: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   layout: Graph size after: 1517 nodes (168), 1665 edges (168), time = 226.895ms.
2021-02-05 15:59:28.079905: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 1506 nodes (-11), 1654 edges (-11), time = 186.796ms.
2021-02-05 15:59:28.079908: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   TensorRTOptimizer: Graph size after: 223 nodes (-1283), 264 edges (-1390), time = 670.537ms.
2021-02-05 15:59:28.079910: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 223 nodes (0), 264 edges (0), time = 4.657ms.
2021-02-05 15:59:28.079913: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: StatefulPartitionedCall/functional_1/TRTEngineOp_0_3_native_segment
2021-02-05 15:59:28.079916: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 9 nodes (0), 8 edges (0), time = 0.269ms.
2021-02-05 15:59:28.079918: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   layout: Graph size after: 9 nodes (0), 8 edges (0), time = 0.173ms.
2021-02-05 15:59:28.079921: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 9 nodes (0), 8 edges (0), time = 0.174ms.
2021-02-05 15:59:28.079924: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   TensorRTOptimizer: Graph size after: 9 nodes (0), 8 edges (0), time = 0.008ms.
2021-02-05 15:59:28.079926: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 9 nodes (0), 8 edges (0), time = 0.152ms.
2021-02-05 15:59:28.079929: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: StatefulPartitionedCall/functional_1/TRTEngineOp_0_8_native_segment
2021-02-05 15:59:28.079933: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.188ms.
2021-02-05 15:59:28.079935: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   layout: Graph size after: 6 nodes (0), 5 edges (0), time = 0.107ms.
2021-02-05 15:59:28.079939: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.138ms.
2021-02-05 15:59:28.079942: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.005ms.
2021-02-05 15:59:28.079945: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.1ms.
2021-02-05 15:59:28.079948: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: TRTEngineOp_0_1_native_segment
2021-02-05 15:59:28.079951: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 8 nodes (0), 7 edges (0), time = 0.184ms.
2021-02-05 15:59:28.079954: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   layout: Graph size after: 8 nodes (0), 7 edges (0), time = 0.135ms.
2021-02-05 15:59:28.079957: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 8 nodes (0), 7 edges (0), time = 0.168ms.
2021-02-05 15:59:28.079984: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   TensorRTOptimizer: Graph size after: 8 nodes (0), 7 edges (0), time = 0.007ms.
2021-02-05 15:59:28.079987: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 8 nodes (0), 7 edges (0), time = 0.136ms.
2021-02-05 15:59:28.079990: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: StatefulPartitionedCall/functional_1/TRTEngineOp_0_9_native_segment
2021-02-05 15:59:28.080025: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.147ms.
2021-02-05 15:59:28.080028: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   layout: Graph size after: 6 nodes (0), 5 edges (0), time = 0.109ms.
2021-02-05 15:59:28.080031: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.143ms.
2021-02-05 15:59:28.080070: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.006ms.
2021-02-05 15:59:28.080074: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.175ms.
2021-02-05 15:59:28.080077: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: TRTEngineOp_0_2_native_segment
2021-02-05 15:59:28.080081: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 1238 nodes (0), 1344 edges (0), time = 115.807ms.
2021-02-05 15:59:28.080084: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   layout: Graph size after: 1238 nodes (0), 1344 edges (0), time = 211.434ms.
2021-02-05 15:59:28.080087: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 1238 nodes (0), 1344 edges (0), time = 122.638ms.
2021-02-05 15:59:28.080091: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   TensorRTOptimizer: Graph size after: 1238 nodes (0), 1344 edges (0), time = 25.404ms.
2021-02-05 15:59:28.080094: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 1238 nodes (0), 1344 edges (0), time = 123.265ms.
2021-02-05 15:59:28.080098: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: TRTEngineOp_0_0_native_segment
2021-02-05 15:59:28.080141: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 8 nodes (0), 7 edges (0), time = 0.224ms.
2021-02-05 15:59:28.080145: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   layout: Graph size after: 8 nodes (0), 7 edges (0), time = 0.164ms.
2021-02-05 15:59:28.080186: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 8 nodes (0), 7 edges (0), time = 0.222ms.
2021-02-05 15:59:28.080190: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   TensorRTOptimizer: Graph size after: 8 nodes (0), 7 edges (0), time = 0.009ms.
2021-02-05 15:59:28.080193: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 8 nodes (0), 7 edges (0), time = 0.151ms.
2021-02-05 15:59:28.080197: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: StatefulPartitionedCall/functional_1/TRTEngineOp_0_7_native_segment
2021-02-05 15:59:28.080200: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.11ms.
2021-02-05 15:59:28.080204: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   layout: Graph size after: 6 nodes (0), 5 edges (0), time = 0.104ms.
2021-02-05 15:59:28.080207: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.097ms.
2021-02-05 15:59:28.080210: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.006ms.
2021-02-05 15:59:28.080214: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.101ms.
2021-02-05 15:59:28.080257: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: StatefulPartitionedCall/functional_1/TRTEngineOp_0_4_native_segment
2021-02-05 15:59:28.080260: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 18 nodes (0), 17 edges (0), time = 0.266ms.
2021-02-05 15:59:28.080301: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   layout: Graph size after: 18 nodes (0), 17 edges (0), time = 0.261ms.
2021-02-05 15:59:28.080304: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 18 nodes (0), 17 edges (0), time = 0.247ms.
2021-02-05 15:59:28.080308: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   TensorRTOptimizer: Graph size after: 18 nodes (0), 17 edges (0), time = 0.017ms.
2021-02-05 15:59:28.080311: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 18 nodes (0), 17 edges (0), time = 0.241ms.
2021-02-05 15:59:28.080315: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: StatefulPartitionedCall/functional_1/TRTEngineOp_0_6_native_segment
2021-02-05 15:59:28.080318: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 18 nodes (0), 17 edges (0), time = 0.359ms.
2021-02-05 15:59:28.080322: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   layout: Graph size after: 18 nodes (0), 17 edges (0), time = 0.252ms.
2021-02-05 15:59:28.080325: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 18 nodes (0), 17 edges (0), time = 0.246ms.
2021-02-05 15:59:28.080329: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   TensorRTOptimizer: Graph size after: 18 nodes (0), 17 edges (0), time = 0.016ms.
2021-02-05 15:59:28.080373: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 18 nodes (0), 17 edges (0), time = 0.243ms.
2021-02-05 15:59:28.080377: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: StatefulPartitionedCall/functional_1/TRTEngineOp_0_5_native_segment
2021-02-05 15:59:28.080415: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 18 nodes (0), 17 edges (0), time = 0.25ms.
2021-02-05 15:59:28.080419: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   layout: Graph size after: 18 nodes (0), 17 edges (0), time = 0.251ms.
2021-02-05 15:59:28.080422: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 18 nodes (0), 17 edges (0), time = 0.307ms.
2021-02-05 15:59:28.080426: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   TensorRTOptimizer: Graph size after: 18 nodes (0), 17 edges (0), time = 0.017ms.
2021-02-05 15:59:28.080429: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   constant_folding: Graph size after: 18 nodes (0), 17 edges (0), time = 0.34ms.
2021-02-05 15:59:30.327755: I tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1205] Linked TensorRT version: 7.0.0
2021-02-05 15:59:30.327809: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libnvinfer.so.7
2021-02-05 15:59:30.327817: I tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1206] Loaded TensorRT version: 7.0.0
2021-02-05 15:59:30.328248: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libnvinfer_plugin.so.7
2021-02-05 15:59:30.935565: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with same pad mode
2021-02-05 15:59:30.935593: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with caffe pad mode
2021-02-05 15:59:30.935598: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with same pad mode
2021-02-05 15:59:30.935601: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with caffe pad mode
2021-02-05 15:59:30.936006: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with same pad mode
2021-02-05 15:59:30.936013: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with caffe pad mode
2021-02-05 15:59:30.936017: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with same pad mode
2021-02-05 15:59:30.936020: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with caffe pad mode
2021-02-05 15:59:30.936421: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with same pad mode
2021-02-05 15:59:30.936427: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with caffe pad mode
2021-02-05 15:59:30.936431: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with same pad mode
2021-02-05 15:59:30.936434: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with caffe pad mode
2021-02-05 15:59:30.936842: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with same pad mode
2021-02-05 15:59:30.936848: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with caffe pad mode
2021-02-05 15:59:30.936851: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with same pad mode
2021-02-05 15:59:30.936855: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with caffe pad mode
2021-02-05 15:59:30.937263: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with same pad mode
2021-02-05 15:59:30.937269: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with caffe pad mode
2021-02-05 15:59:30.937273: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with same pad mode
2021-02-05 15:59:30.937276: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Can't fuse pad and convolution with caffe pad mode
2021-02-05 16:02:43.429385: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
2021-02-05 16:02:44.107285: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
2021-02-05 16:02:44.117449: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
2021-02-05 16:02:44.120380: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
2021-02-05 16:02:44.311930: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
2021-02-05 16:02:44.322992: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
2021-02-05 16:02:44.323367: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
2021-02-05 16:02:44.382562: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:39] TF-TRT Warning: DefaultLogger Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
2021-02-05 16:02:44.383996: W tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:6045] TF-TRT Warning: Validation failed for TensorRTInputPH_0 and input slot 0: Input tensor with shape [1,0,2] is an empty tensor, which is not supported by TRT
2021-02-05 16:02:44.384081: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:757] TF-TRT Warning: Engine creation for TRTEngineOp_0_0 failed. The native segment will be used instead. Reason: Unimplemented: Validation failed for TensorRTInputPH_0 and input slot 0: Input tensor with shape [1,0,2] is an empty tensor, which is not supported by TRT
2021-02-05 16:02:44.384126: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:629] TF-TRT Warning: Engine retrieval for input shapes: [[1,0,2], [1,0,2]] failed. Running native segment for TRTEngineOp_0_0
2021-02-05 16:02:44.385957: W tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:6045] TF-TRT Warning: Validation failed for TensorRTInputPH_0 and input slot 0: Input tensor with shape [1,0,2] is an empty tensor, which is not supported by TRT
2021-02-05 16:02:44.386032: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:757] TF-TRT Warning: Engine creation for TRTEngineOp_0_1 failed. The native segment will be used instead. Reason: Unimplemented: Validation failed for TensorRTInputPH_0 and input slot 0: Input tensor with shape [1,0,2] is an empty tensor, which is not supported by TRT
2021-02-05 16:02:44.386068: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:629] TF-TRT Warning: Engine retrieval for input shapes: [[1,0,2], [1,0,2]] failed. Running native segment for TRTEngineOp_0_1

This is the first time I am using tensorrt, please help ...

Answer 1 · 2021-02-10T04:51:07.000Z

done, instead of using tensorflow inbuilt tensorrt api I rather used standalone tensorrt i.e., import tensorrt as trt and now it is working fine.