Image Classification example with TensorRT 7 and TF 2.1 running into problems
mankeyboy opened this issue · 6 comments
mankeyboy commented
Hi everyone, I'm trying to run the Image Classification example with TF 2.0 and TensorRT 7.
The changes in the API have now made it that the dependency on NETS has been removed but now, it seems that the code asks for a frozen graph file.
I tried downloading the file from here: http://download.tensorflow.org/models/official/resnet_v1_imagenet_savedmodel.tar.gz
and then I put it in a folder and gave the following arguments for a synthetic run:
python image_classification.py --input_saved_model_dir data/resnet_v1_50/1523293981 --output_saved_model_dir trt_engine --batch_size 128 --data_dir . --use_trt --num_iterations 100 --use_synthetic --precision FP16 --mode benchmark
The output that I got was this:
$ python image_classification.py --input_saved_model_dir data/resnet_v1_50/1523293981 --output_saved_model_dir trt_engine --batch_size 128 --data_dir . --use_trt --num_iterations 100 --use_synthetic --precision FP16 --mode benchmark
2020-01-08 08:35:37.437327: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-01-08 08:35:38.144641: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.7
2020-01-08 08:35:38.145234: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.7
2020-01-08 08:35:38.806098: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-01-08 08:35:39.098330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:39.099944: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 1 with properties:
pciBusID: 0001:0b:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:39.101526: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 2 with properties:
pciBusID: 0003:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:39.103104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 3 with properties:
pciBusID: 0030:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:39.104678: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 4 with properties:
pciBusID: 0031:0c:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:39.106247: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 5 with properties:
pciBusID: 0033:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:39.106266: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-01-08 08:35:39.106300: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-01-08 08:35:39.107309: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-01-08 08:35:39.107527: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-01-08 08:35:39.108603: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-01-08 08:35:39.109430: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-01-08 08:35:39.109456: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-01-08 08:35:39.127996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0, 1, 2, 3, 4, 5
Found the following GPUs:
PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')
PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU')
PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU')
PhysicalDevice(name='/physical_device:GPU:3', device_type='GPU')
PhysicalDevice(name='/physical_device:GPU:4', device_type='GPU')
PhysicalDevice(name='/physical_device:GPU:5', device_type='GPU')
2020-01-08 08:35:39.752039: I tensorflow/core/platform/profile_utils/cpu_utils.cc:101] CPU Frequency: 3800000000 Hz
2020-01-08 08:35:39.789166: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x11de6c7f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-01-08 08:35:39.789194: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-01-08 08:35:40.459952: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x11b466030 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-01-08 08:35:40.460003: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla T4, Compute Capability 7.5
2020-01-08 08:35:40.460013: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (1): Tesla T4, Compute Capability 7.5
2020-01-08 08:35:40.460021: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (2): Tesla T4, Compute Capability 7.5
2020-01-08 08:35:40.460030: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (3): Tesla T4, Compute Capability 7.5
2020-01-08 08:35:40.460038: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (4): Tesla T4, Compute Capability 7.5
2020-01-08 08:35:40.460046: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (5): Tesla T4, Compute Capability 7.5
2020-01-08 08:35:40.475154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:40.476623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 1 with properties:
pciBusID: 0001:0b:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:40.478082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 2 with properties:
pciBusID: 0003:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:40.479547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 3 with properties:
pciBusID: 0030:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:40.481018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 4 with properties:
pciBusID: 0031:0c:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:40.482486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 5 with properties:
pciBusID: 0033:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:40.482511: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-01-08 08:35:40.482526: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-01-08 08:35:40.482543: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-01-08 08:35:40.482557: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-01-08 08:35:40.482570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-01-08 08:35:40.482584: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-01-08 08:35:40.482596: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-01-08 08:35:40.499927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0, 1, 2, 3, 4, 5
2020-01-08 08:35:40.499952: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-01-08 08:35:43.993284: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-01-08 08:35:43.993343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0 1 2 3 4 5
2020-01-08 08:35:43.993353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N N N N N N
2020-01-08 08:35:43.993362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 1: N N N N N N
2020-01-08 08:35:43.993370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 2: N N N N N N
2020-01-08 08:35:43.993377: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 3: N N N N N N
2020-01-08 08:35:43.993385: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 4: N N N N N N
2020-01-08 08:35:43.993392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 5: N N N N N N
2020-01-08 08:35:44.004154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15007 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-01-08 08:35:44.007362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 15007 MB memory) -> physical GPU (device: 1, name: Tesla T4, pci bus id: 0001:0b:00.0, compute capability: 7.5)
2020-01-08 08:35:44.010555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 15007 MB memory) -> physical GPU (device: 2, name: Tesla T4, pci bus id: 0003:01:00.0, compute capability: 7.5)
2020-01-08 08:35:44.013793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 15007 MB memory) -> physical GPU (device: 3, name: Tesla T4, pci bus id: 0030:01:00.0, compute capability: 7.5)
2020-01-08 08:35:44.016160: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 15007 MB memory) -> physical GPU (device: 4, name: Tesla T4, pci bus id: 0031:0c:00.0, compute capability: 7.5)
2020-01-08 08:35:44.018537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:5 with 15007 MB memory) -> physical GPU (device: 5, name: Tesla T4, pci bus id: 0033:01:00.0, compute capability: 7.5)
WARNING: Logging before flag parsing goes to stderr.
W0108 08:35:44.289335 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'global_step:0' shape=() dtype=int64_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:44.289505 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/conv2d/kernel:0' shape=(7, 7, 3, 64) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:44.289599 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/conv2d_1/kernel:0' shape=(1, 1, 64, 256) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:44.289681 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/batch_normalization/gamma:0' shape=(256,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:44.289761 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/batch_normalization/beta:0' shape=(256,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:44.682507 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'global_step:0' shape=() dtype=int64_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:44.682631 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/conv2d/kernel:0' shape=(7, 7, 3, 64) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:44.682709 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/conv2d_1/kernel:0' shape=(1, 1, 64, 256) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:44.682779 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/batch_normalization/gamma:0' shape=(256,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:44.682849 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/batch_normalization/beta:0' shape=(256,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:45.116922 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'global_step:0' shape=() dtype=int64_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:45.117060 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/conv2d/kernel:0' shape=(7, 7, 3, 64) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:45.117138 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/conv2d_1/kernel:0' shape=(1, 1, 64, 256) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:45.117208 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/batch_normalization/gamma:0' shape=(256,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:45.117277 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/batch_normalization/beta:0' shape=(256,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:45.661960 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'global_step:0' shape=() dtype=int64_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:45.662095 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/conv2d/kernel:0' shape=(7, 7, 3, 64) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:45.662174 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/conv2d_1/kernel:0' shape=(1, 1, 64, 256) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:45.662245 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/batch_normalization/gamma:0' shape=(256,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:45.662314 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/batch_normalization/beta:0' shape=(256,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2020-01-08 08:35:46.268405: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 6
2020-01-08 08:35:46.268503: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-01-08 08:35:46.272341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:46.273803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 1 with properties:
pciBusID: 0001:0b:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:46.275260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 2 with properties:
pciBusID: 0003:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:46.276709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 3 with properties:
pciBusID: 0030:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:46.278160: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 4 with properties:
pciBusID: 0031:0c:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:46.279609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 5 with properties:
pciBusID: 0033:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:46.279635: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-01-08 08:35:46.279650: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-01-08 08:35:46.279667: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-01-08 08:35:46.279681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-01-08 08:35:46.279695: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-01-08 08:35:46.279708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-01-08 08:35:46.279719: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-01-08 08:35:46.296990: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0, 1, 2, 3, 4, 5
2020-01-08 08:35:46.310118: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-01-08 08:35:46.310133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0 1 2 3 4 5
2020-01-08 08:35:46.310142: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N N N N N N
2020-01-08 08:35:46.310150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 1: N N N N N N
2020-01-08 08:35:46.310158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 2: N N N N N N
2020-01-08 08:35:46.310165: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 3: N N N N N N
2020-01-08 08:35:46.310172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 4: N N N N N N
2020-01-08 08:35:46.310180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 5: N N N N N N
2020-01-08 08:35:46.320461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15007 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-01-08 08:35:46.321944: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 15007 MB memory) -> physical GPU (device: 1, name: Tesla T4, pci bus id: 0001:0b:00.0, compute capability: 7.5)
2020-01-08 08:35:46.323416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 15007 MB memory) -> physical GPU (device: 2, name: Tesla T4, pci bus id: 0003:01:00.0, compute capability: 7.5)
2020-01-08 08:35:46.324887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 15007 MB memory) -> physical GPU (device: 3, name: Tesla T4, pci bus id: 0030:01:00.0, compute capability: 7.5)
2020-01-08 08:35:46.326360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 15007 MB memory) -> physical GPU (device: 4, name: Tesla T4, pci bus id: 0031:0c:00.0, compute capability: 7.5)
2020-01-08 08:35:46.327833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:5 with 15007 MB memory) -> physical GPU (device: 5, name: Tesla T4, pci bus id: 0033:01:00.0, compute capability: 7.5)
2020-01-08 08:35:46.458696: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:814] Optimization results for grappler item: graph_to_optimize
2020-01-08 08:35:46.458737: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] function_optimizer: function_optimizer did nothing. time = 0.002ms.
2020-01-08 08:35:46.458746: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] function_optimizer: function_optimizer did nothing. time = 0.001ms.
W0108 08:35:46.512991 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'global_step:0' shape=() dtype=int64_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:46.513137 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/conv2d/kernel:0' shape=(7, 7, 3, 64) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:46.513223 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/conv2d_1/kernel:0' shape=(1, 1, 64, 256) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:46.513302 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/batch_normalization/gamma:0' shape=(256,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
W0108 08:35:46.513379 140735937334144 wrap_function.py:214] Unable to create a python object for variable <tf.Variable 'resnet_model/batch_normalization/beta:0' shape=(256,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2020-01-08 08:35:51.228884: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.7
I0108 08:35:51.228979 140735937334144 trt_convert.py:200] Linked TensorRT version: (7, 0, 0)
I0108 08:35:51.229106 140735937334144 trt_convert.py:201] Loaded TensorRT version: (7, 0, 0)
Graph convertion...
2020-01-08 08:35:54.037460: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 6
2020-01-08 08:35:54.037547: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-01-08 08:35:54.041049: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:54.042504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 1 with properties:
pciBusID: 0001:0b:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:54.043965: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 2 with properties:
pciBusID: 0003:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:54.045426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 3 with properties:
pciBusID: 0030:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:54.046877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 4 with properties:
pciBusID: 0031:0c:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:54.048335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 5 with properties:
pciBusID: 0033:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:54.048362: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-01-08 08:35:54.048376: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-01-08 08:35:54.048394: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-01-08 08:35:54.048407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-01-08 08:35:54.048421: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-01-08 08:35:54.048434: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-01-08 08:35:54.048445: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-01-08 08:35:54.065654: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0, 1, 2, 3, 4, 5
2020-01-08 08:35:54.076765: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-01-08 08:35:54.076780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0 1 2 3 4 5
2020-01-08 08:35:54.076789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N N N N N N
2020-01-08 08:35:54.076797: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 1: N N N N N N
2020-01-08 08:35:54.076805: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 2: N N N N N N
2020-01-08 08:35:54.076812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 3: N N N N N N
2020-01-08 08:35:54.076820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 4: N N N N N N
2020-01-08 08:35:54.076827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 5: N N N N N N
2020-01-08 08:35:54.087182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15007 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-01-08 08:35:54.088659: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 15007 MB memory) -> physical GPU (device: 1, name: Tesla T4, pci bus id: 0001:0b:00.0, compute capability: 7.5)
2020-01-08 08:35:54.090132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 15007 MB memory) -> physical GPU (device: 2, name: Tesla T4, pci bus id: 0003:01:00.0, compute capability: 7.5)
2020-01-08 08:35:54.091596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 15007 MB memory) -> physical GPU (device: 3, name: Tesla T4, pci bus id: 0030:01:00.0, compute capability: 7.5)
2020-01-08 08:35:54.093065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 15007 MB memory) -> physical GPU (device: 4, name: Tesla T4, pci bus id: 0031:0c:00.0, compute capability: 7.5)
2020-01-08 08:35:54.094533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:5 with 15007 MB memory) -> physical GPU (device: 5, name: Tesla T4, pci bus id: 0033:01:00.0, compute capability: 7.5)
2020-01-08 08:35:54.160492: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:814] Optimization results for grappler item: graph_to_optimize
2020-01-08 08:35:54.160527: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] function_optimizer: function_optimizer did nothing. time = 0.001ms.
2020-01-08 08:35:54.160536: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] function_optimizer: function_optimizer did nothing. time = 0ms.
2020-01-08 08:35:58.984375: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 6
2020-01-08 08:35:58.984474: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-01-08 08:35:58.988094: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:58.989550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 1 with properties:
pciBusID: 0001:0b:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:58.991017: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 2 with properties:
pciBusID: 0003:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:58.992470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 3 with properties:
pciBusID: 0030:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:58.993922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 4 with properties:
pciBusID: 0031:0c:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:58.995369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 5 with properties:
pciBusID: 0033:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:35:58.995395: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-01-08 08:35:58.995411: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-01-08 08:35:58.995428: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-01-08 08:35:58.995441: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-01-08 08:35:58.995455: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-01-08 08:35:58.995467: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-01-08 08:35:58.995479: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-01-08 08:35:59.012695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0, 1, 2, 3, 4, 5
2020-01-08 08:35:59.024318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-01-08 08:35:59.024333: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0 1 2 3 4 5
2020-01-08 08:35:59.024342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N N N N N N
2020-01-08 08:35:59.024349: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 1: N N N N N N
2020-01-08 08:35:59.024357: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 2: N N N N N N
2020-01-08 08:35:59.024364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 3: N N N N N N
2020-01-08 08:35:59.024372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 4: N N N N N N
2020-01-08 08:35:59.024379: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 5: N N N N N N
2020-01-08 08:35:59.034757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15007 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-01-08 08:35:59.036231: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 15007 MB memory) -> physical GPU (device: 1, name: Tesla T4, pci bus id: 0001:0b:00.0, compute capability: 7.5)
2020-01-08 08:35:59.037698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 15007 MB memory) -> physical GPU (device: 2, name: Tesla T4, pci bus id: 0003:01:00.0, compute capability: 7.5)
2020-01-08 08:35:59.039164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 15007 MB memory) -> physical GPU (device: 3, name: Tesla T4, pci bus id: 0030:01:00.0, compute capability: 7.5)
2020-01-08 08:35:59.040635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 15007 MB memory) -> physical GPU (device: 4, name: Tesla T4, pci bus id: 0031:0c:00.0, compute capability: 7.5)
2020-01-08 08:35:59.042100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:5 with 15007 MB memory) -> physical GPU (device: 5, name: Tesla T4, pci bus id: 0033:01:00.0, compute capability: 7.5)
2020-01-08 08:35:59.713597: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:460] There are 8 ops of 5 different types in the graph that are not converted to TensorRT: ConcatV2, ArgMax, Split, NoOp, Placeholder, (For more information see https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#supported-ops).
2020-01-08 08:35:59.812935: W tensorflow/compiler/tf2tensorrt/segment/segment.cc:707] Segment 0 has multiple devices attached: /device:CPU:0, /device:GPU:0, /device:GPU:1,
2020-01-08 08:35:59.823524: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:636] Number of TensorRT candidate segments: 1
2020-01-08 08:35:59.837142: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:259] Detected multiple (2) devices for the segment. Picking first one to continue.
2020-01-08 08:35:59.886190: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:737] Replaced segment 0 consisting of 667 nodes by TRTEngineOp_0.
2020-01-08 08:36:00.417076: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:814] Optimization results for grappler item: tf_graph
2020-01-08 08:36:00.417142: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] constant_folding: Graph size after: 684 nodes (-267), 984 edges (-267), time = 245.259ms.
2020-01-08 08:36:00.417166: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] layout: Graph size after: 684 nodes (0), 984 edges (0), time = 93.266ms.
2020-01-08 08:36:00.417182: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] constant_folding: Graph size after: 684 nodes (0), 984 edges (0), time = 74.758ms.
2020-01-08 08:36:00.417196: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] TensorRTOptimizer: Graph size after: 18 nodes (-666), 14 edges (-970), time = 306.668ms.
2020-01-08 08:36:00.417205: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] constant_folding: Graph size after: 12 nodes (-6), 14 edges (0), time = 2.272ms.
2020-01-08 08:36:00.417214: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:814] Optimization results for grappler item: TRTEngineOp_0_native_segment
2020-01-08 08:36:00.417226: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] constant_folding: Graph size after: 673 nodes (0), 970 edges (0), time = 83.354ms.
2020-01-08 08:36:00.417235: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] layout: Graph size after: 673 nodes (0), 970 edges (0), time = 106.321ms.
2020-01-08 08:36:00.417249: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] constant_folding: Graph size after: 673 nodes (0), 970 edges (0), time = 86.773ms.
2020-01-08 08:36:00.417258: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] TensorRTOptimizer: Graph size after: 673 nodes (0), 970 edges (0), time = 11.085ms.
2020-01-08 08:36:00.417317: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] constant_folding: Graph size after: 673 nodes (0), 970 edges (0), time = 86.46ms.
2020-01-08 08:36:03.795699: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at trt_engine_resource_ops.cc:183 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0)
I0108 08:36:03.795797 140735937334144 trt_convert.py:1077] Could not find TRTEngineOp_0 in TF-TRT cache. This can happen if build() is not called, which means TensorRT engines will be built and cached at runtime.
I0108 08:36:05.921934 140735937334144 builder_impl.py:775] Assets written to: trt_engine/assets
2020-01-08 08:36:12.086638: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 6
2020-01-08 08:36:12.086742: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-01-08 08:36:12.090304: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:36:12.091767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 1 with properties:
pciBusID: 0001:0b:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:36:12.093217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 2 with properties:
pciBusID: 0003:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:36:12.094668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 3 with properties:
pciBusID: 0030:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:36:12.096126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 4 with properties:
pciBusID: 0031:0c:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:36:12.097580: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 5 with properties:
pciBusID: 0033:01:00.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-01-08 08:36:12.097610: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-01-08 08:36:12.097632: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-01-08 08:36:12.097660: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-01-08 08:36:12.097677: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-01-08 08:36:12.097692: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-01-08 08:36:12.097706: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-01-08 08:36:12.097718: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-01-08 08:36:12.114916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0, 1, 2, 3, 4, 5
2020-01-08 08:36:12.126008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-01-08 08:36:12.126022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0 1 2 3 4 5
2020-01-08 08:36:12.126031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N N N N N N
2020-01-08 08:36:12.126040: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 1: N N N N N N
2020-01-08 08:36:12.126047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 2: N N N N N N
2020-01-08 08:36:12.126055: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 3: N N N N N N
2020-01-08 08:36:12.126062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 4: N N N N N N
2020-01-08 08:36:12.126070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 5: N N N N N N
2020-01-08 08:36:12.136387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15007 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-01-08 08:36:12.137867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 15007 MB memory) -> physical GPU (device: 1, name: Tesla T4, pci bus id: 0001:0b:00.0, compute capability: 7.5)
2020-01-08 08:36:12.139335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 15007 MB memory) -> physical GPU (device: 2, name: Tesla T4, pci bus id: 0003:01:00.0, compute capability: 7.5)
2020-01-08 08:36:12.140810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 15007 MB memory) -> physical GPU (device: 3, name: Tesla T4, pci bus id: 0030:01:00.0, compute capability: 7.5)
2020-01-08 08:36:12.142277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 15007 MB memory) -> physical GPU (device: 4, name: Tesla T4, pci bus id: 0031:0c:00.0, compute capability: 7.5)
2020-01-08 08:36:12.143742: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:5 with 15007 MB memory) -> physical GPU (device: 5, name: Tesla T4, pci bus id: 0033:01:00.0, compute capability: 7.5)
2020-01-08 08:36:13.118166: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:814] Optimization results for grappler item: graph_to_optimize
2020-01-08 08:36:13.118227: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] function_optimizer: Graph size after: 17 nodes (13), 19 edges (16), time = 109.383ms.
2020-01-08 08:36:13.118236: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] function_optimizer: Graph size after: 17 nodes (0), 19 edges (0), time = 97.846ms.
2020-01-08 08:36:13.118245: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:814] Optimization results for grappler item: __inference_TRTEngineOp_0_native_segment_24339
2020-01-08 08:36:13.118253: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] function_optimizer: function_optimizer did nothing. time = 0.002ms.
2020-01-08 08:36:13.118260: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] function_optimizer: function_optimizer did nothing. time = 0.001ms.
Benchmark arguments:
batch_size: 128
calib_data_dir: None
data_dir: .
display_every: 100
gpu_mem_cap: 0
input_saved_model_dir: data/resnet_v1_50/1523293981
input_size: 224
max_workspace_size: 1073741824
minimum_segment_size: 2
mode: benchmark
num_calib_inputs: 500
num_classes: 1001
num_iterations: 100
num_warmup_iterations: 50
optimize_offline: False
output_saved_model_dir: trt_engine
precision: FP16
preprocess_method: vgg
target_duration: None
use_synthetic: True
use_trt: True
TensorRT Conversion Params:
is_dynamic_op: True
max_batch_size: 128
max_workspace_size_bytes: 1073741824
maximum_cached_engines: 1
minimum_segment_size: 2
precision_mode: FP16
rewriter_config_template: None
use_calibration: False
Conversion times:
conversion: 34.5s
2020-01-08 08:36:15.661277: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:736] Building a new TensorRT engine for PartitionedCall/TRTEngineOp_0 with input shapes: [[64,224,224,3], [64,224,224,3]]
2020-01-08 08:36:15.661353: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.7
2020-01-08 08:36:15.662172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.7
2020-01-08 08:36:16.004984: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.005042: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.005056: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.005067: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.005439: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.005452: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.005462: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.005473: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.031483: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.031498: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.031512: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.031524: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.031844: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.031858: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.031871: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.031882: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.059912: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.059926: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.059940: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.059951: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.060226: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.060239: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.060249: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.060262: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.099044: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.099059: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.099071: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.099082: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.099280: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.099292: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:36:16.099304: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with same pad mode
2020-01-08 08:36:16.099315: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Can't fuse pad and convolution with caffe pad mode
2020-01-08 08:37:24.457286: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:38] DefaultLogger Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
Traceback (most recent call last):
File "image_classification.py", line 462, in <module>
target_duration=args.target_duration)
File "image_classification.py", line 256, in run_inference
batch_preds = list(graph_func(batch_images).values())[0].numpy()
AttributeError: 'list' object has no attribute 'values'
Can anyone explain the error or what I should be doing instead?
mankeyboy commented
@pooyadavoodi Any guidance here would be helpful :)
pooyadavoodi commented
Looks like the output of the graph (i.e. graph_func(batch_images)) is a list instead of a tensor. Check how many items are in the list. My guess is that the output tensor is the first item in the list. If that's the case, you can do something like this: batch_preds = graph_func(batch_images)[0].values().numpy()
mankeyboy commented
Thanks @pooyadavoodi . Your suggestion worked and I was able to succesfully run the synthetic test for batch size 128. However, when I tried to run it next for batch size 1, the same line in the code gave this error:
2020-01-08 22:34:20.448568: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 1) and num_split 2
[[{{node PartitionedCall/split_inputs/split}}]]
[[PartitionedCall/split_inputs/split/_2]]
2020-01-08 22:34:20.448681: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 1) and num_split 2
[[{{node PartitionedCall/split_inputs/split}}]]
2020-01-08 22:34:20.449397: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 1) and num_split 2
[[{{node PartitionedCall/split_inputs/split}}]]
[[PartitionedCall/TRTEngineOp_0/_6]]
Traceback (most recent call last):
File "image_classification.py", line 462, in <module>
target_duration=args.target_duration)
File "image_classification.py", line 256, in run_inference
batch_preds = graph_func(batch_images)[0].numpy()
File "/home/mayroy13/anaconda3/envs/trt7-py36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1551, in __call__
return self._call_impl(args, kwargs)
File "/home/mayroy13/anaconda3/envs/trt7-py36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1591, in _call_impl
return self._call_flat(args, self.captured_inputs, cancellation_manager)
File "/home/mayroy13/anaconda3/envs/trt7-py36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1692, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/home/mayroy13/anaconda3/envs/trt7-py36/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 545, in call
ctx=ctx)
File "/home/mayroy13/anaconda3/envs/trt7-py36/lib/python3.6/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 1) and num_split 2
[[node PartitionedCall/split_inputs/split (defined at image_classification.py:134) ]]
[[PartitionedCall/split_inputs/split/_2]]
(1) Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 1) and num_split 2
[[node PartitionedCall/split_inputs/split (defined at image_classification.py:134) ]]
0 successful operations.
1 derived errors ignored. [Op:__inference_pruned_25839]
Function call stack:
pruned -> pruned
Further testing points me to this line in the log where the TRT Engine is built when trying to run with even batch sizes:
Building a new TensorRT engine for PartitionedCall/TRTEngineOp_0 with input shapes: [[1,224,224,3], [1,224,224,3]]
How can we modify this to support odd batch sizes as well?
mankeyboy commented
@pooyadavoodi @aaroey Any ideas for above?
vdevaram commented
use the nvidia tensorflow container nvcr.io/nvidia/tensorflow:20.01-tf2-py3. this is working fine for image classification models although object detection is segmentation fault for NMS.