Failed to build TensorRTengine

Question

Failed to build TensorRTengine

Closed this issue 5 years ago · 5 comments

Description

Hi，I have used the following code to transform my saved model with TensorRT in TensorFlow 1.14:

from tensorflow.python.compiler.tensorrt import trt_convert as trt
converter = trt.TrtGraphConverter(input_saved_model_dir=input_saved_model_dir)
converter.convert()
converter.save(output_saved_model_dir)

after finished it, I used the following code to inference:

with tf.Session() as sess:
# First load the SavedModel into the session
tf.saved_model.loader.load(
sess, [tf.saved_model.tag_constants.SERVING], output_saved_model_dir)
output = sess.run([output_tensor], feed_dict={input_tensor: input_data})

Although I get the right output, I received the following warning message:

2019-11-01 19:11:24.404448: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:647] Engine creation for TRTEngineOp_22 failed. The native segment will be used instead. Reason: Internal: Failed to build TensorRT engine
2019-11-01 19:11:24.406408: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:632] Building a new TensorRT engine for TRTEngineOp_21 input shapes: [[1,128], [1,128]]
2019-11-01 19:11:24.406463: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:632] Building a new TensorRT engine for TRTEngineOp_23 input shapes: [[1,128], [1,128]]
2019-11-01 19:11:24.408900: W tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for (Unnamed Layer* 2) [Fully Connected]_output. This is okay if TensorRT does not need the range (e.g. due to node fusion).
2019-11-01 19:11:24.408921: W tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for (Unnamed Layer* 5) [Scale]_output. This is okay if TensorRT does not need the range (e.g. due to node fusion).
2019-11-01 19:11:24.408930: W tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for model/inference/post_cbhg/bidirectional_rnn/bw/bw/while/gru_cell/BiasAdd_1. This is okay if TensorRT does not need the range (e.g. due to node fusion).
2019-11-01 19:11:24.408938: W tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for (Unnamed Layer* 1) [Shuffle]_output. This is okay if TensorRT does not need the range (e.g. due to node fusion).
2019-11-01 19:11:24.408946: W tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for (Unnamed Layer* 4) [Shuffle]_output. This is okay if TensorRT does not need the range (e.g. due to node fusion).

And the cost time is the same as the origin model. I wonder if TensorRT really worked?

Besides, I also tried FP32 FP16 model, the cost time do not changed.

Environment

TensorRT Version: 5.1.5.0
GPU Type: Tesla T4
Nvidia Driver Version: 418.87.01
CUDA Version: 10.1.243
CUDNN Version: 7.6.3
Python Version (if applicable): 3.7.4
TensorFlow Version (if applicable):1.14.1
PyTorch Version (if applicable):
Operating System + Version: Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-142-generic x86_64)
Baremetal or Container (if container which image + tag):

Answer 1 · 2019-11-03T23:08:18.000Z

Hi @MachineJeff,

This seems more like a question for TF-TRT folks (I see you already posted there: tensorflow/tensorrt#148), but I found a related issue here: triton-inference-server/server#48

It looks to me like the error may be a mismatch in your configuration of CUDA/CUDNN/TensorRT.

The simplest way to verify that would be to try using a container instead, our container on NGC nvcr.io/nvidia/tensorflow:19.10-py3 comes with TF 1.14 and TensorRT 6.0.1 as seen in the release notes here: https://docs.nvidia.com/deeplearning/frameworks/tensorflow-release-notes/rel_19.10.html#rel_19.10

While I recommend using a container, if you really don't want to, you can try upgrading your TensorRT source to 6.0 from 5.1.5 and see if that fixes your issue instead.

Answer 2 · 2019-11-04T08:04:08.000Z

@rmccorm4 Hi
Here is my test, I use container nvcr.io/nvidia/tensorflow:19.09-py3 to change my savedmodel with INT8, following code :

from tensorflow.python.compiler.tensorrt import trt_convert as trt
input_saved_model_dir = '/mnt/yichao.li/tts/jingjing/100'
output_saved_model_dir = '/mnt/yichao.li/tts/jingjing/6'

converter = trt.TrtGraphConverter(
	input_saved_model_dir=input_saved_model_dir,
	is_dynamic_op = True,
	precision_mode = 'INT8',
	maximum_cached_engines = 10000,
    use_calibration = False)
converter.convert()
converter.save(output_saved_model_dir)

I deployed it in tf serving, and I received the following meassage:

external/org_tensorflow/tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:632] Building a new TensorRT engine for TRTEngineOp_0 input shapes: [[1,168,512]]
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_1_encoder_convolutions/conv1d/conv1d. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for (Unnamed ITensor* 6). This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_1_encoder_convolutions/conv1d/conv1d/ExpandDims. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_1_encoder_convolutions/conv1d/conv1d-0-TransposeNHWCToNCHW-LayoutOptimizer. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_1_encoder_convolutions/conv1d/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_1_encoder_convolutions/conv1d/conv1d/Squeeze. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for (Unnamed ITensor* 7). This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_1_encoder_convolutions/conv1d/BiasAdd. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_1_encoder_convolutions/batch_normalization/batchnorm/mul_1. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_2_encoder_convolutions/conv1d/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_1_encoder_convolutions/dropout_conv_layer_1_encoder_convolutions/Identity. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_1_encoder_convolutions/batch_normalization/batchnorm/add_1. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_2_encoder_convolutions/conv1d/conv1d-0-TransposeNHWCToNCHW-LayoutOptimizer. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_2_encoder_convolutions/conv1d/conv1d/ExpandDims. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_2_encoder_convolutions/conv1d/conv1d/Squeeze. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for (Unnamed ITensor* 19). This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_2_encoder_convolutions/conv1d/conv1d. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_3_encoder_convolutions/conv1d/BiasAdd. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for (Unnamed ITensor* 20). This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_2_encoder_convolutions/conv1d/BiasAdd. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_2_encoder_convolutions/dropout_conv_layer_2_encoder_convolutions/Identity. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_2_encoder_convolutions/batch_normalization/batchnorm/mul_1. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_3_encoder_convolutions/conv1d/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_2_encoder_convolutions/batch_normalization/batchnorm/add_1. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_3_encoder_convolutions/conv1d/conv1d-0-TransposeNHWCToNCHW-LayoutOptimizer. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_3_encoder_convolutions/conv1d/conv1d/ExpandDims. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for (Unnamed ITensor* 33). This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for (Unnamed ITensor* 32). This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_3_encoder_convolutions/conv1d/conv1d. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_3_encoder_convolutions/conv1d/conv1d/Squeeze. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_3_encoder_convolutions/batch_normalization/batchnorm/mul_1. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for TensorRTOutputPH_0. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for Tacotron_model/inference/encoder_convolutions/conv_layer_3_encoder_convolutions/batch_normalization/batchnorm/add_1. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1467] Quantization range was not found for TensorRTInputPH_0. This is okay if TensorRT does not need the range (e.g. due to node fusion).
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Int8 mode specified but no calibrator specified. Please ensure that you supply Int8 scales for the network layers manually.
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../builder/cudnnBuilder2.cpp (1508) - Misc Error in buildEngine: -1 (Could not find tensor Tacotron_model/inference/encoder_convolutions/conv_layer_1_encoder_convolutions/conv1d/conv1d-0-TransposeNHWCToNCHW-LayoutOptimizer in tensorScales.)
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../builder/cudnnBuilder2.cpp (1508) - Misc Error in buildEngine: -1 (Could not find tensor Tacotron_model/inference/encoder_convolutions/conv_layer_1_encoder_convolutions/conv1d/conv1d-0-TransposeNHWCToNCHW-LayoutOptimizer in tensorScales.)
external/org_tensorflow/tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:647] Engine creation for TRTEngineOp_0 failed. The native segment will be used instead. Reason: Internal: Failed to build TensorRT engine.

that means TensorRT do not work, right? or I ignored something in convertion?

Answer 3 · 2019-11-04T12:39:46.000Z

@rmccorm4 maybe I get the real error point
I use the following code to change the savedmodel with INT8:

converter = trt.TrtGraphConverter(
    input_saved_model_dir=input_saved_model_dir,
    is_dynamic_op = True,
    max_batch_size = 100,
    precision_mode = 'INT8',
    nodes_blacklist = ['model/inference/dense/BiasAdd'],
    maximum_cached_engines = 10000,
    use_calibration = True)
converter.convert()

converter.calibrate(
    fetch_names = ['model/inference/dense/BiasAdd'],
    num_runs = 1,
    feed_dict_fn = feed_dict_fn)

converter.save(output_saved_model_dir)

The convertion is no problem:

2019-11-04 12:26:40.096372: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:752] Optimization results for grappler item: tf_graph
2019-11-04 12:26:40.096420: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:754]   constant folding: Graph size after: 10453 nodes (-5205), 13656 edges (-6121), time = 1580.97803ms.
2019-11-04 12:26:40.096431: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:754]   layout: Graph size after: 10515 nodes (62), 13776 edges (120), time = 431.526ms.
2019-11-04 12:26:40.096438: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:754]   constant folding: Graph size after: 10515 nodes (0), 13776 edges (0), time = 704.257ms.
2019-11-04 12:26:40.096445: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:754]   TensorRTOptimizer: Graph size after: 9908 nodes (-607), 13071 edges (-705), time = 1929.90796ms.

But the calibration got some error:

2019-11-04 12:26:48.726835: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:716] Starting calibration thread on device 0, Calibration Resource @ 0x7fa8fc070bf0
2019-11-04 12:26:48.728281: E tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1561] Node model/inference/encoder_cbhg/bidirectional_rnn/bw/bw/while/gru_cell/concat_1 should have an input named 'model/inference/encoder_cbhg/bidirectional_rnn/bw/bw/while/gru_cell/mul' but it is not available
2019-11-04 12:26:48.728317: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:739] Calibration failed: Invalid argument: Node model/inference/encoder_cbhg/bidirectional_rnn/bw/bw/while/gru_cell/concat_1 should have an input named 'model/inference/encoder_cbhg/bidirectional_rnn/bw/bw/while/gru_cell/mul' but it is not available
2019-11-04 12:26:48.728381: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:716] Starting calibration thread on device 0, Calibration Resource @ 0x7fa910005030
2019-11-04 12:26:48.729420: E tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1561] Node model/inference/encoder_cbhg/bidirectional_rnn/fw/fw/while/gru_cell/concat_1 should have an input named 'model/inference/encoder_cbhg/bidirectional_rnn/fw/fw/while/gru_cell/mul' but it is not available
2019-11-04 12:26:48.729451: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:739] Calibration failed: Invalid argument: Node model/inference/encoder_cbhg/bidirectional_rnn/fw/fw/while/gru_cell/concat_1 should have an input named 'model/inference/encoder_cbhg/bidirectional_rnn/fw/fw/while/gru_cell/mul' but it is not available

the key is:

Node ... should have an input named ... but it is not available

What happened? I am sure my model can run without error.

Answer 4 · 2019-11-05T00:58:54.000Z

Hi @MachineJeff,

I think you're on the right track, but unfortunately that's more TF-TRT specific than I'm familiar with. 🙁

I guess you'll have to wait for an answer from them in your other issue: tensorflow/tensorrt#148

Answer 5 · 2019-11-05T01:52:15.000Z

@rmccorm4 Thanks your reply and watching 👍

GOOD LUCK TO ME : )