No improvemenmt in Faster R-CNN after optimizing it with TF-TRT

Question

No improvemenmt in Faster R-CNN after optimizing it with TF-TRT

parthdhameliya opened this issue 5 years ago · 13 comments

I used TF-TRT to optimize the frozen inference graph of Faster R-CNNtrained model. But there is no improvement in inference speed.
Is tensorrt supported with Faster RCNN because in recent presentation from nvidia it is written that Faster RCNN: Coming Soon.
I am using following code to get tensorrt optimized graph:

trt_graph = trt.create_inference_graph( input_graph_def = frozen_graph_def, outputs = output_node_name, max_batch_size=batch_size, max_workspace_size_bytes=workspace_size, precision_mode=precision) # Get optimized graph

Can anybody please help me what is wrong?

Answer 1 · 2019-08-29T22:20:47.000Z

If you use TF1.14, you should get some speedup.

Please make sure you use the same TF-TRT API that is used in the examples in this repository.

Answer 2 · 2019-08-30T11:38:15.000Z

I am using TF1.14
My backbone archiecture in Faster R-CNN is Resnet-101.

My above mentioned code is right or not? to get optimized model.

Answer 3 · 2019-09-04T09:46:12.000Z

Please help me I am stucked in this problem.

Answer 4 · 2019-09-11T05:58:29.000Z

The problem you are facing is that the function you are using has been modified in TF1.14 and isn't available in that form. To get the speedups or just to see TF-TRT being used, clone the master branch of this repository using git clone https://github.com/tensorflow/tensorrt.git

Install tensorflow-gpu using pip install tensorflow-gpu

Install whatever latest TensorRT version is publically available and put the location to it in your $PATH and $LD_LIBRARY_PATH environment variables.

Following that, enter this repository's top level directory and run python setup.py install to setup the dependencies being used inside the Object Detection API.
Do an export PYTHONPATH=$PWD:$PYTHONPATH after that.

The object detection API used in this example says that you can use a .json config file to give the parameters. As an example, my test json for Object Detection using ssd-inceptionv2 is as follows:

{
  "model_config": {
    "model_name": "ssd_inception_v2_coco",
    "output_dir": "models",
    "override_nms_score_threshold": 0.3,
    "batch_size": 1
  },
  "optimization_config": {
    "use_trt": true,
    "precision_mode": "FP16",
    "force_nms_cpu": true,
    "replace_relu6": true,
    "remove_assert": true
  },
  "benchmark_config": {
    "images_dir": "coco/val2017",
    "annotation_path": "coco/annotations/instances_val2017.json",
    "batch_size": 1,
    "num_images": 1000,
    "output_path": "stats/ssd_inception_v2_coco_trt_fp16_b1.json"
  },
  "assertions": [
    "statistics['map'] > (0.268 - 0.005)"
  ]
}

Make sure to have the dataset folders already setup before you begin the run, which in my case was the coco dataset.
From the top level directory in this repo, you should now be able to run object-detection using the following command:
python -m tftrt.examples.object_detection.test ./tftrt/examples/object_detection/mytest_fp16-b1_ssd_inception.json
Here, the mytest_* file is the json file the output of which I put above.

Answer 5 · 2019-09-11T11:05:41.000Z

Thank you @mankeyboy for your detailed answer.
I am using Faster R-CNN so can you please tell me what changes I have to do in above json file to use it for Faster R-CNN. Also I can not find where it is mentioned that you can use json file to give the parameteres.
I switched to TF 1.12 so that above function runs successfully. I also debugged the optimized graph and it is optimizing it becuse number of nodes is decreased to 93 from 143 in original graph but still it is not showing any improvement in speed during inference.
So can you please tell me what can be the problem?

Answer 6 · 2019-09-13T13:17:50.000Z

@parthdhameliya, the reason why I was telling you to shift to TF1.14( or TF1.15 )is because they are the ones with the best support with TensorRT as well as the fact that, the TF API has changed 1.13.1 onwards. There are two things you can do:

Install tf1.14 or 1.15 (with GPU) and use it with TensorRT and the code on the master branch as I mentioned in the comment above.
Shift to the r1.13 branch on this repository and use the code from there. I'll see if I can point you to a .json that might work.

You can check in the code for the functions whose arguments are these json parameters to play around with this:

{
  "source_model": {
    "model_name": "faster_rcnn_resnet50_coco",
    "output_dir": "models"
  },
  "optimization_config": {
    "use_trt": true,
    "precision_mode": "FP16",
    "calib_images_dir": "coco/val2017",
    "num_calib_images": 1000,
    "force_nms_cpu": true,
    "replace_relu6": true,
    "remove_assert": true,
    "override_nms_score_threshold": 0.3,
    "max_batch_size": 1
  },
  "benchmark_config": {
    "images_dir": "coco/val2017",
    "annotation_path": "coco/annotations/instances_val2017.json",
    "batch_size": 1,
    "num_images": 1000,
    "output_path": "stats/ssd_inception_v2_coco_trt_fp32.json"
  },
  "assertions": [
    "statistics['map'] > (0.268 - 0.005)"
  ]
}

Answer 7 · 2019-09-15T14:49:47.000Z

I got it. Thank you @mankeyboy. But here in this jason file it is taking already available model as an input. But I have my own trained model frozen inference graph. So can you please tell me how can I change input so that it takes my model as an input.
I want to only optimize my model I do not want to benchmark it as well as I am using my own model so I do not need Download, Benchmark and Test part from this procedure so How can I use only optimze function and can give input my own model to this function.

Answer 8 · 2019-09-16T03:54:44.000Z

Check test.py inside tensorrt/tftrt/examples/object_detection, the function which reads the .json file is being called from there. You can modify the .json file according to your requirement and for your models.

Answer 9 · 2019-11-15T21:51:39.000Z

I also have this issue. I converted my own faster rcnn model to tensorRT but notice that it is actually 2 times slower than non-optimized model. I'm using tf1.14 as well and it was converted from frozen model btw. Is there any update on this issue?

Answer 10 · 2019-12-18T08:38:29.000Z

I used TF-TRT to optimize the frozen inference graph of Faster R-CNNtrained model. But there is no improvement in inference speed.
Is tensorrt supported with Faster RCNN because in recent presentation from nvidia it is written that Faster RCNN: Coming Soon.
I am using following code to get tensorrt optimized graph:

trt_graph = trt.create_inference_graph( input_graph_def = frozen_graph_def, outputs = output_node_name, max_batch_size=batch_size, max_workspace_size_bytes=workspace_size, precision_mode=precision) # Get optimized graph

Can anybody please help me what is wrong?

@parthdhameliya can you tell me the output_node_name of Faster R-CNN model that you have converted. I am unable to convert the model which is Faster R-CNN inception-v2

Answer 11 · 2019-12-18T18:24:44.000Z

Output nodes are:
outputs=["detection_boxes", "detection_scores", "detection_classes", "num_detections"],

Answer 12 · 2021-01-07T22:37:11.000Z

Hi @mankeyboy, could you please assist?. I am trying to optimize the model faster_rcnn_inception_v2 with INT8 & NMS but getting some errors at the calibration step, I am running the test.py sample within the docker image nvcr.io/nvidia/tensorflow:20.01-tf1-py3

Here are the json parameters:

{
  "model_config": {
    "model_name": "faster_rcnn_inception_v2",
    "input_dir": "/workspace/",
    "batch_size": 8,
    "override_nms_score_threshold": 0.3
  },
  "optimization_config": {
    "use_trt": true,
    "precision_mode": "INT8",
    "force_nms_cpu": true,    
    "calib_images_dir": "/workspace/data/coco/val2017/",
    "num_calib_images": 16,
    "calib_batch_size": 8,
    "calib_image_shape": [640, 640],
    "max_workspace_size_bytes": 17179869184
  },
  "benchmark_config": {
    "images_dir": "/workspace/data/coco/val2017/",
    "annotation_path": "/workspace/data/coco/annotations/instances_val2017.json",
    "batch_size": 8,
    "image_shape": [640, 640],
    "num_images": 4096,
    "output_path": "stats/faster_rcnn_inception_v2_tf-trt_nms_int8.json"
  },
  "assertions": [
    "statistics['map'] > (0.277 - 0.01)"
  ]
}

Tracelog:

2021-01-07 22:13:55.825262: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7d3c015bd0
2021-01-07 22:13:55.846833: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.849049: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.857768: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in in (110) - 110) - AS Error in initializeCommonContextas, : BLAS Error in initializeCommonContext: 3 (C(Could not initialize cublas, please check cuda installatio).)
2021-01-07 22:13:55.857768: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in in (110) - 110) - AS Error in initializeCommonContextas, : BLAS Error in initializeCommonContext: 3 (C(Could not initialize cublas, please check cuda installatio).)
2021-01-07 22:13:55.857850: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.857896: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.858605: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f8204018790
2021-01-07 22:13:55.878576: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.878722: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.878784: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.879111: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7f640055c0
2021-01-07 22:13:55.880486: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.880635: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.880708: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.881157: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f81c800b9e0
2021-01-07 22:13:55.885218: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.885335: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.885410: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.885923: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7fe0004610
2021-01-07 22:13:55.894804: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f8170004d20
2021-01-07 22:13:55.898693: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.902438: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 1 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.902524: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 1 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.902585: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.903016: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7c7c01ed10
2021-01-07 22:13:55.906085: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.906168: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.906220: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.906510: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7d6c009e10
2021-01-07 22:13:55.923378: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.923486: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.923560: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.924005: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7d300015a0
2021-01-07 22:13:55.925104: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 1 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.927049: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.927106: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.927129: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.927732: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7c7c00a0c0
2021-01-07 22:13:55.929087: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.929172: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.929212: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.929496: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7d6c007300
2021-01-07 22:13:55.931367: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.931456: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.931508: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.931783: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7fe0004920
2021-01-07 22:13:55.933726: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.936154: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.939269: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.939350: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.939400: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.939733: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7cdc017160
2021-01-07 22:13:55.941469: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.941531: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.941575: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.942001: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f804c007710
2021-01-07 22:13:55.944043: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.946233: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:55.954102: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.954183: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.954230: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.954519: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7d30001f30
2021-01-07 22:13:55.956840: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.956919: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:55.956965: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:55.957254: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7c7c00a020
2021-01-07 22:13:56.002298: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 1 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:56.010513: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 1 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:56.012796: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.012902: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:56.013335: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f8064014650
2021-01-07 22:13:56.014969: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 1 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:56.015038: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 1 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:56.015090: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:56.015669: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7d9c015930
2021-01-07 22:13:56.025013: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.049104: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.049194: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.049251: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:56.049960: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7e8c0181a0
2021-01-07 22:13:56.050525: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.052193: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.052259: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.052311: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:56.052678: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7d0c003700
2021-01-07 22:13:56.053798: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.055345: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 1 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:56.055416: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 1 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:56.055469: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:56.055807: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7fb80070a0
2021-01-07 22:13:56.057777: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.057916: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.057988: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:56.058546: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f8134005810
2021-01-07 22:13:56.059945: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.060005: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.060048: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:56.060455: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7e44006260
2021-01-07 22:13:56.062067: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 1 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:56.062130: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 1 (Could not initialize cudnn, please check cudnn installation.)
2021-01-07 22:13:56.062179: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:56.062829: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7fac0090a0
2021-01-07 22:13:56.065062: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.065118: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger FAILED_ALLOCATION: std::exception
2021-01-07 22:13:56.065130: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger Context could not be created.
2021-01-07 22:13:56.065442: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger Calibration failure occurred with no scaling factors detected. This could be due to no int8 calibrator or insufficient custom scales for network layers. Please see int8 sample to setup calibration correctly.
2021-01-07 22:13:56.065476: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger Builder failed while configuring INT8 mode.
2021-01-07 22:13:56.065528: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2021-01-07 22:13:56.065861: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f7da8004380
2021-01-07 22:13:56.068435: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2021-01-07 22:13:56.071109: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)

Update: Solved #230

Answer 13 · 2021-01-08T00:17:56.000Z

@bixia1 : could this be related to the issue you described ?