tensorflow/tensorrt

Invalid argument: Unsupported data type encountered in input 0 in mask_rcnn_resnet50_atrous_coco.

Closed this issue · 3 comments

sgambient commented

Getting the above error on all the RCNN networks which are supposed to have been tested and woked. The full stack trace follows. Is there a to prevent this error ?

2019-08-19 06:47:33.758161: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at trt_engine_op.cc:293 : Invalid argument: Unsupported data type encountered in input 0
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "build/bdist.linux-x86_64/egg/tftrt/examples/object_detection/test.py", line 105, in <module>
  File "build/bdist.linux-x86_64/egg/tftrt/examples/object_detection/test.py", line 76, in test
  File "build/bdist.linux-x86_64/egg/tftrt/examples/object_detection/object_detection.py", line 453, in optimize_model
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/compiler/tensorrt/trt_convert.py", line 353, in calibrate
    fetches, feed_dict=feed_dict_fn() if feed_dict_fn else None)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: Unsupported data type encountered in input 0
         [[node FirstStageFeatureExtractor/resnet_v1_50/resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/TRTEngineOp_2239 (defined at build/bdist.linux-x86_64/egg/tftrt/examples/object_detection/object_det$ction.py:453) ]]
         [[BatchMultiClassNonMaxSuppression_1/MultiClassNonMaxSuppression_5/TRTEngineOp_1500/_7959]]
  (1) Invalid argument: Unsupported data type encountered in input 0
         [[node FirstStageFeatureExtractor/resnet_v1_50/resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/TRTEngineOp_2239 (defined at build/bdist.linux-x86_64/egg/tftrt/examples/object_detection/object_det$ction.py:453) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for u'FirstStageFeatureExtractor/resnet_v1_50/resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/TRTEngineOp_2239':
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "build/bdist.linux-x86_64/egg/tftrt/examples/object_detection/test.py", line 105, in <module>
  File "build/bdist.linux-x86_64/egg/tftrt/examples/object_detection/test.py", line 76, in test
  File "build/bdist.linux-x86_64/egg/tftrt/examples/object_detection/object_detection.py", line 453, in optimize_model
    feed_dict_fn=feed_dict_fn)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/compiler/tensorrt/trt_convert.py", line 347, in calibrate
    name="")
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 443, in import_graph_def
    _ProcessNewOps(graph)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 236, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3751, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3641, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()
sgambient commented

This was on the nvcr Tensorflow container for Python 2. The other detectors seem to work.

pooyadavoodi commented

What version of TF or TF container are you using?

This model works for me using the following config:

{
  "model_config": {
    "model_name": "mask_rcnn_resnet50_atrous_coco",
    "input_dir": "/data/tensorflow/object_detection/models",
    "config_path": "mask_rcnn_resnet50_atrous_trt_fp16_pipeline.config",
    "batch_size": 1,
    "override_nms_score_threshold": 0.3 
  },  
  "optimization_config": {
    "use_trt": true,
    "precision_mode": "FP16",
    "force_nms_cpu": false,
    "max_workspace_size_bytes": 17179869184,
    "replace_relu6": false,
    "remove_assert": false
  },  
  "benchmark_config": {
    "images_dir": "/data/coco-2017/coco2017/val2017",
    "annotation_path": "/data/coco-2017/coco2017/annotations/instances_val2017.json",
    "batch_size": 1,
    "image_shape": [
      640,
      640 
    ],  
    "num_images": 4096,
    "output_path": "stats/mask_rcnn_resnet50_atrous_coco_trt_fp16.json"
  },  
  "assertions": [
    "statistics['map'] > (0.279 - 0.005)"
  ]
}
pooyadavoodi commented

Closing. Please feel free to reopen if you still see the problem.