tensorflow/tensorrt

Using `tensorflow tensort rt integration failed in some cases` - `Subshape must have computed start`

oak-tree opened this issue · 2 comments

oak-tree commented

Error

Invalid argument: Subshape must have computed start >= end since stride is negative, but is 0 and 2 (computed from start 0 and end 9223372036854775807 over shape with rank 2 and stride-1)

What did we do

We tried to run tensortrt convertor from tensorflow on a specific network

Reproduce

  1. Get the model from https://github.com/qqwweee/keras-yolo3
    create model for predict.
  2. Freeze it
  3. Optimize it with tensortrt integration from inside tensorflow (does not matter if you do not have weights).
  4. The result with the error above.

Note that if you remove the yolo_eval it does succeed to run the optimizer

System

Jetxon Xavier:
Tensorflow: 1.14.0
Jetpack : 4.2.1
Ubuntu: 18.04

EDIT

My 1st guess that it relates to dynamic shape calculation. The yolo_eval uses a second input. This 2nd input is for the original image_shape. This values help yolo_eval to reconstruct the boxes.
Note: We do use the dynamic flag for tensorrt

EDIT 2

We have found the problematic lines
https://github.com/qqwweee/keras-yolo3/blob/e6598d13c703029b2686bc2eb8d5c09badf42992/yolo3/model.py#L140
https://github.com/qqwweee/keras-yolo3/blob/e6598d13c703029b2686bc2eb8d5c09badf42992/yolo3/model.py#L141

Those line tries to reverse grid_shape with negative stride.
i.e:

box_xy = (K.sigmoid(feats[..., :2]) + grid) / K.cast(grid_shape[::-1], K.dtype(feats))
box_wh = K.exp(feats[..., 2:4]) * anchors_tensor / K.cast(input_shape[::-1], K.dtype(feats))

For some reason tensor rt inside tensorflow failed to do so this reverse.

A workaround for that will be

grid_shape[...,::-1]
input_shape[...,::-1]
pooyadavoodi commented

Closing. Please reopen in case you still see the issue.