Tensorflow TensorRT mismatch

[maciejskorski]

It appears that Tensorflow TensorRT might have been wrongly linked. On few recent images (including gcr.io/kaggle-gpu-images/python latest 311277776c9b 7 days ago 47.2GB) I see very different linked and loaded TensorRT libs, namely 8.4 vs 8.6.

import tensorflow.compiler as tf_cc
linked_trt_ver=tf_cc.tf2tensorrt._pywrap_py_utils.get_linked_tensorrt_version()
print(f"Linked TRT ver: {linked_trt_ver}")
loaded_trt_ver=tf_cc.tf2tensorrt._pywrap_py_utils.get_loaded_tensorrt_version()
print(f"Loaded TRT ver: {loaded_trt_ver}")
# Linked TRT ver: (8, 4, 3)
# Loaded TRT ver: (8, 6, 1)

This has been Python. Now, the system inference libraries are indeed at 8.6:

dpkg -l | grep TensorRT

Under these recent containers, minimal TensorRT samples crash, for instance this one:

import tensorflow as tf
from tensorflow.python.compiler.tensorrt import trt_convert as trt
from tensorflow.python.saved_model import signature_constants
from tensorflow.python.saved_model import tag_constants

from tensorflow.keras.applications.resnet50 import ResNet50
tf_model_dir = './models/tf_model'
model = ResNet50(include_top=50, weights='imagenet')
model.save(tf_model_dir)

converter = trt.TrtGraphConverterV2(  
   input_saved_model_dir=tf_model_dir,
)
converter.convert()

MAX_BATCH_SIZE=1 
def input_fn():
   img = tf.random.normal((MAX_BATCH_SIZE, 224,224,3),dtype=tf.float32)
   return (img, )

import faulthandler
faulthandler.enable()
converter.build(input_fn=input_fn) #SEGMENTATION FAULT can happen under missconfigured software!