Can't export Motion Deeplab trained on MOTChallenge

[fschvart]

I trained a Motion Deeplab using the MOTChallenge according to the instructions. The training process seems to have run fine (sidenote: is it reasonable that I could only use batch size = 1 due to CUDA OOM error? I use RTX 3090 with 24gb of memory?)
I trained using the --use_two_frames for the dataset preparation.

Here's the full log and error message I get:

2022-09-02 18:36:39.055095: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-09-02 18:36:40.055893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21670 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:02:00.0, compute capability: 8.6
2022-09-02 18:36:40.057630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 21670 MB memory: -> device: 1, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:21:00.0, compute capability: 8.6
I0902 18:36:40.674806 8796 motion_deeplab.py:53] Synchronized Batchnorm is used.
I0902 18:36:40.675889 8796 axial_resnet_instances.py:144] Axial-ResNet final config: {'num_blocks': [3, 4, 6, 3], 'backbone_layer_multiplier': 1.0, 'width_multiplier': 1.0, 'stem_width_multiplier': 1.0, 'output_stride': 16, 'classification_mode': True, 'backbone_type': 'resnet', 'use_axial_beyond_stride': 0, 'backbone_use_transformer_beyond_stride': 0, 'extra_decoder_use_transformer_beyond_stride': 32, 'backbone_decoder_num_stacks': 0, 'backbone_decoder_blocks_per_stage': 1, 'extra_decoder_num_stacks': 0, 'extra_decoder_blocks_per_stage': 1, 'max_num_mask_slots': 128, 'num_mask_slots': 128, 'memory_channels': 256, 'base_transformer_expansion': 1.0, 'global_feed_forward_network_channels': 256, 'high_resolution_output_stride': 4, 'activation': 'relu', 'block_group_config': {'attention_bottleneck_expansion': 2, 'drop_path_keep_prob': 1.0, 'drop_path_beyond_stride': 16, 'drop_path_schedule': 'constant', 'positional_encoding_type': None, 'use_global_beyond_stride': 0, 'use_sac_beyond_stride': -1, 'use_squeeze_and_excite': False, 'conv_use_recompute_grad': False, 'axial_use_recompute_grad': True, 'recompute_within_stride': 0, 'transformer_use_recompute_grad': False, 'axial_layer_config': {'query_shape': (129, 129), 'key_expansion': 1, 'value_expansion': 2, 'memory_flange': (32, 32), 'double_global_attention': False, 'num_heads': 8, 'use_query_rpe_similarity': True, 'use_key_rpe_similarity': True, 'use_content_similarity': True, 'retrieve_value_rpe': True, 'retrieve_value_content': True, 'initialization_std_for_query_key_rpe': 1.0, 'initialization_std_for_value_rpe': 1.0, 'self_attention_activation': 'softmax'}, 'dual_path_transformer_layer_config': {'num_heads': 8, 'bottleneck_expansion': 2, 'key_expansion': 1, 'value_expansion': 2, 'feed_forward_network_channels': 2048, 'use_memory_self_attention': True, 'use_pixel2memory_feedback_attention': True, 'transformer_activation': 'softmax'}}, 'bn_layer': functools.partial(<class 'keras.layers.normalization.batch_normalization.SyncBatchNormalization'>, momentum=0.9900000095367432, epsilon=0.0010000000474974513), 'conv_kernel_weight_decay': 0.0}
I0902 18:36:40.896352 8796 motion_deeplab.py:109] Setting pooling size to (68, 121)
I0902 18:36:40.897225 8796 aspp.py:141] Global average pooling in the ASPP pooling layer was replaced with tiled average pooling using the provided pool_size. Please make sure this behavior is intended.
I0902 18:36:40.898226 8796 aspp.py:141] Global average pooling in the ASPP pooling layer was replaced with tiled average pooling using the provided pool_size. Please make sure this behavior is intended.
WARNING:tensorflow:Skipping full serialization of Keras layer <deeplab2.model.layers.resized_fuse.ResizedFuse object at 0x0000021D913410D0>, because it is not built.
W0902 18:36:57.138313 8796 save_impl.py:71] Skipping full serialization of Keras layer <deeplab2.model.layers.resized_fuse.ResizedFuse object at 0x0000021D913410D0>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <deeplab2.model.layers.resized_fuse.ResizedFuse object at 0x0000021D8F3805B0>, because it is not built.
W0902 18:36:57.139913 8796 save_impl.py:71] Skipping full serialization of Keras layer <deeplab2.model.layers.resized_fuse.ResizedFuse object at 0x0000021D8F3805B0>, because it is not built.
Traceback (most recent call last):
File "C:\deeplab2\export_model.py", line 157, in
app.run(main)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\absl\app.py", line 308, in run
_run_main(main, args)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\absl\app.py", line 254, in _run_main
sys.exit(main(argv))
File "C:\deeplab2\export_model.py", line 152, in main
tf.saved_model.save(
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\saved_model\save.py", line 1290, in save
save_and_return_nodes(obj, export_dir, signatures, options)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\saved_model\save.py", line 1325, in save_and_return_nodes
_build_meta_graph(obj, signatures, options, meta_graph_def))
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\saved_model\save.py", line 1491, in _build_meta_graph
return _build_meta_graph_impl(obj, signatures, options, meta_graph_def)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\saved_model\save.py", line 1443, in _build_meta_graph_impl
saveable_view = _SaveableView(augmented_graph_view, options)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\saved_model\save.py", line 229, in init
self.augmented_graph_view.objects_ids_and_slot_variables_and_paths())
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\training\tracking\graph_view.py", line 544, in objects_ids_and_slot_variables_and_paths
trackable_objects, node_paths = self._breadth_first_traversal()
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\training\tracking\graph_view.py", line 255, in _breadth_first_traversal
for name, dependency in self.list_children(current_trackable):
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\saved_model\save.py", line 143, in list_children
for name, child in super(_AugmentedGraphView, self).list_children(
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\training\tracking\graph_view.py", line 203, in list_children
in obj._trackable_children(save_type, **kwargs).items()]
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\engine\training.py", line 3201, in _trackable_children
children = super(Model, self)._trackable_children(save_type, **kwargs)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\engine\base_layer.py", line 3174, in _trackable_children
children = self._trackable_saved_model_saver.trackable_children(cache)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\base_serialization.py", line 59, in trackable_children
children = self.objects_to_serialize(serialization_cache)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\layer_serialization.py", line 68, in objects_to_serialize
return (self._get_serialized_attributes(
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\layer_serialization.py", line 88, in _get_serialized_attributes
object_dict, function_dict = self._get_serialized_attributes_internal(
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\model_serialization.py", line 56, in _get_serialized_attributes_internal
super(ModelSavedModelSaver, self)._get_serialized_attributes_internal(
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\layer_serialization.py", line 98, in _get_serialized_attributes_internal
functions = save_impl.wrap_layer_functions(self.obj, serialization_cache)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\save_impl.py", line 197, in wrap_layer_functions
fn.get_concrete_function()
File "C:\Users\FabianDual3\miniconda3\lib\contextlib.py", line 126, in exit
next(self.gen)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\save_impl.py", line 359, in tracing_scope
fn.get_concrete_function(*args, **kwargs)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 1239, in get_concrete_function
concrete = self._get_concrete_function_garbage_collected(*args, **kwargs)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 1219, in _get_concrete_function_garbage_collected
self._initialize(args, kwargs, add_initializers_to=initializers)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 785, in _initialize
self._stateful_fn._get_concrete_function_internal_garbage_collected( # pylint: disable=protected-access
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\eager\function.py", line 2480, in _get_concrete_function_internal_garbage_collected
graph_function, _ = self._maybe_define_function(args, kwargs)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\eager\function.py", line 2711, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\eager\function.py", line 2627, in _create_graph_function
func_graph_module.func_graph_from_py_func(
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\framework\func_graph.py", line 1141, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 677, in wrapped_fn
out = weak_wrapped_fn().wrapped(*args, **kwds)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\save_impl.py", line 572, in wrapper
ret = method(*args, **kwargs)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\utils.py", line 168, in wrap_with_training_arg
return control_flow_util.smart_cond(
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\utils\control_flow_util.py", line 105, in smart_cond
return tf.internal.smart_cond.smart_cond(
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\framework\smart_cond.py", line 55, in smart_cond
return false_fn()
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\utils.py", line 170, in
lambda: replace_training_and_call(False))
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\utils.py", line 166, in replace_training_and_call
return wrapped_call(*args, **kwargs)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\save_impl.py", line 652, in call
return call_and_return_conditional_losses(inputs, *args, **kwargs)[0]
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\save_impl.py", line 610, in call
return self.wrapped_call(*args, **kwargs)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\tensorflow\python\util\traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\save_impl.py", line 572, in wrapper
ret = method(*args, **kwargs)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\utils.py", line 168, in wrap_with_training_arg
return control_flow_util.smart_cond(
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\utils\control_flow_util.py", line 105, in smart_cond
return tf.internal.smart_cond.smart_cond(
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\utils.py", line 170, in
lambda: replace_training_and_call(False))
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\utils.py", line 166, in replace_training_and_call
return wrapped_call(*args, **kwargs)
File "C:\Users\FabianDual3\miniconda3\lib\site-packages\keras\saving\saved_model\save_impl.py", line 634, in call_and_return_conditional_losses
call_output = layer_call(*args, **kwargs)
File "c:\deeplab2\video\motion_deeplab.py", line 128, in call
input_tensor = self._add_previous_heatmap_to_input(input_tensor)
File "c:\deeplab2\video\motion_deeplab.py", line 184, in _add_previous_heatmap_to_input
if tf.reduce_all(tf.equal(frame1, frame2)):
tensorflow.python.framework.errors_impl.OperatorNotAllowedInGraphError: Using a symbolic tf.Tensor as a Python bool is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.

[markweberdev]

Hi @fschvart,

at the moment, our codebase is not optimised for GPU training. Thus, I'm unable to help you out regarding the increasing of batch size and figuring out to improve memory management. Something that you might wanna look into, is running our codebase not in graph mode but in eager mode. However, that might require some work on your end.

I have never tried to export the model, but judging from your error message, you might wanna check out https://www.tensorflow.org/api_docs/python/tf/cond .