Issue with Keras / function mae

Question

Issue with Keras / function mae

Opened this issue 5 months ago · 1 comments

Hi,
I'm trying to help some users use the isonet.py program. We are running on a Rocky Linux 9 platform (RHEL linux 9 clone). The user is trying to run a refine job and we're seeing errors like the ones below. In a naive google search looking at the issue it seems that some of the loss functions need to be serialized unless I should be using a different package version (below is the full list of packages in the conda environment).
Please advise,
Jeff

$ isonet.py refine ./subtomo.star --gpuID 0 --preprocessing_ncpus 16 --iterations 4 --noise_start_iter 10,15,20,25 --noise_level 0.05,0.1,0.15,0.2
07-10 15:44:31, INFO
######Isonet starts refining######

07-10 15:44:34, INFO Start Iteration1!
07-10 15:44:34, WARNING The results folder already exists
The old results folder will be renamed (to results~)
/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument alpha is deprecated. Use negative_slope instead.
warnings.warn(
07-10 15:44:36, WARNING You are saving your model as an HDF5 file via model.save() or keras.saving.save_model(model). This file format is considered legacy. We recommend using instead the native Keras format, e.g. model.save('my_model.keras') or keras.saving.save_model(model, 'my_model.keras').
07-10 15:44:39, INFO Noise Level:0.0
07-10 15:45:22, INFO Done preparing subtomograms!
07-10 15:45:22, INFO Start training!
07-10 15:45:23, ERROR Traceback (most recent call last):
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/IsoNet/bin/refine.py", line 128, in run
history = train_data(args) #train based on init model and save new one as model_iter{num_iter}.h5
^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/IsoNet/models/unet/train.py", line 93, in train_data
history = train3D_continue('{}/model_iter{:0>2d}.h5'.format(settings.result_dir,settings.iter_count),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/IsoNet/models/unet/train.py", line 38, in train3D_continue
model = load_model( model_file)
^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/saving/saving_api.py", line 189, in load_model
return legacy_h5_format.load_model_from_hdf5(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/legacy/saving/legacy_h5_format.py", line 155, in load_model_from_hdf5
**saving_utils.compile_args_from_training_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/legacy/saving/saving_utils.py", line 143, in compile_args_from_training_config
loss = _deserialize_nested_config(losses.deserialize, loss_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/legacy/saving/saving_utils.py", line 202, in _deserialize_nested_config
return deserialize_fn(config)
^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/losses/init.py", line 149, in deserialize
return serialization_lib.deserialize_keras_object(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/saving/serialization_lib.py", line 575, in deserialize_keras_object
return deserialize_keras_object(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/saving/serialization_lib.py", line 678, in deserialize_keras_object
return _retrieve_class_or_fn(
^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/saving/serialization_lib.py", line 812, in _retrieve_class_or_fn
raise TypeError(
TypeError: Could not locate function 'mae'. Make sure custom classes are decorated with @keras.saving.register_keras_serializable(). Full object config: {'module': 'keras.metrics', 'class_name': 'function', 'config': 'mae', 'registered_name': 'mae'}

$ conda list

packages in environment at /panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo:

Answer 1 · 2024-07-11T21:01:09.000Z

Hi,

Thanks for reporting. I have been using tensorflow2.5 to test everything. If you can roll back to a older version, that would be great.

We are working on reproducing your issue and update IsoNet so that it can matches latest versions of dependencies.

packages in environment at /panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo:

Name Version Build Channel