ERROR IN EXAMPLE: movielens-1m-keras-with-horovod

[VikashPeddakota999]

System information

• OS Platform and Distribution: Linux Ubuntu 16.04
• TensorFlow version == 2.8.3 and was installed through pip
• TensorFlow-Recommenders-Addons version : '0.6.0-dev' and was installed through source
• Python version: 3.8.10
• Is GPU used? yes
• using "tfra/dev_container:latest-python3.8" docker image and installed from source using https://github.com/tensorflow/recommenders-addons/tree/master#installing-from-source

Describe the bug

While i was trying to replicate the tutorial, I'm getting the following error in model.fit()

`File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1021, in train_function *
return step_function(self, iterator)
File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1010, in step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1000, in run_step **
outputs = model.train_step(data)
File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 859, in train_step
y_pred = self(x, training=True)
File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None

ValueError: Exception encountered when calling layer "dual_channels_deep_model" (type DualChannelsDeepModel).

in user code:

    File "/tmp/ipykernel_187617/3872527011.py", line 193, in call  *
        user_latent = self.user_embedding(user_fea_info)
    File "/tmp/ipykernel_187617/3872527011.py", line 69, in __call__  *
        dense_emb_concat = self.dense_embedding_layer(dense_input_tensors_concat)
    File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler  **
        raise e.with_traceback(filtered_tb) from None
    File "/usr/local/lib/python3.8/dist-packages/tensorflow_recommenders_addons/dynamic_embedding/python/keras/layers/embedding.py", line 626, in call
        unique_embeddings = self.__alltoall_embedding_lookup__(unique_ids)
    File "/usr/local/lib/python3.8/dist-packages/tensorflow_recommenders_addons/dynamic_embedding/python/keras/layers/embedding.py", line 605, in __alltoall_embedding_lookup__
        gather_indices = tf.expand_dims(tf.concat(gather_indices, axis=0), axis=-1)

    ValueError: Exception encountered when calling layer "user_DenseUnifiedEmbeddingLayer" (type HvdAllToAllEmbedding).
    
    Tried to convert 'input' to a tensor and failed. Error: None values not supported.
    
    Call arguments received:
      • ids=tf.Tensor(shape=(None, 2), dtype=int32)


Call arguments received:
  • features={'movie_id': 'tf.Tensor(shape=(None,), dtype=int64)', 'movie_genres': 'tf.Tensor(shape=(None,), dtype=int32)', 'user_id': 'tf.Tensor(shape=(None,), dtype=int64)', 'user_gender': 'tf.Tensor(shape=(None,), dtype=int32)', 'user_occupation_label': 'tf.Tensor(shape=(None,), dtype=int32)', 'timestamp': 'tf.Tensor(shape=(None,), dtype=int32)'}`

Code to reproduce the issue

https://github.com/tensorflow/recommenders-addons/blob/master/demo/dynamic_embedding/movielens-1m-keras-with-horovod/movielens-1m-keras-with-horovod.py

Other info / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

[MoFHeka]

@VikashPeddakota999 Fixed. Please check the file in PR #374