Error during training in training.py
Opened this issue · 1 comments
while training I am getting error in line
valid_logits_batch_tensor = tf.gather_nd(params=logits_batch_tensor, indices=valid_batch_indices)
in function training.py
Error:
InvalidArgumentError (see above for traceback): Invalid indices: [31,1] = [0, 76, 192] does not index into [1,192,192,21]
[[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd = ScatterNd[T=DT_FLOAT, Tindices=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](ToInt32_3/_587, adam_vars/gradients/Reshape_4_grad/Reshape/_589, adam_vars/gradients/GatherNd_1_grad/Shape/_591)]]
[[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd/_593 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_252_adam_vars/gradients/GatherNd_1_grad/ScatterNd", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]
Did any of you faced similar issue ?
I was using tf 1.0 version, after upgrading to tf 1.1 the issue is resolved.