tensorflow.python.framework.errors.InvalidArgumentError: Incompatible shapes: [80,1] vs. [160,1]
SeekPoint opened this issue · 3 comments
INFO:tensorflow:Results after 1190 steps (0.382 sec/batch): loss = 0.665373, recall_at_2 = 0.259558823529, recall_at_1 = 0.133613445378, recall_at_10 = 1.0, recall_at_5 = 0.579831932773.
INFO:tensorflow:Results after 1200 steps (0.382 sec/batch): loss = 0.665354, recall_at_2 = 0.258958333333, recall_at_1 = 0.133229166667, recall_at_10 = 1.0, recall_at_5 = 0.579791666667.
INFO:tensorflow:Results after 1210 steps (0.381 sec/batch): loss = 0.665319, recall_at_2 = 0.259504132231, recall_at_1 = 0.13347107438, recall_at_10 = 1.0, recall_at_5 = 0.580113636364.
INFO:tensorflow:Results after 1220 steps (0.381 sec/batch): loss = 0.66533, recall_at_2 = 0.259733606557, recall_at_1 = 0.133913934426, recall_at_10 = 1.0, recall_at_5 = 0.580276639344.
W tensorflow/core/framework/op_kernel.cc:968] Invalid argument: Incompatible shapes: [80,1] vs. [160,1]
[[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](prediction/Squeeze, prediction/ToFloat)]]
Traceback (most recent call last):
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 972, in _do_call
return fn(*args)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 954, in _run_fn
status, run_metadata)
File "/usr/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/framework/errors.py", line 463, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors.InvalidArgumentError: Incompatible shapes: [80,1] vs. [160,1]
[[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](prediction/Squeeze, prediction/ToFloat)]]
[[Node: recall_at_10/ToInt64/_107 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_320_recall_at_10/ToInt64", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/cpu:0"]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "udc_train.py", line 64, in
tf.app.run()
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "udc_train.py", line 61, in main
estimator.fit(input_fn=input_fn_train, steps=None, monitors=[eval_monitor])
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 333, in fit
max_steps=max_steps)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 708, in _train_model
max_steps=max_steps)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 285, in _monitored_train
None)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 368, in run
run_metadata=run_metadata)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 521, in run
run_metadata=run_metadata)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 488, in run
return self._sess.run(*args, **kwargs)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 625, in run
hook in outputs else None))
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 1215, in after_run
induce_stop = m.step_end(self._last_step, result)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 411, in step_end
return self.every_n_step_end(step, output)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 712, in every_n_step_end
steps=self.eval_steps, metrics=self.metrics, name=self.name)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 399, in evaluate
name=name)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 771, in _evaluate_model
max_steps=steps)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 738, in evaluate
session.run(update_op, feed_dict=feed_dict)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 717, in run
run_metadata_ptr)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 915, in _run
feed_dict_string, options, run_metadata)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 965, in _do_run
target_list, options, run_metadata)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 985, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Incompatible shapes: [80,1] vs. [160,1]
[[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](prediction/Squeeze, prediction/ToFloat)]]
[[Node: recall_at_10/ToInt64/_107 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_320_recall_at_10/ToInt64", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/cpu:0"]]
Caused by op 'prediction/logistic_loss/mul', defined at:
File "udc_train.py", line 64, in
tf.app.run()
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "udc_train.py", line 61, in main
estimator.fit(input_fn=input_fn_train, steps=None, monitors=[eval_monitor])
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 333, in fit
max_steps=max_steps)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 708, in _train_model
max_steps=max_steps)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 285, in _monitored_train
None)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 368, in run
run_metadata=run_metadata)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 521, in run
run_metadata=run_metadata)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 488, in run
return self._sess.run(*args, **kwargs)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 625, in run
hook in outputs else None))
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 1215, in after_run
induce_stop = m.step_end(self._last_step, result)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 411, in step_end
return self.every_n_step_end(step, output)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 712, in every_n_step_end
steps=self.eval_steps, metrics=self.metrics, name=self.name)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 399, in evaluate
name=name)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 760, in _evaluate_model
eval_dict = self._get_eval_ops(features, targets, metrics)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 991, in _get_eval_ops
predictions, loss, _ = self._call_model_fn(features, targets, ModeKeys.EVAL)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 946, in _call_model_fn
return self._model_fn(features, targets, mode=mode)
File "/home/mldl/ub16_prj/chatbot-retrieval/udc_model.py", line 83, in model_fn
tf.concat(0, all_targets))
File "/home/mldl/ub16_prj/chatbot-retrieval/models/dual_encoder.py", line 81, in dual_encoder_model
losses = tf.nn.sigmoid_cross_entropy_with_logits(logits, tf.to_float(targets))
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/ops/nn.py", line 448, in sigmoid_cross_entropy_with_logits
return math_ops.add(relu_logits - logits * targets,
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 751, in binary_op_wrapper
return func(x, y, name=name)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 910, in _mul_dispatch
return gen_math_ops.mul(x, y, name=name)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1519, in mul
result = _op_def_lib.apply_op("Mul", x=x, y=y, name=name)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
op_def=op_def)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/mldl/ub16_prj/VENV_host/py3tf0.10/venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1298, in init
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): Incompatible shapes: [80,1] vs. [160,1]
[[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](prediction/Squeeze, prediction/ToFloat)]]
[[Node: recall_at_10/ToInt64/_107 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_320_recall_at_10/ToInt64", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/cpu:0"]]
(venv) mldl@mldlUB1604:~/ub16_prj/chatbot-retrieval$
I solved this by changing eval_batch_size to 8. But I don't know how to define eval_batch_size by my own choice
thanks, it works,
I also wondering how to change eval_batch_size
hi, i came across with the same problem. do you have some suggestions about how to change the eval_batch_size? thanks