Astronet Walkthrough Prediction - Restoring From Checkpoint Failed
ianfowler opened this issue · 2 comments
ianfowler commented
When following the instructions to use a trained Astronet model to generate predictions:
# Generate a prediction for a new TCE.
bazel-bin/astronet/predict \
--model=AstroCNNModel \
--config_name=local_global \
--model_dir=${MODEL_DIR} \
--kepler_data_dir=${KEPLER_DATA_DIR} \
--kepler_id=11442793 \
--period=14.44912 \
--t0=2.2 \
--duration=0.11267 \
--output_image_file="${HOME}/astronet/kepler-90i.png"
The following error occurs:
I1201 10:39:14.347496 140735978513280 saver.py:1280] Restoring parameters from /Users/ian/exoplanet-ml/astronet/model/model.ckpt-625
2019-12-01 10:39:14.439242: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Invalid argument: tensor_name = global_view_hidden/block_1/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_1/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_1/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_1/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = logits/bias; expected dtype double does not equal original dtype float
tensor_name = logits/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_1/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_1/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_2/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_2/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_3/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_3/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_4/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_4/kernel; expected dtype double does not equal original dtype float
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: tensor_name = global_view_hidden/block_1/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_1/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_1/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_1/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = logits/bias; expected dtype double does not equal original dtype float
tensor_name = logits/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_1/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_1/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_2/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_2/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_3/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_3/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_4/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_4/kernel; expected dtype double does not equal original dtype float
[[{{node save/RestoreV2}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: tensor_name = global_view_hidden/block_1/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_1/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_1/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_1/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = logits/bias; expected dtype double does not equal original dtype float
tensor_name = logits/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_1/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_1/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_2/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_2/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_3/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_3/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_4/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_4/kernel; expected dtype double does not equal original dtype float
[[node save/RestoreV2 (defined at Users/ian/exoplanet-ml/bazel-bin/astronet/predict.runfiles/__main__/astronet/predict.py:175) ]]
Original stack trace for 'save/RestoreV2':
File "Users/ian/exoplanet-ml/bazel-bin/astronet/predict.runfiles/__main__/astronet/predict.py", line 183, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "usr/local/lib/python3.7/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "usr/local/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "Users/ian/exoplanet-ml/bazel-bin/astronet/predict.runfiles/__main__/astronet/predict.py", line 175, in main
for predictions in estimator.predict(input_fn):
File "usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 635, in predict
hooks=all_hooks) as mon_sess:
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1007, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 725, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1200, in __init__
_WrappedSession.__init__(self, self._create_session())
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1205, in _create_session
return self._sess_creator.create_session()
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 871, in create_session
self.tf_sess = self._session_creator.create_session()
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 638, in create_session
self._scaffold.finalize()
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 229, in finalize
self._saver = training_saver._get_saver_or_default() # pylint: disable=protected-access
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 599, in _get_saver_or_default
saver = Saver(sharded=True, allow_empty=True)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 825, in __init__
self.build()
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 502, in _build_internal
restore_sequentially, reshape)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 381, in _AddShardedRestoreOps
name="restore_shard"))
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 328, in _AddRestoreOps
restore_sequentially)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 575, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1696, in restore_v2
name=name)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/ian/exoplanet-ml/bazel-bin/astronet/predict.runfiles/__main__/astronet/predict.py", line 183, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/usr/local/lib/python3.7/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/usr/local/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "/Users/ian/exoplanet-ml/bazel-bin/astronet/predict.runfiles/__main__/astronet/predict.py", line 175, in main
for predictions in estimator.predict(input_fn):
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 635, in predict
hooks=all_hooks) as mon_sess:
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1007, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 725, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1200, in __init__
_WrappedSession.__init__(self, self._create_session())
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1205, in _create_session
return self._sess_creator.create_session()
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 871, in create_session
self.tf_sess = self._session_creator.create_session()
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 647, in create_session
init_fn=self._scaffold.init_fn)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/session_manager.py", line 290, in prepare_session
config=config)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/session_manager.py", line 204, in _restore_checkpoint
saver.restore(sess, checkpoint_filename_with_path)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1322, in restore
err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
tensor_name = global_view_hidden/block_1/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_1/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_1/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_1/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_2/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_3/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_4/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = global_view_hidden/block_5/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_1/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_1/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_1/kernel; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_2/bias; expected dtype double does not equal original dtype float
tensor_name = local_view_hidden/block_2/conv_2/kernel; expected dtype double does not equal original dtype float
tensor_name = logits/bias; expected dtype double does not equal original dtype float
tensor_name = logits/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_1/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_1/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_2/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_2/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_3/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_3/kernel; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_4/bias; expected dtype double does not equal original dtype float
tensor_name = pre_logits_hidden/fully_connected_4/kernel; expected dtype double does not equal original dtype float
[[node save/RestoreV2 (defined at Users/ian/exoplanet-ml/bazel-bin/astronet/predict.runfiles/__main__/astronet/predict.py:175) ]]
Original stack trace for 'save/RestoreV2':
File "Users/ian/exoplanet-ml/bazel-bin/astronet/predict.runfiles/__main__/astronet/predict.py", line 183, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "usr/local/lib/python3.7/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "usr/local/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "Users/ian/exoplanet-ml/bazel-bin/astronet/predict.runfiles/__main__/astronet/predict.py", line 175, in main
for predictions in estimator.predict(input_fn):
File "usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 635, in predict
hooks=all_hooks) as mon_sess:
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1007, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 725, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1200, in __init__
_WrappedSession.__init__(self, self._create_session())
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1205, in _create_session
return self._sess_creator.create_session()
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 871, in create_session
self.tf_sess = self._session_creator.create_session()
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 638, in create_session
self._scaffold.finalize()
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 229, in finalize
self._saver = training_saver._get_saver_or_default() # pylint: disable=protected-access
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 599, in _get_saver_or_default
saver = Saver(sharded=True, allow_empty=True)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 825, in __init__
self.build()
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 502, in _build_internal
restore_sequentially, reshape)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 381, in _AddShardedRestoreOps
name="restore_shard"))
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 328, in _AddRestoreOps
restore_sequentially)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 575, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1696, in restore_v2
name=name)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "usr/local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()
As the error suggests (buried in there), the directory given for the model isn't lining up to where things are stored. I solved it by changing the path:
# Create directory for the extracted TFRecord files.
BASE_DIR="${HOME}/astronet/"
# Generate a prediction for a new TCE.
bazel-bin/astronet/predict \
--model=AstroCNNModel \
--config_name=local_global \
--model_dir=${BASE_DIR} \
--kepler_data_dir=${KEPLER_DATA_DIR} \
--kepler_id=11442793 \
--period=14.44912 \
--t0=2.2 \
--duration=0.11267 \
--output_image_file="${HOME}/astronet/kepler-90i.png"
This prediction for following the demo exactly is slightly different every time from the expected 0.9480018 (which I assume shouldn't be the case first in the sense that it's different every time and second in the sense that it's off by the demo's result by 45%) and gravitates around 50%. (Three trials: 0.5015401002407824, 0.49691700167065095, 0.49994740353818445). Is this an issue with my solution for the first problem (changing the path of the model), or is this an entirely different issue?
ianfowler commented
I figured out why the prediction is wrong:
I1201 00:19:08.316953 140735978513280 estimator.py:612] Could not find trained model in model_dir: /Users/ian/exoplanet-ml/astronet, running initialization to predict.
I1201 00:19:08.334474 140735978513280 estimator.py:1145] Calling model_fn.
W1201 00:19:08.335051 140735978513280 deprecation_wrapper.py:119] From /Users/ian/exoplanet-ml/bazel-bin/astronet/predict.runfiles/__main__/astronet/astro_model/astro_model.py:303: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.
Going back a level in the directory didn't find the right files.
Now I'm sure that the correct files are indeed at the specified directory.
exoplanet-ml ian$ ls $MODEL_DIR
checkpoint eval_test events.out.tfevents.1575219078.bear-mbp model.ckpt-625.data-00000-of-00001 model.ckpt-625.meta
config.json eval_val graph.pbtxt model.ckpt-625.index
If so, why would there be problems restoring?
caitlynlee commented
The original error isn't due to the model not being in the given model directory - as you said in your second post the model is in the right place and the model_dir argument is correct. The issue is a data type mismatch between tf.float32 and tf.float64.
In astronet/predict.py you just need to force the features to be floats instead of doubles (numpy floats).
115: global_view = preprocess.global_view(time, flux, FLAGS.period).astype(np.float32)
...
120: local_view = preprocess.local_view(time, flux, FLAGS.period, FLAGS.duration).astype(np.float32)