Unable to load the saved_model
kunal017 opened this issue · 4 comments
kunal017 commented
I am running the model for countries_s1 model but getting the following error:
Executing python code/model/trainer.py --base_output_dir output/countries_s1/ --path_length 2 --hidden_size 25 --embedding_size 25 --batch_size 256 --beta 0.05 --Lambda 0.05 --use_entity_embeddings 1 --train_entity_embeddings 1 --train_relation_embeddings 1 --data_input_dir datasets/data_preprocessed/countries_s1/ --vocab_dir datasets/data_preprocessed/countries_s1/vocab --model_load_dir saved_models/countries_s1/model.ckpt --load_model 1 --total_iterations 1000 --nell_evaluation 0
HEY
Arguments:
LSTM_layers : 1
Lambda : 0.05
base_output_dir : output/countries_s1/
batch_size : 256
beta : 0.05
create_vocab : 0
data_input_dir : datasets/data_preprocessed/countries_s1/
embedding_size : 25
eval_every : 100
gamma : 1
grad_clip_norm : 5
hidden_size : 25
input_file : train.txt
input_files : ['datasets/data_preprocessed/countries_s1//train.txt']
l2_reg_const : 0.01
learning_rate : 0.001
load_model : True
log_dir : ./logs/
log_file_name : output/countries_s1//3f71_2_0.05_100_0.05/log.txt
max_num_actions : 200
model_dir : output/countries_s1//3f71_2_0.05_100_0.05/model/
model_load_dir : saved_models/countries_s1/model.ckpt
negative_reward : 0
nell_evaluation : 0
num_rollouts : 20
output_dir : output/countries_s1//3f71_2_0.05_100_0.05
output_file :
path_length : 2
path_logger_file : output/countries_s1//3f71_2_0.05_100_0.05
pool : max
positive_reward : 1.0
pretrained_embeddings_action :
pretrained_embeddings_entity :
test_rollouts : 100
total_iterations : 1000
train_entity_embeddings : True
train_relation_embeddings : True
use_entity_embeddings : True
vocab_dir : datasets/data_preprocessed/countries_s1/vocab
INFO:root:reading vocab files...
01/28/2019 01:56:20 AM: [ reading vocab files... ]
INFO:root:Reading mid to name map
01/28/2019 01:56:20 AM: [ Reading mid to name map ]
INFO:root:Done..
01/28/2019 01:56:20 AM: [ Done.. ]
INFO:root:Total number of entities 273
01/28/2019 01:56:20 AM: [ Total number of entities 273 ]
INFO:root:Total number of relations 6
01/28/2019 01:56:20 AM: [ Total number of relations 6 ]
INFO:root:Skipping training
01/28/2019 01:56:20 AM: [ Skipping training ]
INFO:root:Loading model from saved_models/countries_s1/model.ckpt
01/28/2019 01:56:20 AM: [ Loading model from saved_models/countries_s1/model.ckpt ]
Reading vocab...
batcher loaded
KG constructed
Reading vocab...
Contains full graph
batcher loaded
KG constructed
Reading vocab...
Contains full graph
batcher loaded
KG constructed
2019-01-28 01:56:21.112771: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112794: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112801: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112806: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112812: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.988648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1080 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.582
pciBusID 0000:06:00.0
Total memory: 10.91GiB
Free memory: 10.75GiB
2019-01-28 01:56:21.988740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0
2019-01-28 01:56:21.988757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0: Y
2019-01-28 01:56:21.988799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:06:00.0)
INFO:root:Creating TF graph...
01/28/2019 01:56:22 AM: [ Creating TF graph... ]
INFO:root:TF Graph creation done..
01/28/2019 01:56:23 AM: [ TF Graph creation done.. ]
INFO:tensorflow:Restoring parameters from saved_models/countries_s1/model.ckpt
INFO:tensorflow:Restoring parameters from saved_models/countries_s1/model.ckpt
01/28/2019 01:56:23 AM: [ Restoring parameters from saved_models/countries_s1/model.ckpt ]
2019-01-28 01:56:23.830867: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
2019-01-28 01:56:23.846931: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847248: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847249: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847266: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847290: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847248: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847342: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847358: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847437: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847529: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847785: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847753: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847603: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847794: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847881: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847816: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.848095: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.848317: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.848375: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854893: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854890: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854966: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854933: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855353: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855395: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam not found in checkpoint
2019-01-28 01:56:23.855675: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855750: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855791: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855820: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855785: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856273: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856468: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856486: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856592: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856623: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856677: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
Traceback (most recent call last):
File "code/model/trainer.py", line 573, in <module>
trainer.initialize(restore=save_path, sess=sess)
File "code/model/trainer.py", line 143, in initialize
return self.model_saver.restore(sess, restore)
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1560, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
[[Node: save/RestoreV2_17/_3 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_80_save/RestoreV2_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
Caused by op u'save/RestoreV2_9', defined at:
File "code/model/trainer.py", line 573, in <module>
trainer.initialize(restore=save_path, sess=sess)
File "code/model/trainer.py", line 137, in initialize
self.model_saver = tf.train.Saver(max_to_keep=2)
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1140, in __init__
self.build()
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1172, in build
filename=self._filename)
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 688, in build
restore_sequentially, reshape)
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
[spec.tensor.dtype])[0])
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 663, in restore_v2
dtypes=dtypes, name=name)
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
NotFoundError (see above for traceback): Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
[[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
[[Node: save/RestoreV2_17/_3 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_80_save/RestoreV2_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
shehzaadzd commented
It looks like the model was training with the train_entity_embeddings parameter set as False. Could you change that in the config and rerun?
Albert-xj commented
@kunal017 Have you solved this problem? I also encountered
NotFoundError (see above for traceback): Key policy_steps_unroll/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint [[Node: save/RestoreV2_18 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_18/tensor_names, save/RestoreV2_18/shape_and_slices)]]
@shehzaadzd Can you help me?
shehzaadzd commented
Can you try training from scratch with the same parameters?
Albert-xj commented
Yes, retrain and saved model can be successfully loaded