Unable to export model after training

Question

Unable to export model after training

Opened this issue 3 years ago · 1 comments

I'm attempting to train a custom aocr model on an internal dataset. I've labeled the data using a directory of images, and an annotation file as described in the README. This was converted to a dataset (training.tfrecords) and then trained according to the instructions (with some parameters to accomodate our data):

aocr train --max-prediction 20 --full-ascii --num-epoch 300 datasets/gen_text_train/training.tfrecords

This ran to completion, and I attempt to export it as shown in the README:

aocr export --format=frozengraph ./attempted_training

This produces a VERY long error:

Full Error

WARNING:tensorflow:From /home/ericsilk/anaconda3/lib/python3.7/site-packages/aocr/__main__.py:20: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From /home/ericsilk/anaconda3/lib/python3.7/site-packages/aocr/main.py:20: The name tf.logging.ERROR is deprecated. Please use tf.compat.v1.logging.ERROR instead.

2021-02-10 09:18:26.028615: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-02-10 09:18:26.038234: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3399905000 Hz
2021-02-10 09:18:26.038452: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56193ecbc7d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-02-10 09:18:26.038477: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-02-10 09:18:26,039 root INFO phase: export
2021-02-10 09:18:26,039 root INFO model_dir: ./checkpoints
2021-02-10 09:18:26,039 root INFO load_model: True
2021-02-10 09:18:26,039 root INFO output_dir: ./results
2021-02-10 09:18:26,039 root INFO steps_per_checkpoint: 0
2021-02-10 09:18:26,039 root INFO batch_size: 1
2021-02-10 09:18:26,040 root INFO learning_rate: 1.000000
2021-02-10 09:18:26,040 root INFO reg_val: 0
2021-02-10 09:18:26,040 root INFO max_gradient_norm: 5.000000
2021-02-10 09:18:26,040 root INFO clip_gradients: True
2021-02-10 09:18:26,040 root INFO max_image_width 160.000000
2021-02-10 09:18:26,040 root INFO max_prediction_length 8.000000
2021-02-10 09:18:26,040 root INFO channels: 1
2021-02-10 09:18:26,040 root INFO target_embedding_size: 10.000000
2021-02-10 09:18:26,040 root INFO attn_num_hidden: 128
2021-02-10 09:18:26,040 root INFO attn_num_layers: 2
2021-02-10 09:18:26,040 root INFO visualize: False
2021-02-10 09:18:27,582 root INFO Reading model parameters from ./checkpoints/model.ckpt-33936
2021-02-10 09:18:27.635972: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
Identity: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv1/W/Initializer/random_uniform/shape (Const)
conv_conv1/W/Initializer/random_uniform/min (Const)
conv_conv1/W/Initializer/random_uniform/max (Const)
conv_conv1/W/Initializer/random_uniform/RandomUniform (RandomUniform)
conv_conv1/W/Initializer/random_uniform/sub (Sub)
conv_conv1/W/Initializer/random_uniform/mul (Mul)
conv_conv1/W/Initializer/random_uniform (Add)
conv_conv1/W (VariableV2) /device:GPU:0
conv_conv1/W/Assign (Assign) /device:GPU:0
conv_conv1/W/read (Identity) /device:GPU:0
save/Assign_5 (Assign) /device:GPU:0

2021-02-10 09:18:27.636123: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
Identity: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv2/W/Initializer/random_uniform/shape (Const)
conv_conv2/W/Initializer/random_uniform/min (Const)
conv_conv2/W/Initializer/random_uniform/max (Const)
conv_conv2/W/Initializer/random_uniform/RandomUniform (RandomUniform)
conv_conv2/W/Initializer/random_uniform/sub (Sub)
conv_conv2/W/Initializer/random_uniform/mul (Mul)
conv_conv2/W/Initializer/random_uniform (Add)
conv_conv2/W (VariableV2) /device:GPU:0
conv_conv2/W/Assign (Assign) /device:GPU:0
conv_conv2/W/read (Identity) /device:GPU:0
save/Assign_6 (Assign) /device:GPU:0

2021-02-10 09:18:27.636214: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
Identity: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv3/W/Initializer/random_uniform/shape (Const)
conv_conv3/W/Initializer/random_uniform/min (Const)
conv_conv3/W/Initializer/random_uniform/max (Const)
conv_conv3/W/Initializer/random_uniform/RandomUniform (RandomUniform)
conv_conv3/W/Initializer/random_uniform/sub (Sub)
conv_conv3/W/Initializer/random_uniform/mul (Mul)
conv_conv3/W/Initializer/random_uniform (Add)
conv_conv3/W (VariableV2) /device:GPU:0
conv_conv3/W/Assign (Assign) /device:GPU:0
conv_conv3/W/read (Identity) /device:GPU:0
save/Assign_11 (Assign) /device:GPU:0

2021-02-10 09:18:27.636282: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv3/BatchNorm/gamma/Initializer/ones (Const)
conv_conv3/BatchNorm/gamma (VariableV2) /device:GPU:0
conv_conv3/BatchNorm/gamma/Assign (Assign) /device:GPU:0
conv_conv3/BatchNorm/gamma/read (Identity) /device:GPU:0
save/Assign_8 (Assign) /device:GPU:0

2021-02-10 09:18:27.636333: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv3/BatchNorm/beta/Initializer/zeros (Const)
conv_conv3/BatchNorm/beta (VariableV2) /device:GPU:0
conv_conv3/BatchNorm/beta/Assign (Assign) /device:GPU:0
conv_conv3/BatchNorm/beta/read (Identity) /device:GPU:0
save/Assign_7 (Assign) /device:GPU:0

2021-02-10 09:18:27.636387: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv3/BatchNorm/moving_mean/Initializer/zeros (Const)
conv_conv3/BatchNorm/moving_mean (VariableV2) /device:GPU:0
conv_conv3/BatchNorm/moving_mean/Assign (Assign) /device:GPU:0
conv_conv3/BatchNorm/moving_mean/read (Identity) /device:GPU:0
save/Assign_9 (Assign) /device:GPU:0

2021-02-10 09:18:27.636465: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv3/BatchNorm/moving_variance/Initializer/ones (Const)
conv_conv3/BatchNorm/moving_variance (VariableV2) /device:GPU:0
conv_conv3/BatchNorm/moving_variance/Assign (Assign) /device:GPU:0
conv_conv3/BatchNorm/moving_variance/read (Identity) /device:GPU:0
save/Assign_10 (Assign) /device:GPU:0

2021-02-10 09:18:27.636590: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
Identity: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv4/W/Initializer/random_uniform/shape (Const)
conv_conv4/W/Initializer/random_uniform/min (Const)
conv_conv4/W/Initializer/random_uniform/max (Const)
conv_conv4/W/Initializer/random_uniform/RandomUniform (RandomUniform)
conv_conv4/W/Initializer/random_uniform/sub (Sub)
conv_conv4/W/Initializer/random_uniform/mul (Mul)
conv_conv4/W/Initializer/random_uniform (Add)
conv_conv4/W (VariableV2) /device:GPU:0
conv_conv4/W/Assign (Assign) /device:GPU:0
conv_conv4/W/read (Identity) /device:GPU:0
save/Assign_12 (Assign) /device:GPU:0

2021-02-10 09:18:27.636745: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
Identity: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv5/W/Initializer/random_uniform/shape (Const)
conv_conv5/W/Initializer/random_uniform/min (Const)
conv_conv5/W/Initializer/random_uniform/max (Const)
conv_conv5/W/Initializer/random_uniform/RandomUniform (RandomUniform)
conv_conv5/W/Initializer/random_uniform/sub (Sub)
conv_conv5/W/Initializer/random_uniform/mul (Mul)
conv_conv5/W/Initializer/random_uniform (Add)
conv_conv5/W (VariableV2) /device:GPU:0
conv_conv5/W/Assign (Assign) /device:GPU:0
conv_conv5/W/read (Identity) /device:GPU:0
save/Assign_17 (Assign) /device:GPU:0

2021-02-10 09:18:27.636882: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv5/BatchNorm/gamma/Initializer/ones (Const)
conv_conv5/BatchNorm/gamma (VariableV2) /device:GPU:0
conv_conv5/BatchNorm/gamma/Assign (Assign) /device:GPU:0
conv_conv5/BatchNorm/gamma/read (Identity) /device:GPU:0
save/Assign_14 (Assign) /device:GPU:0

2021-02-10 09:18:27.636984: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv5/BatchNorm/beta/Initializer/zeros (Const)
conv_conv5/BatchNorm/beta (VariableV2) /device:GPU:0
conv_conv5/BatchNorm/beta/Assign (Assign) /device:GPU:0
conv_conv5/BatchNorm/beta/read (Identity) /device:GPU:0
save/Assign_13 (Assign) /device:GPU:0

2021-02-10 09:18:27.637085: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv5/BatchNorm/moving_mean/Initializer/zeros (Const)
conv_conv5/BatchNorm/moving_mean (VariableV2) /device:GPU:0
conv_conv5/BatchNorm/moving_mean/Assign (Assign) /device:GPU:0
conv_conv5/BatchNorm/moving_mean/read (Identity) /device:GPU:0
save/Assign_15 (Assign) /device:GPU:0

2021-02-10 09:18:27.637197: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv5/BatchNorm/moving_variance/Initializer/ones (Const)
conv_conv5/BatchNorm/moving_variance (VariableV2) /device:GPU:0
conv_conv5/BatchNorm/moving_variance/Assign (Assign) /device:GPU:0
conv_conv5/BatchNorm/moving_variance/read (Identity) /device:GPU:0
save/Assign_16 (Assign) /device:GPU:0

2021-02-10 09:18:27.637320: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
Identity: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv6/W/Initializer/random_uniform/shape (Const)
conv_conv6/W/Initializer/random_uniform/min (Const)
conv_conv6/W/Initializer/random_uniform/max (Const)
conv_conv6/W/Initializer/random_uniform/RandomUniform (RandomUniform)
conv_conv6/W/Initializer/random_uniform/sub (Sub)
conv_conv6/W/Initializer/random_uniform/mul (Mul)
conv_conv6/W/Initializer/random_uniform (Add)
conv_conv6/W (VariableV2) /device:GPU:0
conv_conv6/W/Assign (Assign) /device:GPU:0
conv_conv6/W/read (Identity) /device:GPU:0
save/Assign_18 (Assign) /device:GPU:0

2021-02-10 09:18:27.637472: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
Identity: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv7/W/Initializer/random_uniform/shape (Const)
conv_conv7/W/Initializer/random_uniform/min (Const)
conv_conv7/W/Initializer/random_uniform/max (Const)
conv_conv7/W/Initializer/random_uniform/RandomUniform (RandomUniform)
conv_conv7/W/Initializer/random_uniform/sub (Sub)
conv_conv7/W/Initializer/random_uniform/mul (Mul)
conv_conv7/W/Initializer/random_uniform (Add)
conv_conv7/W (VariableV2) /device:GPU:0
conv_conv7/W/Assign (Assign) /device:GPU:0
conv_conv7/W/read (Identity) /device:GPU:0
save/Assign_23 (Assign) /device:GPU:0

2021-02-10 09:18:27.637608: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv7/BatchNorm/gamma/Initializer/ones (Const)
conv_conv7/BatchNorm/gamma (VariableV2) /device:GPU:0
conv_conv7/BatchNorm/gamma/Assign (Assign) /device:GPU:0
conv_conv7/BatchNorm/gamma/read (Identity) /device:GPU:0
save/Assign_20 (Assign) /device:GPU:0

2021-02-10 09:18:27.638096: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv7/BatchNorm/beta/Initializer/zeros (Const)
conv_conv7/BatchNorm/beta (VariableV2) /device:GPU:0
conv_conv7/BatchNorm/beta/Assign (Assign) /device:GPU:0
conv_conv7/BatchNorm/beta/read (Identity) /device:GPU:0
save/Assign_19 (Assign) /device:GPU:0

2021-02-10 09:18:27.638206: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv7/BatchNorm/moving_mean/Initializer/zeros (Const)
conv_conv7/BatchNorm/moving_mean (VariableV2) /device:GPU:0
conv_conv7/BatchNorm/moving_mean/Assign (Assign) /device:GPU:0
conv_conv7/BatchNorm/moving_mean/read (Identity) /device:GPU:0
save/Assign_21 (Assign) /device:GPU:0

2021-02-10 09:18:27.638308: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
conv_conv7/BatchNorm/moving_variance/Initializer/ones (Const)
conv_conv7/BatchNorm/moving_variance (VariableV2) /device:GPU:0
conv_conv7/BatchNorm/moving_variance/Assign (Assign) /device:GPU:0
conv_conv7/BatchNorm/moving_variance/read (Identity) /device:GPU:0
save/Assign_22 (Assign) /device:GPU:0

2021-02-10 09:18:27.638472: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform/shape (Const)
bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform/min (Const)
bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform/max (Const)
bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform/RandomUniform (RandomUniform)
bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform/sub (Sub)
bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform/mul (Mul)
bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform (Add)
bidirectional_rnn/fw/basic_lstm_cell/kernel (VariableV2) /device:GPU:0
bidirectional_rnn/fw/basic_lstm_cell/kernel/Assign (Assign) /device:GPU:0
save/Assign_4 (Assign) /device:GPU:0

2021-02-10 09:18:27.638609: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
VariableV2: CPU
Const: CPU XLA_CPU
Fill: CPU XLA_CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
bidirectional_rnn/fw/basic_lstm_cell/bias/Initializer/zeros/shape_as_tensor (Const)
bidirectional_rnn/fw/basic_lstm_cell/bias/Initializer/zeros/Const (Const)
bidirectional_rnn/fw/basic_lstm_cell/bias/Initializer/zeros (Fill)
bidirectional_rnn/fw/basic_lstm_cell/bias (VariableV2) /device:GPU:0
bidirectional_rnn/fw/basic_lstm_cell/bias/Assign (Assign) /device:GPU:0
save/Assign_3 (Assign) /device:GPU:0

2021-02-10 09:18:27.639111: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform/shape (Const)
bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform/min (Const)
bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform/max (Const)
bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform/RandomUniform (RandomUniform)
bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform/sub (Sub)
bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform/mul (Mul)
bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform (Add)
bidirectional_rnn/bw/basic_lstm_cell/kernel (VariableV2) /device:GPU:0
bidirectional_rnn/bw/basic_lstm_cell/kernel/Assign (Assign) /device:GPU:0
save/Assign_2 (Assign) /device:GPU:0

2021-02-10 09:18:27.639261: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
VariableV2: CPU
Const: CPU XLA_CPU
Fill: CPU XLA_CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
bidirectional_rnn/bw/basic_lstm_cell/bias/Initializer/zeros/shape_as_tensor (Const)
bidirectional_rnn/bw/basic_lstm_cell/bias/Initializer/zeros/Const (Const)
bidirectional_rnn/bw/basic_lstm_cell/bias/Initializer/zeros (Fill)
bidirectional_rnn/bw/basic_lstm_cell/bias (VariableV2) /device:GPU:0
bidirectional_rnn/bw/basic_lstm_cell/bias/Assign (Assign) /device:GPU:0
save/Assign_1 (Assign) /device:GPU:0

2021-02-10 09:18:27.639855: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
Identity: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform/shape (Const)
embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform/min (Const)
embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform/max (Const)
embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform/RandomUniform (RandomUniform)
embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform/sub (Sub)
embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform/mul (Mul)
embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform (Add)
embedding_attention_decoder/attention_decoder/AttnW_0 (VariableV2) /device:GPU:0
embedding_attention_decoder/attention_decoder/AttnW_0/Assign (Assign) /device:GPU:0
embedding_attention_decoder/attention_decoder/AttnW_0/read (Identity) /device:GPU:0
save/Assign_29 (Assign) /device:GPU:0

2021-02-10 09:18:27.640798: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
Identity: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform/shape (Const)
embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform/min (Const)
embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform/max (Const)
embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform/RandomUniform (RandomUniform)
embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform/sub (Sub)
embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform/mul (Mul)
embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform (Add)
embedding_attention_decoder/attention_decoder/AttnV_0 (VariableV2) /device:GPU:0
embedding_attention_decoder/attention_decoder/AttnV_0/Assign (Assign) /device:GPU:0
embedding_attention_decoder/attention_decoder/AttnV_0/read (Identity) /device:GPU:0
save/Assign_28 (Assign) /device:GPU:0

2021-02-10 09:18:27.641065: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
Identity: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform/shape (Const)
embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform/min (Const)
embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform/max (Const)
embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform/RandomUniform (RandomUniform)
embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform/sub (Sub)
embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform/mul (Mul)
embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform (Add)
embedding_attention_decoder/attention_decoder/kernel (VariableV2) /device:GPU:0
embedding_attention_decoder/attention_decoder/kernel/Assign (Assign) /device:GPU:0
embedding_attention_decoder/attention_decoder/kernel/read (Identity) /device:GPU:0
save/Assign_31 (Assign) /device:GPU:0

2021-02-10 09:18:27.641259: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
embedding_attention_decoder/attention_decoder/bias/Initializer/Const (Const)
embedding_attention_decoder/attention_decoder/bias (VariableV2) /device:GPU:0
embedding_attention_decoder/attention_decoder/bias/Assign (Assign) /device:GPU:0
embedding_attention_decoder/attention_decoder/bias/read (Identity) /device:GPU:0
save/Assign_30 (Assign) /device:GPU:0

2021-02-10 09:18:27.641389: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
Identity: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform/shape (Const)
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform/min (Const)
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform/max (Const)
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform/RandomUniform (RandomUniform)
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform/sub (Sub)
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform/mul (Mul)
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform (Add)
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel (VariableV2) /device:GPU:0
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Assign (Assign) /device:GPU:0
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/read (Identity) /device:GPU:0
save/Assign_33 (Assign) /device:GPU:0

2021-02-10 09:18:27.641530: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/bias/Initializer/zeros (Const)
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/bias (VariableV2) /device:GPU:0
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/bias/Assign (Assign) /device:GPU:0
embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/bias/read (Identity) /device:GPU:0
save/Assign_32 (Assign) /device:GPU:0

2021-02-10 09:18:27.641686: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
Identity: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform/shape (Const)
embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform/min (Const)
embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform/max (Const)
embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform/RandomUniform (RandomUniform)
embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform/sub (Sub)
embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform/mul (Mul)
embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform (Add)
embedding_attention_decoder/attention_decoder/Attention_0/kernel (VariableV2) /device:GPU:0
embedding_attention_decoder/attention_decoder/Attention_0/kernel/Assign (Assign) /device:GPU:0
embedding_attention_decoder/attention_decoder/Attention_0/kernel/read (Identity) /device:GPU:0
save/Assign_25 (Assign) /device:GPU:0

2021-02-10 09:18:27.641934: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
embedding_attention_decoder/attention_decoder/Attention_0/bias/Initializer/Const (Const)
embedding_attention_decoder/attention_decoder/Attention_0/bias (VariableV2) /device:GPU:0
embedding_attention_decoder/attention_decoder/Attention_0/bias/Assign (Assign) /device:GPU:0
embedding_attention_decoder/attention_decoder/Attention_0/bias/read (Identity) /device:GPU:0
save/Assign_24 (Assign) /device:GPU:0

2021-02-10 09:18:27.642040: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Assign: CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Mul: CPU XLA_CPU
Sub: CPU XLA_CPU
Add: CPU XLA_CPU
Identity: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform/shape (Const)
embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform/min (Const)
embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform/max (Const)
embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform/RandomUniform (RandomUniform)
embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform/sub (Sub)
embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform/mul (Mul)
embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform (Add)
embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel (VariableV2) /device:GPU:0
embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Assign (Assign) /device:GPU:0
embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/read (Identity) /device:GPU:0
save/Assign_27 (Assign) /device:GPU:0

2021-02-10 09:18:27.642139: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: CPU XLA_CPU
Assign: CPU
Const: CPU XLA_CPU
VariableV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
embedding_attention_decoder/attention_decoder/AttnOutputProjection/bias/Initializer/Const (Const)
embedding_attention_decoder/attention_decoder/AttnOutputProjection/bias (VariableV2) /device:GPU:0
embedding_attention_decoder/attention_decoder/AttnOutputProjection/bias/Assign (Assign) /device:GPU:0
embedding_attention_decoder/attention_decoder/AttnOutputProjection/bias/read (Identity) /device:GPU:0
save/Assign_26 (Assign) /device:GPU:0

2021-02-10 09:18:27.642847: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Switch: CPU XLA_CPU
Enter: CPU XLA_CPU
LookupTableFindV2: CPU
LookupTableInsertV2: CPU
MutableHashTableV2: CPU
LookupTableExportV2: CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
MutableHashTable (MutableHashTableV2) /device:GPU:0
MutableHashTable_lookup_table_export_values/LookupTableExportV2 (LookupTableExportV2) /device:GPU:0
MutableHashTable_lookup_table_insert/LookupTableInsertV2 (LookupTableInsertV2) /device:GPU:0
map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2/Enter (Enter) /device:GPU:0
map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2/Enter_1 (Enter) /device:GPU:0
map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2/Switch (Switch) /device:GPU:0
map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2/Enter_2 (Enter) /device:GPU:0
map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2/Enter_3 (Enter) /device:GPU:0
map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2/Switch_2 (Switch) /device:GPU:0
map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2 (LookupTableFindV2) /device:GPU:0

Traceback (most recent call last):
File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [39,10] rhs shape= [98,10]
[[{{node save/Assign_34}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1290, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [39,10] rhs shape= [98,10]
[[node save/Assign_34 (defined at /lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'save/Assign_34':
File "/bin/aocr", line 8, in
sys.exit(main())
File "/lib/python3.7/site-packages/aocr/main.py", line 251, in main
channels=parameters.channels,
File "/lib/python3.7/site-packages/aocr/model/model.py", line 261, in init
self.saver_all = tf.train.Saver(tf.all_variables())
File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 828, in init
self.build()
File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 840, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build
build_restore=build_restore)
File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/lib/python3.7/site-packages/tensorflow_core/python/training/saving/saveable_object_util.py", line 73, in restore
self.op.get_shape().is_fully_defined())
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/state_ops.py", line 227, in assign
validate_shape=validate_shape)
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_state_ops.py", line 66, in assign
use_locking=use_locking, name=name)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ericsilk/anaconda3/bin/aocr", line 8, in
sys.exit(main())
File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/aocr/main.py", line 251, in main
channels=parameters.channels,
File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/aocr/model/model.py", line 268, in init
self.saver_all.restore(self.sess, ckpt.model_checkpoint_path)
File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1326, in restore
err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [39,10] rhs shape= [98,10]
[[node save/Assign_34 (defined at /lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'save/Assign_34':
File "/bin/aocr", line 8, in
sys.exit(main())
File "/lib/python3.7/site-packages/aocr/main.py", line 251, in main
channels=parameters.channels,
File "/lib/python3.7/site-packages/aocr/model/model.py", line 261, in init
self.saver_all = tf.train.Saver(tf.all_variables())
File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 828, in init
self.build()
File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 840, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build
build_restore=build_restore)
File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/lib/python3.7/site-packages/tensorflow_core/python/training/saving/saveable_object_util.py", line 73, in restore
self.op.get_shape().is_fully_defined())
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/state_ops.py", line 227, in assign
validate_shape=validate_shape)
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_state_ops.py", line 66, in assign
use_locking=use_locking, name=name)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()}}

The key bits of which appear to be:

Traceback (most recent call last):
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [39,10] rhs shape= [98,10]
         [[{{node save/Assign_34}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1290, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [39,10] rhs shape= [98,10]
         [[node save/Assign_34 (defined at /lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'save/Assign_34':
  File "/bin/aocr", line 8, in <module>
    sys.exit(main())
  File "/lib/python3.7/site-packages/aocr/__main__.py", line 251, in main
    channels=parameters.channels,
  File "/lib/python3.7/site-packages/aocr/model/model.py", line 261, in __init__
    self.saver_all = tf.train.Saver(tf.all_variables())
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 828, in __init__
    self.build()
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 840, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build
    build_restore=build_restore)
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal
    restore_sequentially, reshape)
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 350, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saving/saveable_object_util.py", line 73, in restore
    self.op.get_shape().is_fully_defined())
  File "/lib/python3.7/site-packages/tensorflow_core/python/ops/state_ops.py", line 227, in assign
    validate_shape=validate_shape)
  File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_state_ops.py", line 66, in assign
    use_locking=use_locking, name=name)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ericsilk/anaconda3/bin/aocr", line 8, in <module>
    sys.exit(main())
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/aocr/__main__.py", line 251, in main
    channels=parameters.channels,
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/aocr/model/model.py", line 268, in __init__
    self.saver_all.restore(self.sess, ckpt.model_checkpoint_path)
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1326, in restore
    err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [39,10] rhs shape= [98,10]
         [[node save/Assign_34 (defined at /lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'save/Assign_34':
  File "/bin/aocr", line 8, in <module>
    sys.exit(main())
  File "/lib/python3.7/site-packages/aocr/__main__.py", line 251, in main
    channels=parameters.channels,
  File "/lib/python3.7/site-packages/aocr/model/model.py", line 261, in __init__
    self.saver_all = tf.train.Saver(tf.all_variables())
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 828, in __init__
    self.build()
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 840, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build
    build_restore=build_restore)
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal
    restore_sequentially, reshape)
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 350, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saving/saveable_object_util.py", line 73, in restore
    self.op.get_shape().is_fully_defined())
  File "/lib/python3.7/site-packages/tensorflow_core/python/ops/state_ops.py", line 227, in assign
    validate_shape=validate_shape)
  File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_state_ops.py", line 66, in assign
    use_locking=use_locking, name=name)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

System info: Ubuntu 18.04.5 LTS, Anaconda 3
Any suggestions?

Answer 1 · 2021-03-20T11:12:41.000Z

I frequently use aocr inside Google Colabs and i haven't got any problems.
I use tensorflow_gpu==1.15 with aocr