Tensorflow object detection api Windows fatal exception: access violation, step stayed at 0
Closed this issue · 1 comments
Trying to train custom object detection model following the tutorial:
https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/index.html
Below is the output when I run main_model_tf2.py to train the model.
One of the things that I noticed is that the step remains 0 throughout.
All the folders involved follows the same structure as the tutorial instructed.
2021-06-12 20:17:32.813643: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-06-12 20:17:35.974727: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library nvcuda.dll
2021-06-12 20:17:36.003520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:02:00.0 name: NVIDIA GeForce GTX 1650 with Max-Q Design computeCapability: 7.5
coreClock: 1.245GHz coreCount: 16 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.34GiB/s
2021-06-12 20:17:36.003577: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-06-12 20:17:36.034868: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-06-12 20:17:36.034898: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
2021-06-12 20:17:36.051579: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cufft64_10.dll
2021-06-12 20:17:36.055829: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library curand64_10.dll
2021-06-12 20:17:36.073808: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusolver64_11.dll
2021-06-12 20:17:36.077387: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusparse64_11.dll
2021-06-12 20:17:36.078442: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-12 20:17:36.078540: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-12 20:17:36.079006: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-12 20:17:36.106947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:02:00.0 name: NVIDIA GeForce GTX 1650 with Max-Q Design computeCapability: 7.5
coreClock: 1.245GHz coreCount: 16 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.34GiB/s
2021-06-12 20:17:36.107008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-12 20:17:36.619265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-12 20:17:36.619301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2021-06-12 20:17:36.619309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2021-06-12 20:17:36.619452: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2145 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 1650 with Max-Q Design, pci bus id: 0000:02:00.0, compute capability: 7.5)
WARNING:tensorflow:Collective ops is not configured at program startup. Some performance features may not be enabled.
W0612 20:17:36.634464 11076 mirrored_strategy.py:379] Collective ops is not configured at program startup. Some performance features may not be enabled.
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
I0612 20:17:36.728192 11076 mirrored_strategy.py:369] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: None
I0612 20:17:36.728192 11076 config_util.py:552] Maybe overwriting train_steps: None
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0612 20:17:36.728192 11076 config_util.py:552] Maybe overwriting use_bfloat16: False
WARNING:tensorflow:From C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py:557: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
W0612 20:17:36.744960 11076 deprecation.py:330] From C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py:557: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
INFO:tensorflow:Reading unweighted datasets: ['annotations/train.record']
I0612 20:17:36.760623 11076 dataset_builder.py:163] Reading unweighted datasets: ['annotations/train.record']
INFO:tensorflow:Reading record datasets for input file: ['annotations/train.record']
I0612 20:17:36.760623 11076 dataset_builder.py:80] Reading record datasets for input file: ['annotations/train.record']
INFO:tensorflow:Number of filenames to read: 1
I0612 20:17:36.760623 11076 dataset_builder.py:81] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0612 20:17:36.760623 11076 dataset_builder.py:87] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\object_detection\builders\dataset_builder.py:101: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
W0612 20:17:36.760623 11076 deprecation.py:330] From C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\object_detection\builders\dataset_builder.py:101: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
WARNING:tensorflow:From C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\object_detection\builders\dataset_builder.py:236: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.map() W0612 20:17:36.776248 11076 deprecation.py:330] From C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\object_detection\builders\dataset_builder.py:236: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.map()
WARNING:tensorflow:From C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\util\dispatch.py:206: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a tf.sparse.SparseTensor and use tf.sparse.to_dense instead.
W0612 20:17:42.585761 11076 deprecation.py:330] From C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\util\dispatch.py:206: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a tf.sparse.SparseTensor and use tf.sparse.to_dense instead.
WARNING:tensorflow:From C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\util\dispatch.py:206: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
seed2 arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
W0612 20:17:45.209203 11076 deprecation.py:330] From C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\util\dispatch.py:206: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
seed2 arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
WARNING:tensorflow:From C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\autograph\impl\api.py:464: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W0612 20:17:46.765527 11076 deprecation.py:330] From C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\autograph\impl\api.py:464: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2021-06-12 20:17:49.246970: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\keras\backend.py:435: UserWarning: tf.keras.backend.set_learning_phase is deprecated and will be removed after 2020-10-11. To update it, simply pass a True/False value to the training argument of the __call__ method of your layer or model.
warnings.warn('tf.keras.backend.set_learning_phase is deprecated and '
2021-06-12 20:18:20.053992: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-12 20:18:20.578938: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8201
2021-06-12 20:18:21.387817: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 934.75MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-06-12 20:18:21.387940: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 934.75MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-06-12 20:18:21.605687: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-06-12 20:18:22.316327: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
2021-06-12 20:18:22.544715: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.54GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-06-12 20:18:22.544804: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.54GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-06-12 20:18:22.680473: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 916.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-06-12 20:18:22.680578: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 916.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-06-12 20:18:22.680686: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.54GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-06-12 20:18:22.680753: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.54GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-06-12 20:18:33.129542: W tensorflow/core/common_runtime/bfc_allocator.cc:456] Allocator (GPU_0_bfc) ran out of memory trying to allocate 400.00MiB (rounded to 419430400)requested by op ResNet50V1_FPN/model/conv2_block2_3_bn/FusedBatchNormV3
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2021-06-12 20:18:33.129852: I tensorflow/core/common_runtime/bfc_allocator.cc:991] BFCAllocator dump for GPU_0_bfc
2021-06-12 20:18:33.130006: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (256): Total Chunks: 77, Chunks in use: 77. 19.2KiB allocated for chunks. 19.2KiB in use in bin. 8.7KiB client-requested in use in bin.
2021-06-12 20:18:33.130118: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (512): Total Chunks: 94, Chunks in use: 81. 47.0KiB allocated for chunks. 40.5KiB in use in bin. 35.3KiB client-requested in use in bin.
2021-06-12 20:18:33.130229: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (1024): Total Chunks: 263, Chunks in use: 261. 276.2KiB allocated for chunks. 274.0KiB in use in bin. 271.0KiB client-requested in use in bin.
2021-06-12 20:18:33.130329: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (2048): Total Chunks: 65, Chunks in use: 61. 134.5KiB allocated for chunks. 123.2KiB in use in bin. 121.5KiB client-requested in use in bin.
2021-06-12 20:18:33.130427: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (4096): Total Chunks: 32, Chunks in use: 31. 135.5KiB allocated for chunks. 130.8KiB in use in bin. 130.8KiB client-requested in use in bin.
2021-06-12 20:18:33.130529: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (8192): Total Chunks: 16, Chunks in use: 16. 134.0KiB allocated for chunks. 134.0KiB in use in bin. 128.0KiB client-requested in use in bin.
2021-06-12 20:18:33.130631: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (16384): Total Chunks: 5, Chunks in use: 4. 131.0KiB allocated for chunks. 103.0KiB in use in bin. 91.0KiB client-requested in use in bin.
2021-06-12 20:18:33.130732: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (32768): Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin.
2021-06-12 20:18:33.130829: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (65536): Total Chunks: 8, Chunks in use: 6. 561.0KiB allocated for chunks. 416.0KiB in use in bin. 384.0KiB client-requested in use in bin.
2021-06-12 20:18:33.130929: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (131072): Total Chunks: 7, Chunks in use: 7. 1.13MiB allocated for chunks. 1.13MiB in use in bin. 1.13MiB client-requested in use in bin.
2021-06-12 20:18:33.131026: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (262144): Total Chunks: 11, Chunks in use: 9. 3.10MiB allocated for chunks. 2.44MiB in use in bin. 2.16MiB client-requested in use in bin.
2021-06-12 20:18:33.131123: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (524288): Total Chunks: 11, Chunks in use: 10. 6.41MiB allocated for chunks. 5.91MiB in use in bin. 5.57MiB client-requested in use in bin.
2021-06-12 20:18:33.131222: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (1048576): Total Chunks: 13, Chunks in use: 12. 14.22MiB allocated for chunks. 13.00MiB in use in bin. 12.00MiB client-requested in use in bin.
2021-06-12 20:18:33.131321: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (2097152): Total Chunks: 24, Chunks in use: 21. 55.75MiB allocated for chunks. 48.75MiB in use in bin. 46.50MiB client-requested in use in bin.
2021-06-12 20:18:33.131698: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (4194304): Total Chunks: 21, Chunks in use: 21. 98.05MiB allocated for chunks. 98.05MiB in use in bin. 98.05MiB client-requested in use in bin.
2021-06-12 20:18:33.131835: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (8388608): Total Chunks: 5, Chunks in use: 5. 50.61MiB allocated for chunks. 50.61MiB in use in bin. 50.61MiB client-requested in use in bin.
2021-06-12 20:18:33.131939: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (16777216): Total Chunks: 18, Chunks in use: 18. 348.91MiB allocated for chunks. 348.91MiB in use in bin. 344.19MiB client-requested in use in bin.
2021-06-12 20:18:33.132030: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-06-12 20:18:33.132129: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (67108864): Total Chunks: 3, Chunks in use: 1. 270.09MiB allocated for chunks. 75.00MiB in use in bin. 75.00MiB client-requested in use in bin.
2021-06-12 20:18:33.132223: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (134217728): Total Chunks: 1, Chunks in use: 0. 136.58MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-06-12 20:18:33.132325: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (268435456): Total Chunks: 3, Chunks in use: 2. 1.13GiB allocated for chunks. 821.95MiB in use in bin. 800.00MiB client-requested in use in bin.
2021-06-12 20:18:33.132408: I tensorflow/core/common_runtime/bfc_allocator.cc:1014] Bin for 400.00MiB was 256.00MiB, Chunk State:
2021-06-12 20:18:33.132526: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Size: 337.16MiB | Requested Size: 19.51MiB | in_use: 0 | bin_num: 20, prev: Size: 4.88MiB | Requested Size: 4.88MiB | in_use: 1 | bin_num: -1, for: Loss/Compare_10/IOU/Equal, stepid: 16762585343613708217, last_action: 126448115942167, for: UNUSED, stepid: 16762585343613708217, last_action: 126448115942071
2021-06-12 20:18:33.132604: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 2249614848
2021-06-12 20:18:33.132724: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0b000000 of size 1280 by op ScratchBuffer action_count 126448115939866 step 0 next 1
2021-06-12 20:18:33.132802: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0b000500 of size 256 by op AssignVariableOp action_count 126448115939867 step 0 next 2
2021-06-12 20:18:33.132875: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0b000600 of size 256 by op AssignVariableOp action_count 126448115939868 step 0 next 3
2021-06-12 20:18:33.132945: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0b000700 of size 256 by op Mul action_count 126448115939941 step 0 next 4
2021-06-12 20:18:33.133013: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0b000800 of size 256 by op SameWorkerRecvDone action_count 126448115939890 step 0 next 5
2021-06-12 20:18:33.133081: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0b000900 of size 78643200 by op SameWorkerRecvDone action_count 126448115939871 step 0 next 6
2021-06-12 20:18:33.133147: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb00900 of size 256 by op SameWorkerRecvDone action_count 126448115939892 step 0 next 7
2021-06-12 20:18:33.133212: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb00a00 of size 256 by op SameWorkerRecvDone action_count 126448115939893 step 0 next 8
2021-06-12 20:18:33.133278: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb00b00 of size 6400 by op SameWorkerRecvDone action_count 126448115939874 step 0 next 9
2021-06-12 20:18:33.133346: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb02400 of size 25600 by op SameWorkerRecvDone action_count 126448115939875 step 0 next 10
2021-06-12 20:18:33.133427: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb08800 of size 25600 by op SameWorkerRecvDone action_count 126448115939876 step 0 next 11
2021-06-12 20:18:33.133505: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb0ec00 of size 25600 by op SameWorkerRecvDone action_count 126448115939877 step 0 next 12
2021-06-12 20:18:33.133577: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb15000 of size 6400 by op SameWorkerRecvDone action_count 126448115939889 step 0 next 13
2021-06-12 20:18:33.133647: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb16900 of size 1792 by op SameWorkerRecvDone action_count 126448115939879 step 0 next 14
2021-06-12 20:18:33.133744: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb17000 of size 1792 by op SameWorkerRecvDone action_count 126448115939880 step 0 next 15
2021-06-12 20:18:33.133813: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb17700 of size 256 by op SameWorkerRecvDone action_count 126448115939881 step 0 next 16
2021-06-12 20:18:33.133879: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb17800 of size 256 by op SameWorkerRecvDone action_count 126448115939882 step 0 next 17
2021-06-12 20:18:33.133947: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb17900 of size 6400 by op SameWorkerRecvDone action_count 126448115939883 step 0 next 18
2021-06-12 20:18:33.134016: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb19200 of size 256 by op SameWorkerRecvDone action_count 126448115939894 step 0 next 19
2021-06-12 20:18:33.134086: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb19300 of size 256 by op Fill action_count 126448115939947 step 0 next 33
2021-06-12 20:18:33.134155: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb19400 of size 256 by op Fill action_count 126448115939948 step 0 next 30
2021-06-12 20:18:33.134221: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb19500 of size 256 by op Fill action_count 126448115939949 step 0 next 29
2021-06-12 20:18:33.134286: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb19600 of size 256 by op Fill action_count 126448115939950 step 0 next 28
2021-06-12 20:18:33.134350: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb19700 of size 256 by op Fill action_count 126448115939951 step 0 next 27
2021-06-12 20:18:33.134414: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb19800 of size 256 by op Fill action_count 126448115939952 step 0 next 26
2021-06-12 20:18:33.134485: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb19900 of size 1024 by op Fill action_count 126448115940185 step 0 next 167
2021-06-12 20:18:33.134553: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb19d00 of size 1024 by op Fill action_count 126448115940186 step 0 next 168
2021-06-12 20:18:33.134607: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb1a100 of size 4096 by op Fill action_count 126448115940192 step 0 next 169
2021-06-12 20:18:33.134616: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb1b100 of size 4096 by op Fill action_count 126448115940193 step 0 next 171
2021-06-12 20:18:33.134624: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb1c100 of size 4096 by op Fill action_count 126448115940194 step 0 next 172
2021-06-12 20:18:33.134632: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb1d100 of size 4096 by op Fill action_count 126448115940195 step 0 next 173
2021-06-12 20:18:33.134639: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb1e100 of size 1024 by op Fill action_count 126448115940201 step 0 next 174
2021-06-12 20:18:33.134648: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb1e500 of size 1024 by op Fill action_count 126448115940202 step 0 next 176
2021-06-12 20:18:33.134657: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb1e900 of size 1024 by op Fill action_count 126448115940203 step 0 next 177
2021-06-12 20:18:33.134666: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb1ed00 of size 1024 by op Fill action_count 126448115940204 step 0 next 178
2021-06-12 20:18:33.134675: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb1f100 of size 1024 by op Fill action_count 126448115940210 step 0 next 179
2021-06-12 20:18:33.134683: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb1f500 of size 1024 by op Fill action_count 126448115940211 step 0 next 182
2021-06-12 20:18:33.134691: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb1f900 of size 1024 by op Fill action_count 126448115940212 step 0 next 183
2021-06-12 20:18:33.134699: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb1fd00 of size 1024 by op Fill action_count 126448115940213 step 0 next 184
2021-06-12 20:18:33.134707: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb20100 of size 4096 by op Fill action_count 126448115940219 step 0 next 186
2021-06-12 20:18:33.134715: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb21100 of size 4096 by op Fill action_count 126448115940220 step 0 next 187
2021-06-12 20:18:33.134723: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb22100 of size 4096 by op Fill action_count 126448115940221 step 0 next 188
2021-06-12 20:18:33.134731: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb23100 of size 4096 by op Fill action_count 126448115940222 step 0 next 189
2021-06-12 20:18:33.134740: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb24100 of size 1024 by op Fill action_count 126448115940228 step 0 next 191
2021-06-12 20:18:33.134748: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb24500 of size 1024 by op Fill action_count 126448115940229 step 0 next 192
2021-06-12 20:18:33.134756: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb24900 of size 1024 by op Fill action_count 126448115940230 step 0 next 193
2021-06-12 20:18:33.134763: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb24d00 of size 1024 by op Fill action_count 126448115940231 step 0 next 194
2021-06-12 20:18:33.134771: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb25100 of size 1024 by op Fill action_count 126448115940237 step 0 next 197
2021-06-12 20:18:33.134779: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb25500 of size 1024 by op Fill action_count 126448115940238 step 0 next 198
2021-06-12 20:18:33.134788: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb25900 of size 1024 by op Fill action_count 126448115940239 step 0 next 199
2021-06-12 20:18:33.134796: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb25d00 of size 1024 by op Fill action_count 126448115940240 step 0 next 200
2021-06-12 20:18:33.134804: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb26100 of size 4096 by op Fill action_count 126448115940246 step 0 next 202
2021-06-12 20:18:33.134812: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb27100 of size 4096 by op Fill action_count 126448115940247 step 0 next 203
2021-06-12 20:18:33.134820: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb28100 of size 4096 by op Fill action_count 126448115940248 step 0 next 204
2021-06-12 20:18:33.134828: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb29100 of size 4096 by op Fill action_count 126448115940249 step 0 next 205
2021-06-12 20:18:33.134836: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb2a100 of size 1024 by op Fill action_count 126448115940255 step 0 next 206
2021-06-12 20:18:33.134844: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb2a500 of size 1024 by op Fill action_count 126448115940256 step 0 next 207
2021-06-12 20:18:33.134853: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb2a900 of size 1024 by op Fill action_count 126448115940257 step 0 next 208
2021-06-12 20:18:33.134862: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb2ad00 of size 1024 by op Fill action_count 126448115940258 step 0 next 209
2021-06-12 20:18:33.134871: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb2b100 of size 1024 by op Fill action_count 126448115940264 step 0 next 212
2021-06-12 20:18:33.134879: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb2b500 of size 1024 by op Fill action_count 126448115940265 step 0 next 34
2021-06-12 20:18:33.134887: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb2b900 of size 256 by op Add action_count 126448115939943 step 0 next 32
2021-06-12 20:18:33.134895: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb2ba00 of size 37632 by op Add action_count 126448115939944 step 0 next 31
2021-06-12 20:18:33.134903: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb34d00 of size 1024 by op Fill action_count 126448115939958 step 0 next 23
2021-06-12 20:18:33.134911: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb35100 of size 1024 by op Fill action_count 126448115939959 step 0 next 22
2021-06-12 20:18:33.134919: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb35500 of size 1024 by op Fill action_count 126448115939960 step 0 next 21
2021-06-12 20:18:33.134927: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb35900 of size 1024 by op Fill action_count 126448115939961 step 0 next 20
2021-06-12 20:18:33.134935: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb35d00 of size 256 by op Fill action_count 126448115939967 step 0 next 35
2021-06-12 20:18:33.134943: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb35e00 of size 256 by op Fill action_count 126448115939968 step 0 next 37
2021-06-12 20:18:33.134951: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb35f00 of size 256 by op Fill action_count 126448115939969 step 0 next 38
2021-06-12 20:18:33.134959: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb36000 of size 256 by op Fill action_count 126448115939970 step 0 next 39
2021-06-12 20:18:33.134967: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb36100 of size 256 by op Fill action_count 126448115939976 step 0 next 40
2021-06-12 20:18:33.134975: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb36200 of size 256 by op Fill action_count 126448115939977 step 0 next 43
2021-06-12 20:18:33.134983: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb36300 of size 256 by op Fill action_count 126448115939978 step 0 next 44
2021-06-12 20:18:33.134991: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb36400 of size 256 by op Fill action_count 126448115939979 step 0 next 45
2021-06-12 20:18:33.135000: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb36500 of size 1024 by op Fill action_count 126448115939985 step 0 next 48
2021-06-12 20:18:33.135008: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb36900 of size 1024 by op Fill action_count 126448115939986 step 0 next 49
2021-06-12 20:18:33.135016: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb36d00 of size 1024 by op Fill action_count 126448115939987 step 0 next 50
2021-06-12 20:18:33.135024: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb37100 of size 1024 by op Fill action_count 126448115939988 step 0 next 51
2021-06-12 20:18:33.135032: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb37500 of size 256 by op Fill action_count 126448115939994 step 0 next 53
2021-06-12 20:18:33.135040: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb37600 of size 256 by op Fill action_count 126448115939995 step 0 next 54
2021-06-12 20:18:33.135076: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb37700 of size 256 by op Fill action_count 126448115939996 step 0 next 55
2021-06-12 20:18:33.135088: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb37800 of size 256 by op Fill action_count 126448115939997 step 0 next 56
2021-06-12 20:18:33.135096: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb37900 of size 256 by op Fill action_count 126448115940003 step 0 next 57
2021-06-12 20:18:33.135105: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb37a00 of size 256 by op Fill action_count 126448115940004 step 0 next 60
2021-06-12 20:18:33.135113: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb37b00 of size 256 by op Fill action_count 126448115940005 step 0 next 61
2021-06-12 20:18:33.135121: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb37c00 of size 256 by op Fill action_count 126448115940006 step 0 next 62
2021-06-12 20:18:33.135129: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb37d00 of size 1024 by op Fill action_count 126448115940012 step 0 next 63
2021-06-12 20:18:33.135137: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb38100 of size 1024 by op Fill action_count 126448115940013 step 0 next 64
2021-06-12 20:18:33.135146: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb38500 of size 1024 by op Fill action_count 126448115940014 step 0 next 65
2021-06-12 20:18:33.135154: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb38900 of size 1024 by op Fill action_count 126448115940015 step 0 next 66
2021-06-12 20:18:33.135162: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb38d00 of size 256 by op Fill action_count 126448115940021 step 0 next 68
2021-06-12 20:18:33.135170: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb38e00 of size 256 by op Fill action_count 126448115940022 step 0 next 69
2021-06-12 20:18:33.135178: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb38f00 of size 256 by op Fill action_count 126448115940023 step 0 next 70
2021-06-12 20:18:33.135186: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb39000 of size 256 by op Fill action_count 126448115940024 step 0 next 71
2021-06-12 20:18:33.135194: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb39100 of size 256 by op Fill action_count 126448115940030 step 0 next 74
2021-06-12 20:18:33.135202: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb39200 of size 256 by op Fill action_count 126448115940031 step 0 next 75
2021-06-12 20:18:33.135210: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb39300 of size 256 by op Fill action_count 126448115940032 step 0 next 76
2021-06-12 20:18:33.135218: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb39400 of size 256 by op Fill action_count 126448115940033 step 0 next 77
2021-06-12 20:18:33.135227: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb39500 of size 1024 by op Fill action_count 126448115940039 step 0 next 79
2021-06-12 20:18:33.135235: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb39900 of size 1024 by op Fill action_count 126448115940040 step 0 next 80
2021-06-12 20:18:33.135243: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb39d00 of size 1024 by op Fill action_count 126448115940041 step 0 next 81
2021-06-12 20:18:33.135251: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3a100 of size 1024 by op Fill action_count 126448115940042 step 0 next 82
2021-06-12 20:18:33.135259: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3a500 of size 2048 by op Fill action_count 126448115940048 step 0 next 83
2021-06-12 20:18:33.135268: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3ad00 of size 2048 by op Fill action_count 126448115940049 step 0 next 86
2021-06-12 20:18:33.135306: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3b500 of size 2048 by op Fill action_count 126448115940050 step 0 next 87
2021-06-12 20:18:33.135317: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3bd00 of size 2048 by op Fill action_count 126448115940051 step 0 next 88
2021-06-12 20:18:33.135326: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3c500 of size 512 by op Fill action_count 126448115940057 step 0 next 91
2021-06-12 20:18:33.135334: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3c700 of size 512 by op Fill action_count 126448115940058 step 0 next 92
2021-06-12 20:18:33.135342: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3c900 of size 512 by op Fill action_count 126448115940059 step 0 next 93
2021-06-12 20:18:33.135350: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3cb00 of size 512 by op Fill action_count 126448115940060 step 0 next 94
2021-06-12 20:18:33.135358: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3cd00 of size 512 by op Fill action_count 126448115940066 step 0 next 97
2021-06-12 20:18:33.135367: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3cf00 of size 512 by op Fill action_count 126448115940067 step 0 next 98
2021-06-12 20:18:33.135375: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3d100 of size 512 by op Fill action_count 126448115940068 step 0 next 99
2021-06-12 20:18:33.135383: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3d300 of size 512 by op Fill action_count 126448115940069 step 0 next 100
2021-06-12 20:18:33.135391: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3d500 of size 2048 by op Fill action_count 126448115940075 step 0 next 36
2021-06-12 20:18:33.135400: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb3dd00 of size 28672 by op Add action_count 126448115939964 step 0 next 25
2021-06-12 20:18:33.135408: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb44d00 of size 65536 by op Add action_count 126448115939955 step 0 next 24
2021-06-12 20:18:33.135416: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb54d00 of size 2048 by op Fill action_count 126448115940076 step 0 next 101
2021-06-12 20:18:33.135424: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb55500 of size 2048 by op Fill action_count 126448115940077 step 0 next 103
2021-06-12 20:18:33.135432: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb55d00 of size 2048 by op Fill action_count 126448115940078 step 0 next 104
2021-06-12 20:18:33.135441: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb56500 of size 512 by op Fill action_count 126448115940084 step 0 next 105
2021-06-12 20:18:33.135449: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb56700 of size 512 by op Fill action_count 126448115940085 step 0 next 107
2021-06-12 20:18:33.135457: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb56900 of size 512 by op Fill action_count 126448115940086 step 0 next 108
2021-06-12 20:18:33.135465: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb56b00 of size 512 by op Fill action_count 126448115940087 step 0 next 109
2021-06-12 20:18:33.135473: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb56d00 of size 512 by op Fill action_count 126448115940093 step 0 next 110
2021-06-12 20:18:33.135481: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb56f00 of size 512 by op Fill action_count 126448115940094 step 0 next 113
2021-06-12 20:18:33.135489: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb57100 of size 512 by op Fill action_count 126448115940095 step 0 next 114
2021-06-12 20:18:33.135497: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb57300 of size 512 by op Fill action_count 126448115940096 step 0 next 115
2021-06-12 20:18:33.135507: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb57500 of size 2048 by op Fill action_count 126448115940102 step 0 next 117
2021-06-12 20:18:33.135516: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb57d00 of size 2048 by op Fill action_count 126448115940103 step 0 next 118
2021-06-12 20:18:33.135525: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb58500 of size 2048 by op Fill action_count 126448115940104 step 0 next 119
2021-06-12 20:18:33.135533: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb58d00 of size 2048 by op Fill action_count 126448115940105 step 0 next 120
2021-06-12 20:18:33.135541: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb59500 of size 512 by op Fill action_count 126448115940111 step 0 next 122
2021-06-12 20:18:33.135549: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb59700 of size 512 by op Fill action_count 126448115940112 step 0 next 123
2021-06-12 20:18:33.135557: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb59900 of size 512 by op Fill action_count 126448115940113 step 0 next 124
2021-06-12 20:18:33.135565: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb59b00 of size 512 by op Fill action_count 126448115940114 step 0 next 125
2021-06-12 20:18:33.135573: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb59d00 of size 512 by op Fill action_count 126448115940120 step 0 next 128
2021-06-12 20:18:33.135582: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb59f00 of size 512 by op Fill action_count 126448115940121 step 0 next 129
2021-06-12 20:18:33.135590: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5a100 of size 512 by op Fill action_count 126448115940122 step 0 next 130
2021-06-12 20:18:33.135598: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5a300 of size 512 by op Fill action_count 126448115940123 step 0 next 131
2021-06-12 20:18:33.135606: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5a500 of size 2048 by op Fill action_count 126448115940129 step 0 next 133
2021-06-12 20:18:33.135614: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5ad00 of size 2048 by op Fill action_count 126448115940130 step 0 next 134
2021-06-12 20:18:33.135622: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5b500 of size 2048 by op Fill action_count 126448115940131 step 0 next 135
2021-06-12 20:18:33.135630: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5bd00 of size 2048 by op Fill action_count 126448115940132 step 0 next 136
2021-06-12 20:18:33.135639: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5c500 of size 512 by op Fill action_count 126448115940138 step 0 next 137
2021-06-12 20:18:33.135647: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5c700 of size 512 by op Fill action_count 126448115940139 step 0 next 138
2021-06-12 20:18:33.135655: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5c900 of size 512 by op Fill action_count 126448115940140 step 0 next 139
2021-06-12 20:18:33.135662: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5cb00 of size 512 by op Fill action_count 126448115940141 step 0 next 140
2021-06-12 20:18:33.135670: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5cd00 of size 512 by op Fill action_count 126448115940147 step 0 next 143
2021-06-12 20:18:33.135678: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5cf00 of size 512 by op Fill action_count 126448115940148 step 0 next 144
2021-06-12 20:18:33.135686: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5d100 of size 512 by op Fill action_count 126448115940149 step 0 next 145
2021-06-12 20:18:33.135695: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5d300 of size 512 by op Fill action_count 126448115940150 step 0 next 146
2021-06-12 20:18:33.135704: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5d500 of size 2048 by op Fill action_count 126448115940156 step 0 next 148
2021-06-12 20:18:33.135713: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5dd00 of size 2048 by op Fill action_count 126448115940157 step 0 next 149
2021-06-12 20:18:33.135722: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5e500 of size 2048 by op Fill action_count 126448115940158 step 0 next 150
2021-06-12 20:18:33.135730: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5ed00 of size 2048 by op Fill action_count 126448115940159 step 0 next 151
2021-06-12 20:18:33.135738: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb5f500 of size 4096 by op Fill action_count 126448115940165 step 0 next 152
2021-06-12 20:18:33.135745: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb60500 of size 4096 by op Fill action_count 126448115940166 step 0 next 155
2021-06-12 20:18:33.135753: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb61500 of size 4096 by op Fill action_count 126448115940167 step 0 next 156
2021-06-12 20:18:33.135762: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb62500 of size 4096 by op Fill action_count 126448115940168 step 0 next 157
2021-06-12 20:18:33.135770: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb63500 of size 1024 by op Fill action_count 126448115940174 step 0 next 160
2021-06-12 20:18:33.135778: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb63900 of size 1024 by op Fill action_count 126448115940175 step 0 next 161
2021-06-12 20:18:33.135786: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb63d00 of size 1024 by op Fill action_count 126448115940176 step 0 next 162
2021-06-12 20:18:33.135794: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb64100 of size 1024 by op Fill action_count 126448115940177 step 0 next 163
2021-06-12 20:18:33.135802: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb64500 of size 1024 by op Fill action_count 126448115940183 step 0 next 166
2021-06-12 20:18:33.135810: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb64900 of size 1024 by op Fill action_count 126448115940184 step 0 next 46
2021-06-12 20:18:33.135818: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb64d00 of size 65536 by op Add action_count 126448115939982 step 0 next 47
2021-06-12 20:18:33.135826: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb74d00 of size 65536 by op Add action_count 126448115939991 step 0 next 52
2021-06-12 20:18:33.135835: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb84d00 of size 98304 by op Add action_count 126448115940009 step 0 next 41
2021-06-12 20:18:33.135843: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fb9cd00 of size 147456 by op Add action_count 126448115939973 step 0 next 42
2021-06-12 20:18:33.135852: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbc0d00 of size 65536 by op Add action_count 126448115940018 step 0 next 67
2021-06-12 20:18:33.135860: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbd0d00 of size 2048 by op Fill action_count 126448115940408 step 0 next 294
2021-06-12 20:18:33.135868: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbd1500 of size 2048 by op Fill action_count 126448115940409 step 0 next 295
2021-06-12 20:18:33.135876: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbd1d00 of size 2048 by op Fill action_count 126448115940410 step 0 next 296
2021-06-12 20:18:33.135884: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbd2500 of size 2048 by op Fill action_count 126448115940411 step 0 next 297
2021-06-12 20:18:33.135893: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbd2d00 of size 8192 by op Fill action_count 126448115940417 step 0 next 299
2021-06-12 20:18:33.135903: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbd4d00 of size 8192 by op Fill action_count 126448115940418 step 0 next 300
2021-06-12 20:18:33.135911: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbd6d00 of size 8192 by op Fill action_count 126448115940419 step 0 next 301
2021-06-12 20:18:33.135919: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbd8d00 of size 8192 by op Fill action_count 126448115940420 step 0 next 302
2021-06-12 20:18:33.135927: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdad00 of size 1024 by op Fill action_count 126448115940426 step 0 next 304
2021-06-12 20:18:33.135935: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdb100 of size 1024 by op Fill action_count 126448115940432 step 0 next 305
2021-06-12 20:18:33.135943: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdb500 of size 1024 by op Fill action_count 126448115940438 step 0 next 307
2021-06-12 20:18:33.135951: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdb900 of size 1024 by op Fill action_count 126448115940439 step 0 next 308
2021-06-12 20:18:33.135959: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdbd00 of size 1024 by op Fill action_count 126448115940440 step 0 next 309
2021-06-12 20:18:33.135967: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdc100 of size 1024 by op Fill action_count 126448115940441 step 0 next 310
2021-06-12 20:18:33.135976: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdc500 of size 1024 by op Fill action_count 126448115940447 step 0 next 312
2021-06-12 20:18:33.135983: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdc900 of size 1024 by op Fill action_count 126448115940453 step 0 next 314
2021-06-12 20:18:33.135991: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdcd00 of size 1024 by op Fill action_count 126448115940454 step 0 next 315
2021-06-12 20:18:33.136000: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdd100 of size 1024 by op Fill action_count 126448115940455 step 0 next 316
2021-06-12 20:18:33.136007: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdd500 of size 1024 by op Fill action_count 126448115940456 step 0 next 317
2021-06-12 20:18:33.136015: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdd900 of size 1024 by op Fill action_count 126448115940462 step 0 next 318
2021-06-12 20:18:33.136024: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbddd00 of size 1024 by op Fill action_count 126448115940463 step 0 next 319
2021-06-12 20:18:33.136032: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbde100 of size 1024 by op Fill action_count 126448115940464 step 0 next 320
2021-06-12 20:18:33.136040: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbde500 of size 1024 by op Fill action_count 126448115940465 step 0 next 321
2021-06-12 20:18:33.136048: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbde900 of size 1024 by op Fill action_count 126448115940471 step 0 next 323
2021-06-12 20:18:33.136056: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbded00 of size 1024 by op Fill action_count 126448115940472 step 0 next 324
2021-06-12 20:18:33.136064: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdf100 of size 1024 by op Fill action_count 126448115940473 step 0 next 325
2021-06-12 20:18:33.136072: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdf500 of size 1024 by op Fill action_count 126448115940474 step 0 next 326
2021-06-12 20:18:33.136080: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdf900 of size 256 by op Mul action_count 126448115940491 step 0 next 327
2021-06-12 20:18:33.136090: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdfa00 of size 256 by op Cast action_count 126448115940476 step 0 next 328
2021-06-12 20:18:33.136099: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdfb00 of size 256 by op Cast action_count 126448115940479 step 0 next 329
2021-06-12 20:18:33.136107: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdfc00 of size 256 by op Cast action_count 126448115940482 step 0 next 330
2021-06-12 20:18:33.136115: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdfd00 of size 256 by op Cast action_count 126448115940485 step 0 next 331
2021-06-12 20:18:33.136123: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdfe00 of size 256 by op Cast action_count 126448115940488 step 0 next 332
2021-06-12 20:18:33.136131: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbdff00 of size 1024 by op Fill action_count 126448115940496 step 0 next 334
2021-06-12 20:18:33.136140: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe0300 of size 1024 by op Fill action_count 126448115940497 step 0 next 335
2021-06-12 20:18:33.136147: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe0700 of size 1024 by op Fill action_count 126448115940498 step 0 next 336
2021-06-12 20:18:33.136156: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe0b00 of size 1024 by op Fill action_count 126448115940499 step 0 next 337
2021-06-12 20:18:33.136164: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe0f00 of size 1024 by op Fill action_count 126448115940505 step 0 next 339
2021-06-12 20:18:33.136172: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe1300 of size 1024 by op Fill action_count 126448115940506 step 0 next 340
2021-06-12 20:18:33.136180: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe1700 of size 1024 by op Fill action_count 126448115940507 step 0 next 341
2021-06-12 20:18:33.136188: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe1b00 of size 1024 by op Fill action_count 126448115940508 step 0 next 342
2021-06-12 20:18:33.136196: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe1f00 of size 1024 by op Fill action_count 126448115940514 step 0 next 344
2021-06-12 20:18:33.136204: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe2300 of size 1024 by op Fill action_count 126448115940515 step 0 next 345
2021-06-12 20:18:33.136212: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe2700 of size 1024 by op Fill action_count 126448115940516 step 0 next 346
2021-06-12 20:18:33.136220: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe2b00 of size 1024 by op Fill action_count 126448115940517 step 0 next 347
2021-06-12 20:18:33.136228: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe2f00 of size 1024 by op Fill action_count 126448115940523 step 0 next 349
2021-06-12 20:18:33.136237: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe3300 of size 1024 by op Fill action_count 126448115940524 step 0 next 350
2021-06-12 20:18:33.136245: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe3700 of size 1024 by op Fill action_count 126448115940525 step 0 next 351
2021-06-12 20:18:33.136253: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe3b00 of size 1024 by op Fill action_count 126448115940526 step 0 next 352
2021-06-12 20:18:33.136261: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe3f00 of size 256 by op Fill action_count 126448115940532 step 0 next 353
2021-06-12 20:18:33.136268: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe4000 of size 1024 by op Fill action_count 126448115940538 step 0 next 354
2021-06-12 20:18:33.136277: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe4400 of size 1024 by op Fill action_count 126448115940539 step 0 next 355
2021-06-12 20:18:33.136286: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe4800 of size 1024 by op Fill action_count 126448115940540 step 0 next 356
2021-06-12 20:18:33.136295: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe4c00 of size 1024 by op Fill action_count 126448115940541 step 0 next 357
2021-06-12 20:18:33.136304: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe5000 of size 1024 by op Fill action_count 126448115940547 step 0 next 359
2021-06-12 20:18:33.136312: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe5400 of size 1024 by op Fill action_count 126448115940548 step 0 next 360
2021-06-12 20:18:33.136320: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe5800 of size 1024 by op Fill action_count 126448115940549 step 0 next 361
2021-06-12 20:18:33.136328: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe5c00 of size 1024 by op Fill action_count 126448115940550 step 0 next 362
2021-06-12 20:18:33.136336: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe6000 of size 1024 by op Fill action_count 126448115940556 step 0 next 364
2021-06-12 20:18:33.136344: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe6400 of size 1024 by op Fill action_count 126448115940557 step 0 next 365
2021-06-12 20:18:33.136352: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe6800 of size 1024 by op Fill action_count 126448115940558 step 0 next 366
2021-06-12 20:18:33.136360: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe6c00 of size 1024 by op Fill action_count 126448115940559 step 0 next 367
2021-06-12 20:18:33.136368: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe7000 of size 1024 by op Fill action_count 126448115940565 step 0 next 369
2021-06-12 20:18:33.136377: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe7400 of size 1024 by op Fill action_count 126448115940566 step 0 next 370
2021-06-12 20:18:33.136385: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe7800 of size 1024 by op Fill action_count 126448115940567 step 0 next 371
2021-06-12 20:18:33.136393: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at b0fbe7c00 of size 1024 by op Fill action_count 126448115940568 step 0 next 372
Windows fatal exception: access violation
Thread 0x000056c4 (most recent call first):
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\multiprocessing\pool.py", line 576 in _handle_results
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\threading.py", line 892 in run
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\threading.py", line 954 in _bootstrap_inner
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\threading.py", line 912 in _bootstrap
Thread 0x000038f4 (most recent call first):
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\multiprocessing\pool.py", line 528 in _handle_tasks
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\threading.py", line 892 in run
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\threading.py", line 954 in _bootstrap_inner
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\threading.py", line 912 in _bootstrap
Thread 0x0000275c (most recent call first):
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\multiprocessing\connection.py", line 816 in _exhaustive_wait
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\multiprocessing\connection.py", line 884 in wait
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\multiprocessing\pool.py", line 499 in _wait_for_updates
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\multiprocessing\pool.py", line 519 in _handle_workers
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\threading.py", line 892 in run
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\threading.py", line 954 in _bootstrap_inner
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\threading.py", line 912 in _bootstrap
Thread 0x00004788 (most recent call first):
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\multiprocessing\pool.py", line 114 in worker
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\threading.py", line 892 in run
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\threading.py", line 954 in _bootstrap_inner
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\threading.py", line 912 in _bootstrap
Thread 0x00002b44 (most recent call first):
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\eager\execute.py", line 59 in quick_execute
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\eager\function.py", line 591 in call
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\eager\function.py", line 1960 in _call_flat
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\eager\function.py", line 3023 in call
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\eager\def_function.py", line 950 in _call
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\eager\def_function.py", line 889 in call
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 86 in call_for_each_replica
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\distribute\mirrored_strategy.py", line 678 in _call_for_each_replica
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 2833 in call_for_each_replica
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 1285 in run
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py", line 173 in _ensure_model_is_built
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py", line 394 in load_fine_tune_checkpoint
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py", line 599 in train_loop
File "C:\Users\Edward\Projects\Tensorflow\workspace\training_demo\model_main_tf2.py", line 106 in main
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\absl\app.py", line 251 in _run_main
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\absl\app.py", line 303 in run
File "C:\Users\Edward\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\platform\app.py", line 40 in run
File "C:\Users\Edward\Projects\Tensorflow\workspace\training_demo\model_main_tf2.py", line 115 in
This seems like a memory allocation issue: Your GPU is running out of memory. I would try reducing the training batch size