Keras Callback Error specific GPU
touhi99 opened this issue · 0 comments
touhi99 commented
I am having the following issues after first epoch of the training
Note that, I am using forcefully only 1 gpu in the machine with this
os.environ["TF_FORCE_GPU_ALLOW_GROWTH"] = "true"
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
Program runs fine without the callback, but including callbacks having this issue.
`2019-09-04 10:04:01.301387: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 9.58GiB (rounded to 10290790400). Current allocation summary follows.
2019-09-04 10:04:01.301716: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): Total Chunks: 200, Chunks in use: 199. 50.0KiB allocated for chunks. 49.8KiB in use in bin. 2.6KiB client-requested in use in bin.
2019-09-04 10:04:01.301921: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): Total Chunks: 8, Chunks in use: 7. 4.0KiB allocated for chunks. 3.5KiB in use in bin. 2.4KiB client-requested in use in bin.
2019-09-04 10:04:01.302028: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): Total Chunks: 2, Chunks in use: 2. 2.2KiB allocated for chunks. 2.2KiB in use in bin. 2.0KiB client-requested in use in bin.
2019-09-04 10:04:01.302127: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): Total Chunks: 23, Chunks in use: 23. 46.0KiB allocated for chunks. 46.0KiB in use in bin. 45.0KiB client-requested in use in bin.
2019-09-04 10:04:01.302224: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): Total Chunks: 4, Chunks in use: 3. 16.5KiB allocated for chunks. 12.5KiB in use in bin. 12.5KiB client-requested in use in bin.
2019-09-04 10:04:01.302358: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): Total Chunks: 19, Chunks in use: 19. 156.0KiB allocated for chunks. 156.0KiB in use in bin. 156.0KiB client-requested in use in bin.
2019-09-04 10:04:01.302453: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): Total Chunks: 1, Chunks in use: 1. 16.5KiB allocated for chunks. 16.5KiB in use in bin. 16.3KiB client-requested in use in bin.
2019-09-04 10:04:01.302548: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): Total Chunks: 1, Chunks in use: 1. 32.0KiB allocated for chunks. 32.0KiB in use in bin. 32.0KiB client-requested in use in bin.
2019-09-04 10:04:01.302644: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): Total Chunks: 5, Chunks in use: 5. 336.0KiB allocated for chunks. 336.0KiB in use in bin. 336.0KiB client-requested in use in bin.
2019-09-04 10:04:01.302783: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): Total Chunks: 5, Chunks in use: 5. 992.0KiB allocated for chunks. 992.0KiB in use in bin. 992.0KiB client-requested in use in bin.
2019-09-04 10:04:01.302874: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-09-04 10:04:01.302969: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): Total Chunks: 1, Chunks in use: 1. 794.5KiB allocated for chunks. 794.5KiB in use in bin. 448.0KiB client-requested in use in bin.
2019-09-04 10:04:01.303064: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): Total Chunks: 32, Chunks in use: 32. 32.00MiB allocated for chunks. 32.00MiB in use in bin. 32.00MiB client-requested in use in bin.
2019-09-04 10:04:01.303166: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): Total Chunks: 34, Chunks in use: 34. 68.46MiB allocated for chunks. 68.46MiB in use in bin. 68.46MiB client-requested in use in bin.
2019-09-04 10:04:01.303263: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): Total Chunks: 18, Chunks in use: 18. 79.56MiB allocated for chunks. 79.56MiB in use in bin. 72.00MiB client-requested in use in bin.
2019-09-04 10:04:01.303358: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): Total Chunks: 18, Chunks in use: 17. 146.89MiB allocated for chunks. 136.00MiB in use in bin. 136.00MiB client-requested in use in bin.
2019-09-04 10:04:01.303453: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): Total Chunks: 4, Chunks in use: 4. 67.54MiB allocated for chunks. 67.54MiB in use in bin. 64.00MiB client-requested in use in bin.
2019-09-04 10:04:01.303546: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-09-04 10:04:01.303639: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): Total Chunks: 4, Chunks in use: 4. 256.00MiB allocated for chunks. 256.00MiB in use in bin. 256.00MiB client-requested in use in bin.
2019-09-04 10:04:01.303737: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): Total Chunks: 2, Chunks in use: 1. 372.16MiB allocated for chunks. 197.90MiB in use in bin. 174.25MiB client-requested in use in bin.
2019-09-04 10:04:01.303833: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): Total Chunks: 2, Chunks in use: 0. 3.00GiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-09-04 10:04:01.303925: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 9.58GiB was 256.00MiB, Chunk State:
2019-09-04 10:04:01.304017: I tensorflow/core/common_runtime/bfc_allocator.cc:786] Size: 1.00GiB | Requested Size: 275.0KiB | in_use: 0 | bin_num: 20
2019-09-04 10:04:01.304103: I tensorflow/core/common_runtime/bfc_allocator.cc:786] Size: 2.00GiB | Requested Size: 528.95MiB | in_use: 0 | bin_num: 20
2019-09-04 10:04:01.304183: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 2147483648
2019-09-04 10:04:01.304263: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f2344000000 next 18446744073709551615 of size 2147483648
2019-09-04 10:04:01.304343: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1073741824
2019-09-04 10:04:01.304425: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f23c4000000 next 18446744073709551615 of size 1073741824
2019-09-04 10:04:01.304503: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 536870912
2019-09-04 10:04:01.304586: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424000000 next 120 of size 4194304
2019-09-04 10:04:01.304670: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424400000 next 121 of size 256
2019-09-04 10:04:01.304753: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424400100 next 122 of size 256
2019-09-04 10:04:01.304837: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424400200 next 123 of size 2048
2019-09-04 10:04:01.304864: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424400a00 next 124 of size 256
2019-09-04 10:04:01.304940: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424400b00 next 125 of size 8192
2019-09-04 10:04:01.305021: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424402b00 next 126 of size 4194304
2019-09-04 10:04:01.305102: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424802b00 next 127 of size 256
2019-09-04 10:04:01.305174: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424802c00 next 128 of size 2048
2019-09-04 10:04:01.305258: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424803400 next 129 of size 8192
2019-09-04 10:04:01.305293: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805400 next 130 of size 256
2019-09-04 10:04:01.305370: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805500 next 131 of size 256
2019-09-04 10:04:01.305453: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805600 next 132 of size 256
2019-09-04 10:04:01.305536: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805700 next 133 of size 256
2019-09-04 10:04:01.305622: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805800 next 134 of size 256
2019-09-04 10:04:01.305703: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805900 next 135 of size 256
2019-09-04 10:04:01.305733: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805a00 next 136 of size 8388608
2019-09-04 10:04:01.305811: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425005a00 next 137 of size 8388608
2019-09-04 10:04:01.305894: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425805a00 next 138 of size 256
2019-09-04 10:04:01.305990: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425805b00 next 139 of size 256
2019-09-04 10:04:01.306086: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425805c00 next 140 of size 256
2019-09-04 10:04:01.306170: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425805d00 next 141 of size 204800
2019-09-04 10:04:01.306252: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425837d00 next 142 of size 204800
2019-09-04 10:04:01.306281: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425869d00 next 143 of size 4194304
2019-09-04 10:04:01.306357: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c69d00 next 144 of size 256
2019-09-04 10:04:01.306438: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c69e00 next 145 of size 256
2019-09-04 10:04:01.306515: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c69f00 next 146 of size 256
2019-09-04 10:04:01.306546: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a000 next 147 of size 256
2019-09-04 10:04:01.306620: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a100 next 148 of size 256
2019-09-04 10:04:01.306700: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a200 next 149 of size 256
2019-09-04 10:04:01.306727: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a300 next 150 of size 256
2019-09-04 10:04:01.306803: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a400 next 151 of size 256
2019-09-04 10:04:01.306884: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a500 next 152 of size 256
2019-09-04 10:04:01.306912: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a600 next 153 of size 256
2019-09-04 10:04:01.306986: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a700 next 154 of size 256
2019-09-04 10:04:01.307068: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a800 next 155 of size 256
2019-09-04 10:04:01.307143: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a900 next 156 of size 2097152
2019-09-04 10:04:01.307175: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425e6a900 next 157 of size 2097152
2019-09-04 10:04:01.307248: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242606a900 next 158 of size 2097152
2019-09-04 10:04:01.307328: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242626a900 next 159 of size 2097152
2019-09-04 10:04:01.307406: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646a900 next 160 of size 256
2019-09-04 10:04:01.307481: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646aa00 next 161 of size 256
2019-09-04 10:04:01.307597: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646ab00 next 162 of size 2048
2019-09-04 10:04:01.307631: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646b300 next 163 of size 2048
2019-09-04 10:04:01.307711: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646bb00 next 164 of size 2048
2019-09-04 10:04:01.307738: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646c300 next 165 of size 2048
2019-09-04 10:04:01.307814: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646cb00 next 166 of size 1048576
2019-09-04 10:04:01.307891: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242656cb00 next 167 of size 1048576
2019-09-04 10:04:01.307922: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242666cb00 next 168 of size 1048576
2019-09-04 10:04:01.307995: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242676cb00 next 169 of size 1048576
2019-09-04 10:04:01.308121: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242686cb00 next 170 of size 512
2019-09-04 10:04:01.308198: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242686cd00 next 171 of size 2048
2019-09-04 10:04:01.308231: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242686d500 next 172 of size 2048
2019-09-04 10:04:01.308307: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242686dd00 next 173 of size 2048
2019-09-04 10:04:01.308387: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242686e500 next 174 of size 2048
2019-09-04 10:04:01.308524: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242686ed00 next 175 of size 1048576
2019-09-04 10:04:01.308611: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242696ed00 next 176 of size 1048576
2019-09-04 10:04:01.308639: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2426a6ed00 next 177 of size 1048576
2019-09-04 10:04:01.308651: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2426b6ed00 next 178 of size 1048576
2019-09-04 10:04:01.308661: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2426c6ed00 next 179 of size 2097152
2019-09-04 10:04:01.308672: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2426e6ed00 next 180 of size 2097152
2019-09-04 10:04:01.308681: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242706ed00 next 181 of size 2097152
2019-09-04 10:04:01.308690: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242726ed00 next 182 of size 2097152
2019-09-04 10:04:01.308699: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242746ed00 next 183 of size 2048
2019-09-04 10:04:01.308708: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242746f500 next 184 of size 2048
2019-09-04 10:04:01.308717: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242746fd00 next 185 of size 2048
2019-09-04 10:04:01.308725: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427470500 next 186 of size 2048
2019-09-04 10:04:01.308734: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427470d00 next 187 of size 1048576
2019-09-04 10:04:01.308743: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427570d00 next 188 of size 1048576
2019-09-04 10:04:01.308751: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427670d00 next 189 of size 1048576
2019-09-04 10:04:01.308760: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427770d00 next 190 of size 1048576
2019-09-04 10:04:01.308768: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427870d00 next 191 of size 2097152
2019-09-04 10:04:01.308777: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427a70d00 next 192 of size 2097152
2019-09-04 10:04:01.308786: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427c70d00 next 193 of size 2097152
2019-09-04 10:04:01.308795: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427e70d00 next 194 of size 2097152
2019-09-04 10:04:01.308806: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428070d00 next 195 of size 2097152
2019-09-04 10:04:01.308818: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428270d00 next 196 of size 2097152
2019-09-04 10:04:01.308830: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428470d00 next 197 of size 2097152
2019-09-04 10:04:01.308843: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428670d00 next 198 of size 2097152
2019-09-04 10:04:01.308854: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428870d00 next 199 of size 2048
2019-09-04 10:04:01.308862: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428871500 next 200 of size 2048
2019-09-04 10:04:01.308869: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428871d00 next 201 of size 2048
2019-09-04 10:04:01.308876: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428872500 next 202 of size 2048
2019-09-04 10:04:01.308888: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428872d00 next 203 of size 1048576
2019-09-04 10:04:01.308904: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428972d00 next 204 of size 1048576
2019-09-04 10:04:01.308918: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428a72d00 next 205 of size 1048576
2019-09-04 10:04:01.308930: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428b72d00 next 206 of size 1048576
2019-09-04 10:04:01.308941: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c72d00 next 207 of size 256
2019-09-04 10:04:01.308953: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c72e00 next 208 of size 256
2019-09-04 10:04:01.308964: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c72f00 next 209 of size 256
2019-09-04 10:04:01.308977: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73000 next 210 of size 256
2019-09-04 10:04:01.308989: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73100 next 211 of size 256
2019-09-04 10:04:01.309001: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73200 next 212 of size 256
2019-09-04 10:04:01.309012: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73300 next 213 of size 256
2019-09-04 10:04:01.309024: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73400 next 214 of size 256
2019-09-04 10:04:01.309044: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73500 next 215 of size 256
2019-09-04 10:04:01.309061: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73600 next 216 of size 256
2019-09-04 10:04:01.309072: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73700 next 217 of size 256
2019-09-04 10:04:01.309084: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73800 next 218 of size 256
2019-09-04 10:04:01.309099: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73900 next 219 of size 256
2019-09-04 10:04:01.309113: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73a00 next 220 of size 256
2019-09-04 10:04:01.309126: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73b00 next 221 of size 256
2019-09-04 10:04:01.309139: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73c00 next 222 of size 256
2019-09-04 10:04:01.309150: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73d00 next 223 of size 256
2019-09-04 10:04:01.309161: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73e00 next 224 of size 256
2019-09-04 10:04:01.309174: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73f00 next 225 of size 256
2019-09-04 10:04:01.309188: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74000 next 226 of size 256
2019-09-04 10:04:01.309199: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74100 next 227 of size 256
2019-09-04 10:04:01.309210: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74200 next 228 of size 256
2019-09-04 10:04:01.309223: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74300 next 229 of size 256
2019-09-04 10:04:01.309237: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74400 next 230 of size 256
2019-09-04 10:04:01.309249: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74500 next 231 of size 256
2019-09-04 10:04:01.309260: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74600 next 232 of size 256
2019-09-04 10:04:01.309272: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74700 next 233 of size 256
2019-09-04 10:04:01.309286: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74800 next 234 of size 256
2019-09-04 10:04:01.309298: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74900 next 235 of size 256
2019-09-04 10:04:01.309309: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74a00 next 236 of size 256
2019-09-04 10:04:01.309321: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74b00 next 237 of size 256
2019-09-04 10:04:01.309334: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74c00 next 238 of size 256
2019-09-04 10:04:01.309348: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74d00 next 239 of size 256
2019-09-04 10:04:01.309359: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74e00 next 240 of size 256
2019-09-04 10:04:01.309371: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74f00 next 241 of size 256
2019-09-04 10:04:01.309384: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75000 next 242 of size 256
2019-09-04 10:04:01.309398: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75100 next 243 of size 256
2019-09-04 10:04:01.309410: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75200 next 244 of size 256
2019-09-04 10:04:01.309421: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75300 next 245 of size 256
2019-09-04 10:04:01.309434: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75400 next 246 of size 256
2019-09-04 10:04:01.309447: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75500 next 247 of size 256
2019-09-04 10:04:01.309460: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75600 next 248 of size 256
2019-09-04 10:04:01.309470: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75700 next 249 of size 256
2019-09-04 10:04:01.309482: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75800 next 250 of size 256
2019-09-04 10:04:01.309496: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75900 next 251 of size 256
2019-09-04 10:04:01.309508: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75a00 next 252 of size 256
2019-09-04 10:04:01.309520: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75b00 next 253 of size 256
2019-09-04 10:04:01.309531: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75c00 next 254 of size 256
2019-09-04 10:04:01.309545: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75d00 next 255 of size 256
2019-09-04 10:04:01.309558: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75e00 next 256 of size 256
2019-09-04 10:04:01.309570: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75f00 next 257 of size 256
2019-09-04 10:04:01.309581: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76000 next 258 of size 256
2019-09-04 10:04:01.309594: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76100 next 259 of size 256
2019-09-04 10:04:01.309607: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76200 next 260 of size 256
2019-09-04 10:04:01.309621: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76300 next 261 of size 256
2019-09-04 10:04:01.309633: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76400 next 262 of size 256
2019-09-04 10:04:01.309645: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76500 next 263 of size 256
2019-09-04 10:04:01.309660: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76600 next 264 of size 256
2019-09-04 10:04:01.309676: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76700 next 265 of size 256
2019-09-04 10:04:01.309683: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76800 next 266 of size 256
2019-09-04 10:04:01.309690: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76900 next 267 of size 256
2019-09-04 10:04:01.309697: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76a00 next 268 of size 256
2019-09-04 10:04:01.309704: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76b00 next 269 of size 256
2019-09-04 10:04:01.309710: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76c00 next 270 of size 256
2019-09-04 10:04:01.309717: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76d00 next 271 of size 256
2019-09-04 10:04:01.309724: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76e00 next 272 of size 256
2019-09-04 10:04:01.309730: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76f00 next 273 of size 256
2019-09-04 10:04:01.309737: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77000 next 2713 of size 256
2019-09-04 10:04:01.309744: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77100 next 3471 of size 256
2019-09-04 10:04:01.309750: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77200 next 3575 of size 256
2019-09-04 10:04:01.309757: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77300 next 2265 of size 256
2019-09-04 10:04:01.309765: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77400 next 1127 of size 256
2019-09-04 10:04:01.309772: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77500 next 2772 of size 256
2019-09-04 10:04:01.309779: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77600 next 1975 of size 256
2019-09-04 10:04:01.309786: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77700 next 2690 of size 256
2019-09-04 10:04:01.309795: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77800 next 673 of size 256
2019-09-04 10:04:01.309818: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77900 next 3121 of size 256
2019-09-04 10:04:01.309837: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77a00 next 2802 of size 256
2019-09-04 10:04:01.309846: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77b00 next 2764 of size 256
2019-09-04 10:04:01.309853: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77c00 next 380 of size 256
2019-09-04 10:04:01.309866: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77d00 next 4360 of size 256
2019-09-04 10:04:01.309879: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77e00 next 2592 of size 256
2019-09-04 10:04:01.309888: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77f00 next 2588 of size 256
2019-09-04 10:04:01.309896: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78000 next 4795 of size 256
2019-09-04 10:04:01.309908: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78100 next 5023 of size 256
2019-09-04 10:04:01.309919: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78200 next 4579 of size 256
2019-09-04 10:04:01.309930: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78300 next 3630 of size 256
2019-09-04 10:04:01.309944: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78400 next 1171 of size 256
2019-09-04 10:04:01.309957: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78500 next 452 of size 256
2019-09-04 10:04:01.309970: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78600 next 905 of size 256
2019-09-04 10:04:01.309991: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78700 next 1584 of size 256
2019-09-04 10:04:01.310005: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78800 next 1764 of size 256
2019-09-04 10:04:01.310287: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78900 next 3939 of size 256
2019-09-04 10:04:01.310319: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78a00 next 3634 of size 256
2019-09-04 10:04:01.310334: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78b00 next 1666 of size 256
2019-09-04 10:04:01.310347: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78c00 next 4867 of size 256
2019-09-04 10:04:01.310358: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78d00 next 773 of size 256
2019-09-04 10:04:01.310370: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78e00 next 1070 of size 256
2019-09-04 10:04:01.310384: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78f00 next 3763 of size 256
2019-09-04 10:04:01.310396: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79000 next 3999 of size 256
2019-09-04 10:04:01.310408: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79100 next 5116 of size 256
2019-09-04 10:04:01.310418: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79200 next 413 of size 256
2019-09-04 10:04:01.310430: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79300 next 2756 of size 256
2019-09-04 10:04:01.310444: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79400 next 3671 of size 256
2019-09-04 10:04:01.310457: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79500 next 2245 of size 256
2019-09-04 10:04:01.310470: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79600 next 1210 of size 256
2019-09-04 10:04:01.310483: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79700 next 4088 of size 256
2019-09-04 10:04:01.310496: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79800 next 4845 of size 256
2019-09-04 10:04:01.310508: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79900 next 3292 of size 256
2019-09-04 10:04:01.310515: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79a00 next 1470 of size 256
2019-09-04 10:04:01.310523: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79b00 next 4112 of size 256
2019-09-04 10:04:01.310531: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79c00 next 5037 of size 256
2019-09-04 10:04:01.310538: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79d00 next 2101 of size 256
2019-09-04 10:04:01.310545: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79e00 next 4886 of size 256
2019-09-04 10:04:01.310554: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79f00 next 2726 of size 256
2019-09-04 10:04:01.310568: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a000 next 4427 of size 256
2019-09-04 10:04:01.310584: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a100 next 4547 of size 256
2019-09-04 10:04:01.310597: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a200 next 1654 of size 256
2019-09-04 10:04:01.310611: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a300 next 1558 of size 256
2019-09-04 10:04:01.310626: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a400 next 1940 of size 256
2019-09-04 10:04:01.310638: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a500 next 2236 of size 256
2019-09-04 10:04:01.310653: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a600 next 2508 of size 256
2019-09-04 10:04:01.310668: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a700 next 3571 of size 256
2019-09-04 10:04:01.310682: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a800 next 3579 of size 256
2019-09-04 10:04:01.310695: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a900 next 4800 of size 256
2019-09-04 10:04:01.310708: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7aa00 next 3962 of size 256
2019-09-04 10:04:01.310721: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7ab00 next 3399 of size 256
2019-09-04 10:04:01.310732: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7ac00 next 278 of size 256
2019-09-04 10:04:01.310744: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7ad00 next 2305 of size 256
2019-09-04 10:04:01.310758: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7ae00 next 4783 of size 256
2019-09-04 10:04:01.310771: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7af00 next 943 of size 256
2019-09-04 10:04:01.310784: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7b000 next 3276 of size 2097152
2019-09-04 10:04:01.310798: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428e7b000 next 5118 of size 2097152
2019-09-04 10:04:01.310811: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242907b000 next 4761 of size 2097152
2019-09-04 10:04:01.310825: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242927b000 next 1591 of size 2097152
2019-09-04 10:04:01.310837: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242947b000 next 1639 of size 2097152
2019-09-04 10:04:01.310848: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242967b000 next 1978 of size 2097152
2019-09-04 10:04:01.310861: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242987b000 next 456 of size 2097152
2019-09-04 10:04:01.310874: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2429a7b000 next 4961 of size 2097152
2019-09-04 10:04:01.310888: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2429c7b000 next 2629 of size 1048576
2019-09-04 10:04:01.310902: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2429d7b000 next 2943 of size 1048576
2019-09-04 10:04:01.310915: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2429e7b000 next 4963 of size 1048576
2019-09-04 10:04:01.310928: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2429f7b000 next 4000 of size 1048576
2019-09-04 10:04:01.310941: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a07b000 next 392 of size 1048576
2019-09-04 10:04:01.310955: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a17b000 next 2828 of size 1048576
2019-09-04 10:04:01.310969: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a27b000 next 1332 of size 1048576
2019-09-04 10:04:01.310982: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a37b000 next 2455 of size 1048576
2019-09-04 10:04:01.310996: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a47b000 next 3905 of size 2097152
2019-09-04 10:04:01.311009: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a67b000 next 4184 of size 2097152
2019-09-04 10:04:01.311021: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a87b000 next 646 of size 2097152
2019-09-04 10:04:01.311033: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242aa7b000 next 1157 of size 2097152
2019-09-04 10:04:01.311046: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f242ac7b000 next 2607 of size 256
2019-09-04 10:04:01.311061: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242ac7b100 next 2590 of size 2097152
2019-09-04 10:04:01.311075: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242ae7b100 next 1492 of size 2097152
2019-09-04 10:04:01.311088: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b07b100 next 3483 of size 2097152
2019-09-04 10:04:01.311101: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b27b100 next 2871 of size 2097152
2019-09-04 10:04:01.311114: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b47b100 next 624 of size 1048576
2019-09-04 10:04:01.311125: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b57b100 next 4272 of size 1048576
2019-09-04 10:04:01.311137: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b67b100 next 4520 of size 1048576
2019-09-04 10:04:01.311149: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b77b100 next 5058 of size 1048576
2019-09-04 10:04:01.311163: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b87b100 next 1219 of size 1048576
2019-09-04 10:04:01.311177: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b97b100 next 2844 of size 1048576
2019-09-04 10:04:01.311191: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242ba7b100 next 3786 of size 1048576
2019-09-04 10:04:01.311205: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242bb7b100 next 3441 of size 1048576
2019-09-04 10:04:01.311217: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f242bc7b100 next 4798 of size 512
2019-09-04 10:04:01.311229: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242bc7b300 next 3834 of size 256
2019-09-04 10:04:01.311242: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242bc7b400 next 499 of size 256
2019-09-04 10:04:01.311256: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242bc7b500 next 4935 of size 256
2019-09-04 10:04:01.311270: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f242bc7b600 next 3946 of size 4096
2019-09-04 10:04:01.311286: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242bc7c600 next 2203 of size 4608
2019-09-04 10:04:01.311300: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f242bc7d800 next 2157 of size 11420160
2019-09-04 10:04:01.311312: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242c761a00 next 4900 of size 2338816
2019-09-04 10:04:01.311323: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242c99ca00 next 1658 of size 2338816
2019-09-04 10:04:01.311334: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f242cbd7a00 next 4097 of size 182720000
2019-09-04 10:04:01.311347: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2437a19000 next 18446744073709551615 of size 207515648
2019-09-04 10:04:01.311364: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 268435456
2019-09-04 10:04:01.311380: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2444000000 next 57 of size 67108864
2019-09-04 10:04:01.311394: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2448000000 next 62 of size 2048
2019-09-04 10:04:01.311407: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2448000800 next 63 of size 8388608
2019-09-04 10:04:01.311419: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2448800800 next 64 of size 1024
2019-09-04 10:04:01.311433: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2448800c00 next 65 of size 4194304
2019-09-04 10:04:01.311447: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2448c00c00 next 66 of size 8388608
2019-09-04 10:04:01.311461: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2449400c00 next 67 of size 8192
2019-09-04 10:04:01.311475: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2449402c00 next 68 of size 8192
2019-09-04 10:04:01.311487: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2449404c00 next 69 of size 256
2019-09-04 10:04:01.311500: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2449404d00 next 70 of size 8388608
2019-09-04 10:04:01.311512: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2449c04d00 next 71 of size 2048
2019-09-04 10:04:01.311527: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2449c05500 next 72 of size 16777216
2019-09-04 10:04:01.311541: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244ac05500 next 73 of size 67108864
2019-09-04 10:04:01.311555: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244ec05500 next 74 of size 8388608
2019-09-04 10:04:01.311568: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f405500 next 75 of size 8192
2019-09-04 10:04:01.311582: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f407500 next 76 of size 4194304
2019-09-04 10:04:01.311595: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f807500 next 77 of size 256
2019-09-04 10:04:01.311608: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f807600 next 78 of size 256
2019-09-04 10:04:01.311621: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f807700 next 79 of size 512
2019-09-04 10:04:01.311636: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f807900 next 80 of size 2048
2019-09-04 10:04:01.311649: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f808100 next 81 of size 256
2019-09-04 10:04:01.311662: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f808200 next 82 of size 204800
2019-09-04 10:04:01.311674: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a200 next 83 of size 256
2019-09-04 10:04:01.311686: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a300 next 84 of size 256
2019-09-04 10:04:01.311700: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a400 next 85 of size 256
2019-09-04 10:04:01.311713: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a500 next 86 of size 256
2019-09-04 10:04:01.311727: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a600 next 87 of size 256
2019-09-04 10:04:01.311742: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a700 next 88 of size 512
2019-09-04 10:04:01.311755: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a900 next 89 of size 512
2019-09-04 10:04:01.311768: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83ab00 next 90 of size 256
2019-09-04 10:04:01.311779: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83ac00 next 91 of size 256
2019-09-04 10:04:01.311790: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83ad00 next 92 of size 256
2019-09-04 10:04:01.311801: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83ae00 next 93 of size 256
2019-09-04 10:04:01.311815: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83af00 next 94 of size 256
2019-09-04 10:04:01.311829: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83b000 next 95 of size 8388608
2019-09-04 10:04:01.311842: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245003b000 next 96 of size 8388608
2019-09-04 10:04:01.311856: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245083b000 next 97 of size 8388608
2019-09-04 10:04:01.311869: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245103b000 next 98 of size 8388608
2019-09-04 10:04:01.311880: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245183b000 next 99 of size 8388608
2019-09-04 10:04:01.311892: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b000 next 100 of size 256
2019-09-04 10:04:01.311903: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b100 next 101 of size 256
2019-09-04 10:04:01.311916: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b200 next 102 of size 256
2019-09-04 10:04:01.312242: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b300 next 103 of size 256
2019-09-04 10:04:01.312281: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b400 next 104 of size 256
2019-09-04 10:04:01.312295: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b500 next 105 of size 256
2019-09-04 10:04:01.312309: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b600 next 106 of size 256
2019-09-04 10:04:01.312323: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b700 next 107 of size 8388608
2019-09-04 10:04:01.312336: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245283b700 next 108 of size 8192
2019-09-04 10:04:01.312349: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245283d700 next 109 of size 8192
2019-09-04 10:04:01.312362: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245283f700 next 110 of size 8192
2019-09-04 10:04:01.312376: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2452841700 next 111 of size 8192
2019-09-04 10:04:01.312389: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2452843700 next 112 of size 256
2019-09-04 10:04:01.312403: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2452843800 next 113 of size 8192
2019-09-04 10:04:01.312417: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2452845800 next 114 of size 8192
2019-09-04 10:04:01.312430: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2452847800 next 115 of size 4194304
2019-09-04 10:04:01.312443: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2452c47800 next 116 of size 4194304
2019-09-04 10:04:01.312455: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2453047800 next 117 of size 4194304
2019-09-04 10:04:01.312469: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2453447800 next 118 of size 4194304
2019-09-04 10:04:01.312485: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2453847800 next 18446744073709551615 of size 8095744
2019-09-04 10:04:01.312501: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 134217728
2019-09-04 10:04:01.312515: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2464000000 next 53 of size 67108864
2019-09-04 10:04:01.312528: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2468000000 next 18446744073709551615 of size 67108864
2019-09-04 10:04:01.312540: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 67108864
2019-09-04 10:04:01.312554: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2478000000 next 40 of size 16777216
2019-09-04 10:04:01.312568: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2479000000 next 43 of size 8388608
2019-09-04 10:04:01.312582: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2479800000 next 44 of size 8388608
2019-09-04 10:04:01.312595: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a000000 next 45 of size 8192
2019-09-04 10:04:01.312608: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a002000 next 46 of size 256
2019-09-04 10:04:01.312620: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a002100 next 47 of size 4194304
2019-09-04 10:04:01.312632: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a402100 next 48 of size 256
2019-09-04 10:04:01.312645: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a402200 next 49 of size 256
2019-09-04 10:04:01.312660: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a402300 next 50 of size 65536
2019-09-04 10:04:01.312674: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a412300 next 51 of size 196608
2019-09-04 10:04:01.312687: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a442300 next 54 of size 8192
2019-09-04 10:04:01.312699: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a444300 next 55 of size 8388608
2019-09-04 10:04:01.312710: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247ac44300 next 58 of size 65536
2019-09-04 10:04:01.312722: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247ac54300 next 59 of size 4096
2019-09-04 10:04:01.312734: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247ac55300 next 60 of size 65536
2019-09-04 10:04:01.312747: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247ac65300 next 61 of size 65536
2019-09-04 10:04:01.312761: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247ac75300 next 18446744073709551615 of size 20491520
2019-09-04 10:04:01.312775: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 33554432
2019-09-04 10:04:01.312788: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24be000000 next 32 of size 8388608
2019-09-04 10:04:01.312802: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24be800000 next 35 of size 8388608
2019-09-04 10:04:01.312814: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24bf000000 next 18446744073709551615 of size 16777216
2019-09-04 10:04:01.312825: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 16777216
2019-09-04 10:04:01.312841: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da000000 next 24 of size 4194304
2019-09-04 10:04:01.312855: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da400000 next 26 of size 256
2019-09-04 10:04:01.312868: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da400100 next 27 of size 12288
2019-09-04 10:04:01.312881: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da403100 next 28 of size 8192
2019-09-04 10:04:01.312893: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da405100 next 29 of size 4194304
2019-09-04 10:04:01.312904: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da805100 next 30 of size 2048
2019-09-04 10:04:01.312917: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da805900 next 33 of size 16896
2019-09-04 10:04:01.312932: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da809b00 next 34 of size 8192
2019-09-04 10:04:01.312946: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da80bb00 next 36 of size 2048
2019-09-04 10:04:01.312959: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da80c300 next 37 of size 81920
2019-09-04 10:04:01.312972: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da820300 next 38 of size 256
2019-09-04 10:04:01.312983: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da820400 next 41 of size 4096
2019-09-04 10:04:01.312994: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da821400 next 42 of size 32768
2019-09-04 10:04:01.313007: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da829400 next 18446744073709551615 of size 8219648
2019-09-04 10:04:01.313020: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1048576
2019-09-04 10:04:01.313035: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400000 next 1 of size 1280
2019-09-04 10:04:01.313049: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400500 next 2 of size 256
2019-09-04 10:04:01.313062: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400600 next 3 of size 256
2019-09-04 10:04:01.313073: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400700 next 4 of size 256
2019-09-04 10:04:01.313085: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400800 next 5 of size 256
2019-09-04 10:04:01.313097: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400900 next 6 of size 256
2019-09-04 10:04:01.313111: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400a00 next 7 of size 256
2019-09-04 10:04:01.313124: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400b00 next 8 of size 512
2019-09-04 10:04:01.313137: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400d00 next 10 of size 256
2019-09-04 10:04:01.313150: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400e00 next 11 of size 256
2019-09-04 10:04:01.313162: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400f00 next 12 of size 8192
2019-09-04 10:04:01.313175: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549402f00 next 16 of size 204800
2019-09-04 10:04:01.313189: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549434f00 next 17 of size 256
2019-09-04 10:04:01.313203: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549435000 next 18 of size 512
2019-09-04 10:04:01.313217: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549435200 next 19 of size 8192
2019-09-04 10:04:01.313230: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549437200 next 20 of size 8192
2019-09-04 10:04:01.313242: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549439200 next 21 of size 256
2019-09-04 10:04:01.313254: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549439300 next 22 of size 256
2019-09-04 10:04:01.313267: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549439400 next 25 of size 512
2019-09-04 10:04:01.313282: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549439600 next 18446744073709551615 of size 813568
2019-09-04 10:04:01.313295: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 4194304
2019-09-04 10:04:01.313309: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f254b200000 next 18446744073709551615 of size 4194304
2019-09-04 10:04:01.313322: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 4194304
2019-09-04 10:04:01.313334: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f254b600000 next 18446744073709551615 of size 4194304
2019-09-04 10:04:01.313345: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 8388608
2019-09-04 10:04:01.313360: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2553000000 next 15 of size 4194304
2019-09-04 10:04:01.313374: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2553400000 next 18446744073709551615 of size 4194304
2019-09-04 10:04:01.313387: I tensorflow/core/common_runtime/bfc_allocator.cc:809] Summary of in-use Chunks by size:
2019-09-04 10:04:01.313413: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 199 Chunks of size 256 totalling 49.8KiB
2019-09-04 10:04:01.313427: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 7 Chunks of size 512 totalling 3.5KiB
2019-09-04 10:04:01.313440: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1024 totalling 1.0KiB
2019-09-04 10:04:01.313455: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280 totalling 1.2KiB
2019-09-04 10:04:01.313471: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 23 Chunks of size 2048 totalling 46.0KiB
2019-09-04 10:04:01.313485: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 2 Chunks of size 4096 totalling 8.0KiB
2019-09-04 10:04:01.313498: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 4608 totalling 4.5KiB
2019-09-04 10:04:01.313512: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 18 Chunks of size 8192 totalling 144.0KiB
2019-09-04 10:04:01.313525: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 12288 totalling 12.0KiB
2019-09-04 10:04:01.313540: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 16896 totalling 16.5KiB
2019-09-04 10:04:01.313555: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 32768 totalling 32.0KiB
2019-09-04 10:04:01.313570: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 65536 totalling 256.0KiB
2019-09-04 10:04:01.313585: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 81920 totalling 80.0KiB
2019-09-04 10:04:01.313598: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 196608 totalling 192.0KiB
2019-09-04 10:04:01.313611: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 204800 totalling 800.0KiB
2019-09-04 10:04:01.313626: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 813568 totalling 794.5KiB
2019-09-04 10:04:01.313642: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 32 Chunks of size 1048576 totalling 32.00MiB
2019-09-04 10:04:01.313657: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 32 Chunks of size 2097152 totalling 64.00MiB
2019-09-04 10:04:01.313670: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 2 Chunks of size 2338816 totalling 4.46MiB
2019-09-04 10:04:01.313682: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 16 Chunks of size 4194304 totalling 64.00MiB
2019-09-04 10:04:01.313694: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 8095744 totalling 7.72MiB
2019-09-04 10:04:01.313707: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 8219648 totalling 7.84MiB
2019-09-04 10:04:01.313722: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 17 Chunks of size 8388608 totalling 136.00MiB
2019-09-04 10:04:01.313738: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 16777216 totalling 48.00MiB
2019-09-04 10:04:01.313753: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 20491520 totalling 19.54MiB
2019-09-04 10:04:01.313766: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 67108864 totalling 256.00MiB
2019-09-04 10:04:01.313779: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 207515648 totalling 197.90MiB
2019-09-04 10:04:01.313793: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 839.85MiB
2019-09-04 10:04:01.313806: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 4296015872 memory_limit_: 10979036365 available bytes: 6683020493 curr_region_allocation_bytes_: 4294967296
2019-09-04 10:04:01.313829: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
Limit: 10979036365
InUse: 880645376
MaxInUse: 3286797056
NumAllocs: 5943743
MaxAllocSize: 1107755008
2019-09-04 10:04:01.314288: W tensorflow/core/common_runtime/bfc_allocator.cc:319] __________________________________________________________________________*****___******************
2019-09-04 10:04:01.314450: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at conv_ops.cc:486 : Resource exhausted: OOM when allocating tensor with shape[1142,1024,50,44] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-09-04 10:04:11.583601: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.50GiB (rounded to 2689638400). Current allocation summary follows.
2019-09-04 10:04:11.583934: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): Total Chunks: 200, Chunks in use: 199. 50.0KiB allocated for chunks. 49.8KiB in use in bin. 2.6KiB client-requested in use in bin.
2019-09-04 10:04:11.584147: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): Total Chunks: 8, Chunks in use: 7. 4.0KiB allocated for chunks. 3.5KiB in use in bin. 2.4KiB client-requested in use in bin.
2019-09-04 10:04:11.584269: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): Total Chunks: 2, Chunks in use: 2. 2.2KiB allocated for chunks. 2.2KiB in use in bin. 2.0KiB client-requested in use in bin.
2019-09-04 10:04:11.584295: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): Total Chunks: 23, Chunks in use: 23. 46.0KiB allocated for chunks. 46.0KiB in use in bin. 45.0KiB client-requested in use in bin.
2019-09-04 10:04:11.584311: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): Total Chunks: 4, Chunks in use: 3. 16.5KiB allocated for chunks. 12.5KiB in use in bin. 12.5KiB client-requested in use in bin.
2019-09-04 10:04:11.584394: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): Total Chunks: 19, Chunks in use: 19. 156.0KiB allocated for chunks. 156.0KiB in use in bin. 156.0KiB client-requested in use in bin.
2019-09-04 10:04:11.584412: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): Total Chunks: 1, Chunks in use: 1. 16.5KiB allocated for chunks. 16.5KiB in use in bin. 16.3KiB client-requested in use in bin.
2019-09-04 10:04:11.584422: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): Total Chunks: 1, Chunks in use: 1. 32.0KiB allocated for chunks. 32.0KiB in use in bin. 32.0KiB client-requested in use in bin.
2019-09-04 10:04:11.584432: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): Total Chunks: 5, Chunks in use: 5. 336.0KiB allocated for chunks. 336.0KiB in use in bin. 336.0KiB client-requested in use in bin.
2019-09-04 10:04:11.584441: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): Total Chunks: 5, Chunks in use: 5. 992.0KiB allocated for chunks. 992.0KiB in use in bin. 992.0KiB client-requested in use in bin.
2019-09-04 10:04:11.584450: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-09-04 10:04:11.584460: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): Total Chunks: 1, Chunks in use: 1. 794.5KiB allocated for chunks. 794.5KiB in use in bin. 448.0KiB client-requested in use in bin.
2019-09-04 10:04:11.584469: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): Total Chunks: 32, Chunks in use: 32. 32.00MiB allocated for chunks. 32.00MiB in use in bin. 32.00MiB client-requested in use in bin.
2019-09-04 10:04:11.584479: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): Total Chunks: 34, Chunks in use: 34. 68.46MiB allocated for chunks. 68.46MiB in use in bin. 68.46MiB client-requested in use in bin.
2019-09-04 10:04:11.584489: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): Total Chunks: 18, Chunks in use: 18. 79.56MiB allocated for chunks. 79.56MiB in use in bin. 72.00MiB client-requested in use in bin.
2019-09-04 10:04:11.584499: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): Total Chunks: 18, Chunks in use: 17. 146.89MiB allocated for chunks. 136.00MiB in use in bin. 136.00MiB client-requested in use in bin.
2019-09-04 10:04:11.584532: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): Total Chunks: 4, Chunks in use: 4. 67.54MiB allocated for chunks. 67.54MiB in use in bin. 64.00MiB client-requested in use in bin.
2019-09-04 10:04:11.584550: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-09-04 10:04:11.584564: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): Total Chunks: 4, Chunks in use: 4. 256.00MiB allocated for chunks. 256.00MiB in use in bin. 256.00MiB client-requested in use in bin.
2019-09-04 10:04:11.584574: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): Total Chunks: 2, Chunks in use: 1. 372.16MiB allocated for chunks. 197.90MiB in use in bin. 174.25MiB client-requested in use in bin.
2019-09-04 10:04:11.584584: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): Total Chunks: 4, Chunks in use: 1. 9.22GiB allocated for chunks. 4.90GiB in use in bin. 4.90GiB client-requested in use in bin.
2019-09-04 10:04:11.584599: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 2.50GiB was 256.00MiB, Chunk State:
2019-09-04 10:04:11.584615: I tensorflow/core/common_runtime/bfc_allocator.cc:786] Size: 1.00GiB | Requested Size: 940.98MiB | in_use: 0 | bin_num: 20
2019-09-04 10:04:11.584631: I tensorflow/core/common_runtime/bfc_allocator.cc:786] Size: 1.32GiB | Requested Size: 128B | in_use: 0 | bin_num: 20, prev: Size: 4.90GiB | Requested Size: 4.90GiB | in_use: 1 | bin_num: -1
2019-09-04 10:04:11.584645: I tensorflow/core/common_runtime/bfc_allocator.cc:786] Size: 2.00GiB | Requested Size: 528.95MiB | in_use: 0 | bin_num: 20
2019-09-04 10:04:11.584659: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 6683020288
2019-09-04 10:04:11.584683: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f21b4000000 next 4517 of size 5262336000
2019-09-04 10:04:11.584698: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f22eda8e000 next 18446744073709551615 of size 1420684288
2019-09-04 10:04:11.584709: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 2147483648
2019-09-04 10:04:11.584721: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f2344000000 next 18446744073709551615 of size 2147483648
2019-09-04 10:04:11.584733: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1073741824
2019-09-04 10:04:11.584756: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f23c4000000 next 18446744073709551615 of size 1073741824
2019-09-04 10:04:11.584771: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 536870912
2019-09-04 10:04:11.584789: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424000000 next 120 of size 4194304
2019-09-04 10:04:11.584810: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424400000 next 121 of size 256
2019-09-04 10:04:11.584825: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424400100 next 122 of size 256
2019-09-04 10:04:11.584850: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424400200 next 123 of size 2048
2019-09-04 10:04:11.584869: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424400a00 next 124 of size 256
2019-09-04 10:04:11.584885: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424400b00 next 125 of size 8192
2019-09-04 10:04:11.584906: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424402b00 next 126 of size 4194304
2019-09-04 10:04:11.584923: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424802b00 next 127 of size 256
2019-09-04 10:04:11.584939: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424802c00 next 128 of size 2048
2019-09-04 10:04:11.584957: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424803400 next 129 of size 8192
2019-09-04 10:04:11.584976: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805400 next 130 of size 256
2019-09-04 10:04:11.584994: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805500 next 131 of size 256
2019-09-04 10:04:11.585015: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805600 next 132 of size 256
2019-09-04 10:04:11.585030: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805700 next 133 of size 256
2019-09-04 10:04:11.585046: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805800 next 134 of size 256
2019-09-04 10:04:11.585066: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805900 next 135 of size 256
2019-09-04 10:04:11.585083: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2424805a00 next 136 of size 8388608
2019-09-04 10:04:11.585102: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425005a00 next 137 of size 8388608
2019-09-04 10:04:11.585120: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425805a00 next 138 of size 256
2019-09-04 10:04:11.585136: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425805b00 next 139 of size 256
2019-09-04 10:04:11.585156: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425805c00 next 140 of size 256
2019-09-04 10:04:11.585175: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425805d00 next 141 of size 204800
2019-09-04 10:04:11.585198: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425837d00 next 142 of size 204800
2019-09-04 10:04:11.585210: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425869d00 next 143 of size 4194304
2019-09-04 10:04:11.585220: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c69d00 next 144 of size 256
2019-09-04 10:04:11.585229: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c69e00 next 145 of size 256
2019-09-04 10:04:11.585238: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c69f00 next 146 of size 256
2019-09-04 10:04:11.585247: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a000 next 147 of size 256
2019-09-04 10:04:11.585256: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a100 next 148 of size 256
2019-09-04 10:04:11.585264: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a200 next 149 of size 256
2019-09-04 10:04:11.585273: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a300 next 150 of size 256
2019-09-04 10:04:11.585282: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a400 next 151 of size 256
2019-09-04 10:04:11.585291: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a500 next 152 of size 256
2019-09-04 10:04:11.585299: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a600 next 153 of size 256
2019-09-04 10:04:11.585309: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a700 next 154 of size 256
2019-09-04 10:04:11.585317: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a800 next 155 of size 256
2019-09-04 10:04:11.585326: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425c6a900 next 156 of size 2097152
2019-09-04 10:04:11.585335: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2425e6a900 next 157 of size 2097152
2019-09-04 10:04:11.585344: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242606a900 next 158 of size 2097152
2019-09-04 10:04:11.585354: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242626a900 next 159 of size 2097152
2019-09-04 10:04:11.585362: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646a900 next 160 of size 256
2019-09-04 10:04:11.585370: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646aa00 next 161 of size 256
2019-09-04 10:04:11.585378: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646ab00 next 162 of size 2048
2019-09-04 10:04:11.585388: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646b300 next 163 of size 2048
2019-09-04 10:04:11.585396: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646bb00 next 164 of size 2048
2019-09-04 10:04:11.585406: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646c300 next 165 of size 2048
2019-09-04 10:04:11.585414: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242646cb00 next 166 of size 1048576
2019-09-04 10:04:11.585423: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242656cb00 next 167 of size 1048576
2019-09-04 10:04:11.585432: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242666cb00 next 168 of size 1048576
2019-09-04 10:04:11.585441: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242676cb00 next 169 of size 1048576
2019-09-04 10:04:11.585452: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242686cb00 next 170 of size 512
2019-09-04 10:04:11.585462: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242686cd00 next 171 of size 2048
2019-09-04 10:04:11.585471: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242686d500 next 172 of size 2048
2019-09-04 10:04:11.585479: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242686dd00 next 173 of size 2048
2019-09-04 10:04:11.585489: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242686e500 next 174 of size 2048
2019-09-04 10:04:11.585496: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242686ed00 next 175 of size 1048576
2019-09-04 10:04:11.585504: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242696ed00 next 176 of size 1048576
2019-09-04 10:04:11.585513: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2426a6ed00 next 177 of size 1048576
2019-09-04 10:04:11.585521: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2426b6ed00 next 178 of size 1048576
2019-09-04 10:04:11.585531: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2426c6ed00 next 179 of size 2097152
2019-09-04 10:04:11.585539: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2426e6ed00 next 180 of size 2097152
2019-09-04 10:04:11.585548: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242706ed00 next 181 of size 2097152
2019-09-04 10:04:11.585557: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242726ed00 next 182 of size 2097152
2019-09-04 10:04:11.585572: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242746ed00 next 183 of size 2048
2019-09-04 10:04:11.585588: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242746f500 next 184 of size 2048
2019-09-04 10:04:11.585604: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242746fd00 next 185 of size 2048
2019-09-04 10:04:11.585622: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427470500 next 186 of size 2048
2019-09-04 10:04:11.585638: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427470d00 next 187 of size 1048576
2019-09-04 10:04:11.585653: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427570d00 next 188 of size 1048576
2019-09-04 10:04:11.585664: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427670d00 next 189 of size 1048576
2019-09-04 10:04:11.585672: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427770d00 next 190 of size 1048576
2019-09-04 10:04:11.585680: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427870d00 next 191 of size 2097152
2019-09-04 10:04:11.585689: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427a70d00 next 192 of size 2097152
2019-09-04 10:04:11.585698: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427c70d00 next 193 of size 2097152
2019-09-04 10:04:11.585714: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2427e70d00 next 194 of size 2097152
2019-09-04 10:04:11.585727: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428070d00 next 195 of size 2097152
2019-09-04 10:04:11.585735: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428270d00 next 196 of size 2097152
2019-09-04 10:04:11.585742: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428470d00 next 197 of size 2097152
2019-09-04 10:04:11.585749: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428670d00 next 198 of size 2097152
2019-09-04 10:04:11.585756: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428870d00 next 199 of size 2048
2019-09-04 10:04:11.585763: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428871500 next 200 of size 2048
2019-09-04 10:04:11.585770: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428871d00 next 201 of size 2048
2019-09-04 10:04:11.585777: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428872500 next 202 of size 2048
2019-09-04 10:04:11.585784: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428872d00 next 203 of size 1048576
2019-09-04 10:04:11.585790: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428972d00 next 204 of size 1048576
2019-09-04 10:04:11.585797: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428a72d00 next 205 of size 1048576
2019-09-04 10:04:11.585804: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428b72d00 next 206 of size 1048576
2019-09-04 10:04:11.585811: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c72d00 next 207 of size 256
2019-09-04 10:04:11.585818: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c72e00 next 208 of size 256
2019-09-04 10:04:11.585826: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c72f00 next 209 of size 256
2019-09-04 10:04:11.585848: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73000 next 210 of size 256
2019-09-04 10:04:11.585868: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73100 next 211 of size 256
2019-09-04 10:04:11.585884: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73200 next 212 of size 256
2019-09-04 10:04:11.585900: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73300 next 213 of size 256
2019-09-04 10:04:11.585917: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73400 next 214 of size 256
2019-09-04 10:04:11.585941: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73500 next 215 of size 256
2019-09-04 10:04:11.585960: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73600 next 216 of size 256
2019-09-04 10:04:11.585990: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73700 next 217 of size 256
2019-09-04 10:04:11.586011: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73800 next 218 of size 256
2019-09-04 10:04:11.586028: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73900 next 219 of size 256
2019-09-04 10:04:11.586049: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73a00 next 220 of size 256
2019-09-04 10:04:11.586069: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73b00 next 221 of size 256
2019-09-04 10:04:11.586090: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73c00 next 222 of size 256
2019-09-04 10:04:11.586108: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73d00 next 223 of size 256
2019-09-04 10:04:11.586125: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73e00 next 224 of size 256
2019-09-04 10:04:11.586144: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c73f00 next 225 of size 256
2019-09-04 10:04:11.586163: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74000 next 226 of size 256
2019-09-04 10:04:11.586184: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74100 next 227 of size 256
2019-09-04 10:04:11.586202: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74200 next 228 of size 256
2019-09-04 10:04:11.586220: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74300 next 229 of size 256
2019-09-04 10:04:11.586239: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74400 next 230 of size 256
2019-09-04 10:04:11.586258: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74500 next 231 of size 256
2019-09-04 10:04:11.586278: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74600 next 232 of size 256
2019-09-04 10:04:11.586296: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74700 next 233 of size 256
2019-09-04 10:04:11.586315: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74800 next 234 of size 256
2019-09-04 10:04:11.586337: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74900 next 235 of size 256
2019-09-04 10:04:11.586355: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74a00 next 236 of size 256
2019-09-04 10:04:11.586371: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74b00 next 237 of size 256
2019-09-04 10:04:11.586389: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74c00 next 238 of size 256
2019-09-04 10:04:11.586408: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74d00 next 239 of size 256
2019-09-04 10:04:11.586427: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74e00 next 240 of size 256
2019-09-04 10:04:11.586447: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c74f00 next 241 of size 256
2019-09-04 10:04:11.586462: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75000 next 242 of size 256
2019-09-04 10:04:11.586480: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75100 next 243 of size 256
2019-09-04 10:04:11.586502: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75200 next 244 of size 256
2019-09-04 10:04:11.586524: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75300 next 245 of size 256
2019-09-04 10:04:11.586539: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75400 next 246 of size 256
2019-09-04 10:04:11.586550: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75500 next 247 of size 256
2019-09-04 10:04:11.586559: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75600 next 248 of size 256
2019-09-04 10:04:11.586568: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75700 next 249 of size 256
2019-09-04 10:04:11.586581: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75800 next 250 of size 256
2019-09-04 10:04:11.586593: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75900 next 251 of size 256
2019-09-04 10:04:11.586605: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75a00 next 252 of size 256
2019-09-04 10:04:11.586619: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75b00 next 253 of size 256
2019-09-04 10:04:11.586633: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75c00 next 254 of size 256
2019-09-04 10:04:11.586647: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75d00 next 255 of size 256
2019-09-04 10:04:11.586660: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75e00 next 256 of size 256
2019-09-04 10:04:11.586672: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c75f00 next 257 of size 256
2019-09-04 10:04:11.586681: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76000 next 258 of size 256
2019-09-04 10:04:11.586693: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76100 next 259 of size 256
2019-09-04 10:04:11.586707: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76200 next 260 of size 256
2019-09-04 10:04:11.586720: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76300 next 261 of size 256
2019-09-04 10:04:11.586734: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76400 next 262 of size 256
2019-09-04 10:04:11.586744: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76500 next 263 of size 256
2019-09-04 10:04:11.586756: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76600 next 264 of size 256
2019-09-04 10:04:11.586768: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76700 next 265 of size 256
2019-09-04 10:04:11.586782: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76800 next 266 of size 256
2019-09-04 10:04:11.586796: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76900 next 267 of size 256
2019-09-04 10:04:11.586809: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76a00 next 268 of size 256
2019-09-04 10:04:11.586820: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76b00 next 269 of size 256
2019-09-04 10:04:11.586831: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76c00 next 270 of size 256
2019-09-04 10:04:11.586844: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76d00 next 271 of size 256
2019-09-04 10:04:11.586854: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76e00 next 272 of size 256
2019-09-04 10:04:11.586867: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c76f00 next 273 of size 256
2019-09-04 10:04:11.586878: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77000 next 2713 of size 256
2019-09-04 10:04:11.586891: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77100 next 3471 of size 256
2019-09-04 10:04:11.586904: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77200 next 3575 of size 256
2019-09-04 10:04:11.586917: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77300 next 2265 of size 256
2019-09-04 10:04:11.586928: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77400 next 1127 of size 256
2019-09-04 10:04:11.586939: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77500 next 2772 of size 256
2019-09-04 10:04:11.586953: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77600 next 1975 of size 256
2019-09-04 10:04:11.586964: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77700 next 2690 of size 256
2019-09-04 10:04:11.586975: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77800 next 673 of size 256
2019-09-04 10:04:11.586987: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77900 next 3121 of size 256
2019-09-04 10:04:11.587000: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77a00 next 2802 of size 256
2019-09-04 10:04:11.587013: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77b00 next 2764 of size 256
2019-09-04 10:04:11.587025: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77c00 next 380 of size 256
2019-09-04 10:04:11.587036: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77d00 next 4360 of size 256
2019-09-04 10:04:11.587049: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77e00 next 2592 of size 256
2019-09-04 10:04:11.587062: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c77f00 next 2588 of size 256
2019-09-04 10:04:11.587075: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78000 next 4795 of size 256
2019-09-04 10:04:11.587084: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78100 next 5023 of size 256
2019-09-04 10:04:11.587095: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78200 next 4579 of size 256
2019-09-04 10:04:11.587110: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78300 next 3630 of size 256
2019-09-04 10:04:11.587123: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78400 next 1171 of size 256
2019-09-04 10:04:11.587135: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78500 next 452 of size 256
2019-09-04 10:04:11.587145: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78600 next 905 of size 256
2019-09-04 10:04:11.587158: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78700 next 1584 of size 256
2019-09-04 10:04:11.587171: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78800 next 1764 of size 256
2019-09-04 10:04:11.587184: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78900 next 3939 of size 256
2019-09-04 10:04:11.587192: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78a00 next 3634 of size 256
2019-09-04 10:04:11.587213: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78b00 next 1666 of size 256
2019-09-04 10:04:11.587232: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78c00 next 4867 of size 256
2019-09-04 10:04:11.587247: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78d00 next 773 of size 256
2019-09-04 10:04:11.587264: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78e00 next 1070 of size 256
2019-09-04 10:04:11.587285: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c78f00 next 3763 of size 256
2019-09-04 10:04:11.587302: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79000 next 3999 of size 256
2019-09-04 10:04:11.587320: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79100 next 5116 of size 256
2019-09-04 10:04:11.587341: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79200 next 413 of size 256
2019-09-04 10:04:11.587356: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79300 next 2756 of size 256
2019-09-04 10:04:11.587374: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79400 next 3671 of size 256
2019-09-04 10:04:11.587393: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79500 next 2245 of size 256
2019-09-04 10:04:11.587408: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79600 next 1210 of size 256
2019-09-04 10:04:11.587425: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79700 next 4088 of size 256
2019-09-04 10:04:11.587443: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79800 next 4845 of size 256
2019-09-04 10:04:11.587461: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79900 next 3292 of size 256
2019-09-04 10:04:11.587480: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79a00 next 1470 of size 256
2019-09-04 10:04:11.587499: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79b00 next 4112 of size 256
2019-09-04 10:04:11.587515: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79c00 next 5037 of size 256
2019-09-04 10:04:11.587533: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79d00 next 2101 of size 256
2019-09-04 10:04:11.587552: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79e00 next 4886 of size 256
2019-09-04 10:04:11.587568: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c79f00 next 2726 of size 256
2019-09-04 10:04:11.587587: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a000 next 4427 of size 256
2019-09-04 10:04:11.587607: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a100 next 4547 of size 256
2019-09-04 10:04:11.587620: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a200 next 1654 of size 256
2019-09-04 10:04:11.587633: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a300 next 1558 of size 256
2019-09-04 10:04:11.587643: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a400 next 1940 of size 256
2019-09-04 10:04:11.587655: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a500 next 2236 of size 256
2019-09-04 10:04:11.587666: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a600 next 2508 of size 256
2019-09-04 10:04:11.587681: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a700 next 3571 of size 256
2019-09-04 10:04:11.587694: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a800 next 3579 of size 256
2019-09-04 10:04:11.587706: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7a900 next 4800 of size 256
2019-09-04 10:04:11.587716: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7aa00 next 3962 of size 256
2019-09-04 10:04:11.587729: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7ab00 next 3399 of size 256
2019-09-04 10:04:11.587743: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7ac00 next 278 of size 256
2019-09-04 10:04:11.587752: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7ad00 next 2305 of size 256
2019-09-04 10:04:11.587763: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7ae00 next 4783 of size 256
2019-09-04 10:04:11.587775: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7af00 next 943 of size 256
2019-09-04 10:04:11.587789: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428c7b000 next 3276 of size 2097152
2019-09-04 10:04:11.587801: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2428e7b000 next 5118 of size 2097152
2019-09-04 10:04:11.587812: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242907b000 next 4761 of size 2097152
2019-09-04 10:04:11.587823: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242927b000 next 1591 of size 2097152
2019-09-04 10:04:11.587837: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242947b000 next 1639 of size 2097152
2019-09-04 10:04:11.587850: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242967b000 next 1978 of size 2097152
2019-09-04 10:04:11.587860: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242987b000 next 456 of size 2097152
2019-09-04 10:04:11.587873: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2429a7b000 next 4961 of size 2097152
2019-09-04 10:04:11.587893: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2429c7b000 next 2629 of size 1048576
2019-09-04 10:04:11.587912: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2429d7b000 next 2943 of size 1048576
2019-09-04 10:04:11.587929: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2429e7b000 next 4963 of size 1048576
2019-09-04 10:04:11.587945: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2429f7b000 next 4000 of size 1048576
2019-09-04 10:04:11.587966: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a07b000 next 392 of size 1048576
2019-09-04 10:04:11.587985: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a17b000 next 2828 of size 1048576
2019-09-04 10:04:11.588004: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a27b000 next 1332 of size 1048576
2019-09-04 10:04:11.588021: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a37b000 next 2455 of size 1048576
2019-09-04 10:04:11.588038: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a47b000 next 3905 of size 2097152
2019-09-04 10:04:11.588058: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a67b000 next 4184 of size 2097152
2019-09-04 10:04:11.588080: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242a87b000 next 646 of size 2097152
2019-09-04 10:04:11.588095: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242aa7b000 next 1157 of size 2097152
2019-09-04 10:04:11.588109: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f242ac7b000 next 2607 of size 256
2019-09-04 10:04:11.588123: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242ac7b100 next 2590 of size 2097152
2019-09-04 10:04:11.588137: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242ae7b100 next 1492 of size 2097152
2019-09-04 10:04:11.588151: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b07b100 next 3483 of size 2097152
2019-09-04 10:04:11.588165: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b27b100 next 2871 of size 2097152
2019-09-04 10:04:11.588178: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b47b100 next 624 of size 1048576
2019-09-04 10:04:11.588191: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b57b100 next 4272 of size 1048576
2019-09-04 10:04:11.588203: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b67b100 next 4520 of size 1048576
2019-09-04 10:04:11.588214: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b77b100 next 5058 of size 1048576
2019-09-04 10:04:11.588228: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b87b100 next 1219 of size 1048576
2019-09-04 10:04:11.588242: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242b97b100 next 2844 of size 1048576
2019-09-04 10:04:11.588255: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242ba7b100 next 3786 of size 1048576
2019-09-04 10:04:11.588268: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242bb7b100 next 3441 of size 1048576
2019-09-04 10:04:11.588281: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f242bc7b100 next 4798 of size 512
2019-09-04 10:04:11.588293: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242bc7b300 next 3834 of size 256
2019-09-04 10:04:11.588307: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242bc7b400 next 499 of size 256
2019-09-04 10:04:11.588318: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242bc7b500 next 4935 of size 256
2019-09-04 10:04:11.588339: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f242bc7b600 next 3946 of size 4096
2019-09-04 10:04:11.588360: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242bc7c600 next 2203 of size 4608
2019-09-04 10:04:11.588382: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f242bc7d800 next 2157 of size 11420160
2019-09-04 10:04:11.588405: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242c761a00 next 4900 of size 2338816
2019-09-04 10:04:11.588427: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f242c99ca00 next 1658 of size 2338816
2019-09-04 10:04:11.588445: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f242cbd7a00 next 4097 of size 182720000
2019-09-04 10:04:11.588466: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2437a19000 next 18446744073709551615 of size 207515648
2019-09-04 10:04:11.588481: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 268435456
2019-09-04 10:04:11.588495: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2444000000 next 57 of size 67108864
2019-09-04 10:04:11.588508: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2448000000 next 62 of size 2048
2019-09-04 10:04:11.588522: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2448000800 next 63 of size 8388608
2019-09-04 10:04:11.588534: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2448800800 next 64 of size 1024
2019-09-04 10:04:11.588545: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2448800c00 next 65 of size 4194304
2019-09-04 10:04:11.588556: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2448c00c00 next 66 of size 8388608
2019-09-04 10:04:11.588568: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2449400c00 next 67 of size 8192
2019-09-04 10:04:11.588582: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2449402c00 next 68 of size 8192
2019-09-04 10:04:11.588596: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2449404c00 next 69 of size 256
2019-09-04 10:04:11.588607: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2449404d00 next 70 of size 8388608
2019-09-04 10:04:11.588619: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2449c04d00 next 71 of size 2048
2019-09-04 10:04:11.588632: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2449c05500 next 72 of size 16777216
2019-09-04 10:04:11.588643: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244ac05500 next 73 of size 67108864
2019-09-04 10:04:11.588656: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244ec05500 next 74 of size 8388608
2019-09-04 10:04:11.588664: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f405500 next 75 of size 8192
2019-09-04 10:04:11.588672: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f407500 next 76 of size 4194304
2019-09-04 10:04:11.588684: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f807500 next 77 of size 256
2019-09-04 10:04:11.588695: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f807600 next 78 of size 256
2019-09-04 10:04:11.588708: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f807700 next 79 of size 512
2019-09-04 10:04:11.588721: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f807900 next 80 of size 2048
2019-09-04 10:04:11.588736: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f808100 next 81 of size 256
2019-09-04 10:04:11.588749: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f808200 next 82 of size 204800
2019-09-04 10:04:11.588762: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a200 next 83 of size 256
2019-09-04 10:04:11.588773: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a300 next 84 of size 256
2019-09-04 10:04:11.588789: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a400 next 85 of size 256
2019-09-04 10:04:11.588803: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a500 next 86 of size 256
2019-09-04 10:04:11.588817: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a600 next 87 of size 256
2019-09-04 10:04:11.588831: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a700 next 88 of size 512
2019-09-04 10:04:11.588844: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83a900 next 89 of size 512
2019-09-04 10:04:11.588857: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83ab00 next 90 of size 256
2019-09-04 10:04:11.588869: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83ac00 next 91 of size 256
2019-09-04 10:04:11.588883: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83ad00 next 92 of size 256
2019-09-04 10:04:11.588897: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83ae00 next 93 of size 256
2019-09-04 10:04:11.588911: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83af00 next 94 of size 256
2019-09-04 10:04:11.588924: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f244f83b000 next 95 of size 8388608
2019-09-04 10:04:11.588936: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245003b000 next 96 of size 8388608
2019-09-04 10:04:11.588947: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245083b000 next 97 of size 8388608
2019-09-04 10:04:11.588956: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245103b000 next 98 of size 8388608
2019-09-04 10:04:11.588969: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245183b000 next 99 of size 8388608
2019-09-04 10:04:11.588983: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b000 next 100 of size 256
2019-09-04 10:04:11.588997: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b100 next 101 of size 256
2019-09-04 10:04:11.589010: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b200 next 102 of size 256
2019-09-04 10:04:11.589024: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b300 next 103 of size 256
2019-09-04 10:04:11.589036: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b400 next 104 of size 256
2019-09-04 10:04:11.589050: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b500 next 105 of size 256
2019-09-04 10:04:11.589061: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b600 next 106 of size 256
2019-09-04 10:04:11.589075: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245203b700 next 107 of size 8388608
2019-09-04 10:04:11.589089: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245283b700 next 108 of size 8192
2019-09-04 10:04:11.589102: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245283d700 next 109 of size 8192
2019-09-04 10:04:11.589116: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f245283f700 next 110 of size 8192
2019-09-04 10:04:11.589126: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2452841700 next 111 of size 8192
2019-09-04 10:04:11.589138: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2452843700 next 112 of size 256
2019-09-04 10:04:11.589152: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2452843800 next 113 of size 8192
2019-09-04 10:04:11.589165: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2452845800 next 114 of size 8192
2019-09-04 10:04:11.589179: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2452847800 next 115 of size 4194304
2019-09-04 10:04:11.589192: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2452c47800 next 116 of size 4194304
2019-09-04 10:04:11.589206: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2453047800 next 117 of size 4194304
2019-09-04 10:04:11.589219: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2453447800 next 118 of size 4194304
2019-09-04 10:04:11.589234: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2453847800 next 18446744073709551615 of size 8095744
2019-09-04 10:04:11.589248: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 134217728
2019-09-04 10:04:11.589262: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2464000000 next 53 of size 67108864
2019-09-04 10:04:11.589277: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2468000000 next 18446744073709551615 of size 67108864
2019-09-04 10:04:11.589290: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 67108864
2019-09-04 10:04:11.589304: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2478000000 next 40 of size 16777216
2019-09-04 10:04:11.589315: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2479000000 next 43 of size 8388608
2019-09-04 10:04:11.589329: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2479800000 next 44 of size 8388608
2019-09-04 10:04:11.589343: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a000000 next 45 of size 8192
2019-09-04 10:04:11.589354: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a002000 next 46 of size 256
2019-09-04 10:04:11.589368: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a002100 next 47 of size 4194304
2019-09-04 10:04:11.589381: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a402100 next 48 of size 256
2019-09-04 10:04:11.589392: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a402200 next 49 of size 256
2019-09-04 10:04:11.589405: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a402300 next 50 of size 65536
2019-09-04 10:04:11.589420: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a412300 next 51 of size 196608
2019-09-04 10:04:11.589434: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a442300 next 54 of size 8192
2019-09-04 10:04:11.589448: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247a444300 next 55 of size 8388608
2019-09-04 10:04:11.589461: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247ac44300 next 58 of size 65536
2019-09-04 10:04:11.589477: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247ac54300 next 59 of size 4096
2019-09-04 10:04:11.589490: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247ac55300 next 60 of size 65536
2019-09-04 10:04:11.589502: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247ac65300 next 61 of size 65536
2019-09-04 10:04:11.589516: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f247ac75300 next 18446744073709551615 of size 20491520
2019-09-04 10:04:11.589530: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 33554432
2019-09-04 10:04:11.589544: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24be000000 next 32 of size 8388608
2019-09-04 10:04:11.589558: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24be800000 next 35 of size 8388608
2019-09-04 10:04:11.589570: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24bf000000 next 18446744073709551615 of size 16777216
2019-09-04 10:04:11.589583: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 16777216
2019-09-04 10:04:11.589594: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da000000 next 24 of size 4194304
2019-09-04 10:04:11.589608: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da400000 next 26 of size 256
2019-09-04 10:04:11.589623: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da400100 next 27 of size 12288
2019-09-04 10:04:11.589636: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da403100 next 28 of size 8192
2019-09-04 10:04:11.589650: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da405100 next 29 of size 4194304
2019-09-04 10:04:11.589663: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da805100 next 30 of size 2048
2019-09-04 10:04:11.589676: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da805900 next 33 of size 16896
2019-09-04 10:04:11.589690: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da809b00 next 34 of size 8192
2019-09-04 10:04:11.589704: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da80bb00 next 36 of size 2048
2019-09-04 10:04:11.589718: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da80c300 next 37 of size 81920
2019-09-04 10:04:11.589731: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da820300 next 38 of size 256
2019-09-04 10:04:11.589745: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da820400 next 41 of size 4096
2019-09-04 10:04:11.589755: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da821400 next 42 of size 32768
2019-09-04 10:04:11.589767: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f24da829400 next 18446744073709551615 of size 8219648
2019-09-04 10:04:11.589780: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1048576
2019-09-04 10:04:11.589795: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400000 next 1 of size 1280
2019-09-04 10:04:11.589809: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400500 next 2 of size 256
2019-09-04 10:04:11.589822: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400600 next 3 of size 256
2019-09-04 10:04:11.589835: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400700 next 4 of size 256
2019-09-04 10:04:11.589848: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400800 next 5 of size 256
2019-09-04 10:04:11.589862: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400900 next 6 of size 256
2019-09-04 10:04:11.589875: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400a00 next 7 of size 256
2019-09-04 10:04:11.589888: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400b00 next 8 of size 512
2019-09-04 10:04:11.589901: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400d00 next 10 of size 256
2019-09-04 10:04:11.589913: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400e00 next 11 of size 256
2019-09-04 10:04:11.589923: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549400f00 next 12 of size 8192
2019-09-04 10:04:11.589937: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549402f00 next 16 of size 204800
2019-09-04 10:04:11.589951: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549434f00 next 17 of size 256
2019-09-04 10:04:11.589965: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549435000 next 18 of size 512
2019-09-04 10:04:11.589988: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549435200 next 19 of size 8192
2019-09-04 10:04:11.589999: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549437200 next 20 of size 8192
2019-09-04 10:04:11.590011: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549439200 next 21 of size 256
2019-09-04 10:04:11.590022: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549439300 next 22 of size 256
2019-09-04 10:04:11.590033: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549439400 next 25 of size 512
2019-09-04 10:04:11.590044: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2549439600 next 18446744073709551615 of size 813568
2019-09-04 10:04:11.590059: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 4194304
2019-09-04 10:04:11.590074: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f254b200000 next 18446744073709551615 of size 4194304
2019-09-04 10:04:11.590087: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 4194304
2019-09-04 10:04:11.590100: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f254b600000 next 18446744073709551615 of size 4194304
2019-09-04 10:04:11.590108: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 8388608
2019-09-04 10:04:11.590119: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2553000000 next 15 of size 4194304
2019-09-04 10:04:11.590130: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2553400000 next 18446744073709551615 of size 4194304
2019-09-04 10:04:11.590144: I tensorflow/core/common_runtime/bfc_allocator.cc:809] Summary of in-use Chunks by size:
2019-09-04 10:04:11.590170: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 199 Chunks of size 256 totalling 49.8KiB
2019-09-04 10:04:11.590185: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 7 Chunks of size 512 totalling 3.5KiB
2019-09-04 10:04:11.590199: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1024 totalling 1.0KiB
2019-09-04 10:04:11.590212: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280 totalling 1.2KiB
2019-09-04 10:04:11.590226: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 23 Chunks of size 2048 totalling 46.0KiB
2019-09-04 10:04:11.590235: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 2 Chunks of size 4096 totalling 8.0KiB
2019-09-04 10:04:11.590249: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 4608 totalling 4.5KiB
2019-09-04 10:04:11.590262: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 18 Chunks of size 8192 totalling 144.0KiB
2019-09-04 10:04:11.590279: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 12288 totalling 12.0KiB
2019-09-04 10:04:11.590295: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 16896 totalling 16.5KiB
2019-09-04 10:04:11.590310: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 32768 totalling 32.0KiB
2019-09-04 10:04:11.590325: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 65536 totalling 256.0KiB
2019-09-04 10:04:11.590338: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 81920 totalling 80.0KiB
2019-09-04 10:04:11.590349: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 196608 totalling 192.0KiB
2019-09-04 10:04:11.590360: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 204800 totalling 800.0KiB
2019-09-04 10:04:11.590375: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 813568 totalling 794.5KiB
2019-09-04 10:04:11.590390: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 32 Chunks of size 1048576 totalling 32.00MiB
2019-09-04 10:04:11.590406: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 32 Chunks of size 2097152 totalling 64.00MiB
2019-09-04 10:04:11.590420: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 2 Chunks of size 2338816 totalling 4.46MiB
2019-09-04 10:04:11.590434: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 16 Chunks of size 4194304 totalling 64.00MiB
2019-09-04 10:04:11.590448: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 8095744 totalling 7.72MiB
2019-09-04 10:04:11.590459: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 8219648 totalling 7.84MiB
2019-09-04 10:04:11.590475: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 17 Chunks of size 8388608 totalling 136.00MiB
2019-09-04 10:04:11.590491: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 16777216 totalling 48.00MiB
2019-09-04 10:04:11.590506: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 20491520 totalling 19.54MiB
2019-09-04 10:04:11.590520: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 67108864 totalling 256.00MiB
2019-09-04 10:04:11.590534: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 207515648 totalling 197.90MiB
2019-09-04 10:04:11.590549: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 5262336000 totalling 4.90GiB
2019-09-04 10:04:11.590565: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 5.72GiB
2019-09-04 10:04:11.590588: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 10979036160 memory_limit_: 10979036365 available bytes: 205 curr_region_allocation_bytes_: 8589934592
2019-09-04 10:04:11.590614: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
Limit: 10979036365
InUse: 6142981376
MaxInUse: 7216919808
NumAllocs: 5943748
MaxAllocSize: 5262336000
2019-09-04 10:04:11.590690: W tensorflow/core/common_runtime/bfc_allocator.cc:319] ************************************************__________________________________________**_*******
2019-09-04 10:04:11.590739: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at conv_ops.cc:486 : Resource exhausted: OOM when allocating tensor with shape[1142,256,50,46] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-09-04 10:04:11.628917: W tensorflow/core/common_runtime/bfc_allocator.cc:237] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.08GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-09-04 10:04:11.655198: W tensorflow/core/common_runtime/bfc_allocator.cc:237] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.30GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
Traceback (most recent call last):
File "main.py", line 531, in <module>
TErecognizer(ip.MAX_LEN, ip.x_tr, ip.x_te, ip.y_tr, ip.y_te, ip.n_tag, ip.tag2idx, ip.testing_value).recognizerModel(False)
File "/home/alamtl/Share/temptagger/model.py", line 184, in recognizerModel
history = model.fit( [self.x_tr], self.y_tr, validation_split=0.1, batch_size=self.BATCH_SIZE, epochs=self.n_epochs, verbose=1, callbacks=[es, mc, self.sc])
File "/home/alamtl/.local/miniconda3/envs/base_alamtl/lib/python3.7/site-packages/keras/engine/training.py", line 1039, in fit
validation_steps=validation_steps)
File "/home/alamtl/.local/miniconda3/envs/base_alamtl/lib/python3.7/site-packages/keras/engine/training_arrays.py", line 217, in fit_loop
callbacks.on_epoch_end(epoch, epoch_logs)
File "/home/alamtl/.local/miniconda3/envs/base_alamtl/lib/python3.7/site-packages/keras/callbacks.py", line 79, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "/home/alamtl/.local/miniconda3/envs/base_alamtl/lib/python3.7/site-packages/seqeval/callbacks.py", line 80, in on_epoch_end
self.on_epoch_end_fit(epoch, logs)
File "/home/alamtl/.local/miniconda3/envs/base_alamtl/lib/python3.7/site-packages/seqeval/callbacks.py", line 87, in on_epoch_end_fit
y_true, y_pred = self.predict(X, y)
File "/home/alamtl/.local/miniconda3/envs/base_alamtl/lib/python3.7/site-packages/seqeval/callbacks.py", line 49, in predict
y_pred = self.model.predict_on_batch(X)
File "/home/alamtl/.local/miniconda3/envs/base_alamtl/lib/python3.7/site-packages/keras/engine/training.py", line 1274, in predict_on_batch
outputs = self.predict_function(ins)
File "/home/alamtl/.local/miniconda3/envs/base_alamtl/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__
return self._call(inputs)
File "/home/alamtl/.local/miniconda3/envs/base_alamtl/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call
fetched = self._callable_fn(*array_vals)
File "/home/alamtl/.local/miniconda3/envs/base_alamtl/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1458, in __call__
run_metadata_ptr)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[1142,1024,50,44] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node lambda_1/module_apply_default/bilm/CNN/Conv2D_6}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[crf_1/while_1/Identity/_898]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[1142,1024,50,44] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node lambda_1/module_apply_default/bilm/CNN/Conv2D_6}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
0 successful operations.
0 derived errors ignored.
`