Unable to run the train_deep_logo_cnn file
falcon97 opened this issue · 1 comments
I ran the crop_and_aug and the gen_train_valid_test files successfully, following which I executed the train_deep_logo_cnn file. I get a OOM error. Here's the trace back:
C:\Anaconda2\envs\py35\python.exe C:/Users/user/Downloads/DeepLogo/train_deep_logo_cnn.py
Training set (50000, 32, 64, 3) (50000,)
Valid set (5000, 32, 64, 3) (5000,)
Test set (5000, 32, 64, 3) (5000,)
Training set (50000, 64, 32, 3) (50000, 27)
Valid set (5000, 64, 32, 3) (5000, 27)
Test set (5000, 64, 32, 3) (5000, 27)
2017-06-16 20:36:35.350598: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations.
2017-06-16 20:36:35.350862: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-16 20:36:35.351095: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-16 20:36:35.351327: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-16 20:36:35.351562: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-16 20:36:35.351810: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-16 20:36:35.352067: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-16 20:36:35.352329: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-06-16 20:36:36.141969: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:940] Found device 0 with properties:
name: GeForce GTX 965M
major: 5 minor: 2 memoryClockRate (GHz) 1.15
pciBusID 0000:01:00.0
Total memory: 4.00GiB
Free memory: 3.33GiB
2017-06-16 20:36:36.142257: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:961] DMA: 0
2017-06-16 20:36:36.142378: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0: Y
2017-06-16 20:36:36.143175: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 965M, pci bus id: 0000:01:00.0)
initialized
Minibatch loss at step 0: 66399.546875
Minibatch accuracy: 0.0%
2017-06-16 20:36:52.850640: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 117.19MiB. Current allocation summary follows.
2017-06-16 20:36:52.851405: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.852243: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.853164: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.854093: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.855036: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.855967: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (8192): Total Chunks: 1, Chunks in use: 0 11.8KiB allocated for chunks. 7.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.856908: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (16384): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.857840: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (32768): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.858762: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.859684: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (131072): Total Chunks: 1, Chunks in use: 0 216.0KiB allocated for chunks. 4B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.860662: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.861635: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (524288): Total Chunks: 1, Chunks in use: 0 800.0KiB allocated for chunks. 300.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.862653: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.869983: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.870734: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.871491: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.872356: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (16777216): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.873183: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (33554432): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.873948: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (67108864): Total Chunks: 1, Chunks in use: 0 64.00MiB allocated for chunks. 1.50MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.874735: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.875504: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-06-16 20:36:52.876278: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:660] Bin for 117.19MiB was 64.00MiB, Chunk State:
2017-06-16 20:36:52.878767: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:666] Size: 64.00MiB | Requested Size: 1.50MiB | in_use: 0, prev: Size: 512B | Requested Size: 512B | in_use: 1, next: Size: 8.0KiB | Requested Size: 8.0KiB | in_use: 1
2017-06-16 20:36:52.879971: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702500000 of size 1280
2017-06-16 20:36:52.880760: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702500500 of size 256
2017-06-16 20:36:52.881668: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702500600 of size 256
2017-06-16 20:36:52.882104: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702500700 of size 256
2017-06-16 20:36:52.882527: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702500800 of size 256
2017-06-16 20:36:52.883397: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702500900 of size 512
2017-06-16 20:36:52.884240: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702500B00 of size 8192
2017-06-16 20:36:52.884919: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702502B00 of size 256
2017-06-16 20:36:52.885345: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702502C00 of size 256
2017-06-16 20:36:52.885763: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702502D00 of size 256
2017-06-16 20:36:52.886198: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702502E00 of size 256
2017-06-16 20:36:52.886678: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702502F00 of size 14592
2017-06-16 20:36:52.887172: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702506800 of size 256
2017-06-16 20:36:52.887581: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702506900 of size 307200
2017-06-16 20:36:52.887834: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702551900 of size 256
2017-06-16 20:36:52.888084: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702551A00 of size 819200
2017-06-16 20:36:52.888343: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702619A00 of size 512
2017-06-16 20:36:52.888595: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000702619C00 of size 67108864
2017-06-16 20:36:52.888861: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706619C00 of size 8192
2017-06-16 20:36:52.889117: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070661BC00 of size 221184
2017-06-16 20:36:52.889377: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706651C00 of size 256
2017-06-16 20:36:52.889639: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706651D00 of size 256
2017-06-16 20:36:52.889901: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706651E00 of size 256
2017-06-16 20:36:52.890168: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706651F00 of size 256
2017-06-16 20:36:52.890444: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706652000 of size 256
2017-06-16 20:36:52.890716: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706652100 of size 256
2017-06-16 20:36:52.892403: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706652200 of size 256
2017-06-16 20:36:52.892829: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706652300 of size 256
2017-06-16 20:36:52.893166: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706652400 of size 256
2017-06-16 20:36:52.893383: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706652500 of size 256
2017-06-16 20:36:52.893601: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706652600 of size 256
2017-06-16 20:36:52.893823: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706655600 of size 256
2017-06-16 20:36:52.894073: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706655700 of size 307200
2017-06-16 20:36:52.894333: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007066A0700 of size 256
2017-06-16 20:36:52.894588: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000706768800 of size 512
2017-06-16 20:36:52.894847: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A768A00 of size 8192
2017-06-16 20:36:52.895110: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A7A0A00 of size 256
2017-06-16 20:36:52.895377: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A7A0B00 of size 256
2017-06-16 20:36:52.895637: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A7A0C00 of size 256
2017-06-16 20:36:52.895899: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A7A0D00 of size 14592
2017-06-16 20:36:52.896167: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A7A4600 of size 14592
2017-06-16 20:36:52.896436: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A7A7F00 of size 256
2017-06-16 20:36:52.896710: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A7A8000 of size 256
2017-06-16 20:36:52.896978: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A7A8100 of size 307200
2017-06-16 20:36:52.897252: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A7F3100 of size 307200
2017-06-16 20:36:52.897537: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A83E100 of size 256
2017-06-16 20:36:52.897822: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A83E200 of size 256
2017-06-16 20:36:52.898099: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A83E300 of size 819200
2017-06-16 20:36:52.898389: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A906300 of size 819200
2017-06-16 20:36:52.898674: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A9CE300 of size 512
2017-06-16 20:36:52.898954: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A9CE500 of size 512
2017-06-16 20:36:52.901430: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070A9CE700 of size 67108864
2017-06-16 20:36:52.901871: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000070E9CE700 of size 67108864
2017-06-16 20:36:52.902092: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007129CE700 of size 8192
2017-06-16 20:36:52.902307: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007129D0700 of size 8192
2017-06-16 20:36:52.902531: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007129D2700 of size 221184
2017-06-16 20:36:52.902764: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000712A08700 of size 221184
2017-06-16 20:36:52.903454: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000712A3E700 of size 256
2017-06-16 20:36:52.903781: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000712A3E800 of size 256
2017-06-16 20:36:52.903996: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000712A3E900 of size 14592
2017-06-16 20:36:52.904212: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000712A42200 of size 307200
2017-06-16 20:36:52.904669: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000712A8D200 of size 819200
2017-06-16 20:36:52.904891: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000712B55200 of size 67108864
2017-06-16 20:36:52.905113: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000716B55200 of size 122880000
2017-06-16 20:36:52.905345: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000071E085200 of size 2802052608
2017-06-16 20:36:52.905606: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000706652700 of size 12032
2017-06-16 20:36:52.905860: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007066A0800 of size 819200
2017-06-16 20:36:52.906121: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000706768A00 of size 67108864
2017-06-16 20:36:52.906388: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000070A76AA00 of size 221184
2017-06-16 20:36:52.907314: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:693] Summary of in-use Chunks by size:
2017-06-16 20:36:52.907740: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:696] 32 Chunks of size 256 totalling 8.0KiB
2017-06-16 20:36:52.908070: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:696] 5 Chunks of size 512 totalling 2.5KiB
2017-06-16 20:36:52.908289: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.3KiB
2017-06-16 20:36:52.908626: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:696] 5 Chunks of size 8192 totalling 40.0KiB
2017-06-16 20:36:52.908844: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:696] 4 Chunks of size 14592 totalling 57.0KiB
2017-06-16 20:36:52.909065: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:696] 3 Chunks of size 221184 totalling 648.0KiB
2017-06-16 20:36:52.909296: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:696] 5 Chunks of size 307200 totalling 1.46MiB
2017-06-16 20:36:52.909880: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:696] 4 Chunks of size 819200 totalling 3.13MiB
2017-06-16 20:36:52.910264: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:696] 4 Chunks of size 67108864 totalling 256.00MiB
2017-06-16 20:36:52.910488: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 122880000 totalling 117.19MiB
2017-06-16 20:36:52.910710: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 2802052608 totalling 2.61GiB
2017-06-16 20:36:52.910935: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:700] Sum Total of in-use chunks: 2.98GiB
2017-06-16 20:36:52.911180: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:702] Stats:
Limit: 3267117056
InUse: 3198955776
MaxInUse: 3198955776
NumAllocs: 267
MaxAllocSize: 2802052608
2017-06-16 20:36:52.911627: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\bfc_allocator.cc:277] ***_***********************************************************************xxxxxxxxxxxxxxxxxxxxxxxxx
2017-06-16 20:36:52.911992: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\framework\op_kernel.cc:1158] Resource exhausted: OOM when allocating tensor with shape[5000,3,64,32]
Traceback (most recent call last):
File "C:\Anaconda2\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 1139, in _do_call
return fn(*args)
File "C:\Anaconda2\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 1121, in _run_fn
status, run_metadata)
File "C:\Anaconda2\envs\py35\lib\contextlib.py", line 66, in exit
next(self.gen)
File "C:\Anaconda2\envs\py35\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[5000,3,64,32]
[[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](Const_5, Variable/read)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/user/Downloads/DeepLogo/train_deep_logo_cnn.py", line 256, in
main()
File "C:/Users/user/Downloads/DeepLogo/train_deep_logo_cnn.py", line 237, in main
accuracy(valid_prediction.eval(), valid_labels))
File "C:\Anaconda2\envs\py35\lib\site-packages\tensorflow\python\framework\ops.py", line 606, in eval
return _eval_using_default_session(self, feed_dict, self.graph, session)
File "C:\Anaconda2\envs\py35\lib\site-packages\tensorflow\python\framework\ops.py", line 3928, in _eval_using_default_session
return session.run(tensors, feed_dict)
File "C:\Anaconda2\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 789, in run
run_metadata_ptr)
File "C:\Anaconda2\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 997, in _run
feed_dict_string, options, run_metadata)
File "C:\Anaconda2\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 1132, in _do_run
target_list, options, run_metadata)
File "C:\Anaconda2\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[5000,3,64,32]
[[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](Const_5, Variable/read)]]
Caused by op 'Conv2D_3', defined at:
File "C:/Users/user/Downloads/DeepLogo/train_deep_logo_cnn.py", line 256, in
main()
File "C:/Users/user/Downloads/DeepLogo/train_deep_logo_cnn.py", line 197, in main
w_conv3, b_conv3, w_fc1, b_fc1, w_fc2, b_fc2))
File "C:/Users/user/Downloads/DeepLogo/train_deep_logo_cnn.py", line 69, in model
tf.nn.conv2d(data, w_conv1, [1, 1, 1, 1], padding='SAME') + b_conv1)
File "C:\Anaconda2\envs\py35\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 399, in conv2d
data_format=data_format, name=name)
File "C:\Anaconda2\envs\py35\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 767, in apply_op
op_def=op_def)
File "C:\Anaconda2\envs\py35\lib\site-packages\tensorflow\python\framework\ops.py", line 2506, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Anaconda2\envs\py35\lib\site-packages\tensorflow\python\framework\ops.py", line 1269, in init
self._traceback = _extract_stack()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[5000,3,64,32]
[[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](Const_5, Variable/read)]]
Process finished with exit code 1
I tried changing the number of channels in both the gen_train_valid_test, but I get the following error in the gen_train_valid_test file:
ValueError: could not broadcast input array from shape (185,32,64,3) into shape (185,32,64,5)
Where am I going wrong?
how did you use import common?
I tried to use this but it's not working. It says CLASS_NAME is not a valid
Please help
Thanks