problems when finetuning
Ten000hours opened this issue · 0 comments
hi, when i run the code
GLOG_logtostderr=1 ../../build/tools/finetune_net.bin pascal_finetune_solver.prototxt ~/rcnn/data/caffe_nets/ilsvrc_2012_train_iter_310k 2>&1 | tee log.txt
the caffe net initailizition seems stopped suddenly,the log.txt shows like this
I0406 16:51:33.727171 26542 finetune_net.cpp:25] Starting Optimization I0406 16:51:33.727485 26542 solver.cpp:41] Creating training net. I0406 16:51:33.728703 26542 net.cpp:75] Creating Layer data I0406 16:51:33.728741 26542 net.cpp:111] data -> data I0406 16:51:33.728757 26542 net.cpp:111] data -> label I0406 16:51:33.728803 26542 window_data_layer.cpp:279] Window data layer: foreground (object) overlap threshold: 0.5 background (non-object) overlap threshold: 0.5 foreground sampling fraction: 0.25 I0406 16:51:33.737037 26542 window_data_layer.cpp:345] num: 0 /home/xuxiangrui/Pascal_voc/celeba/VOCdevkit/VOC2007/JPEGImages/000002.jpg 3 594 423 windows to process: 2762 I0406 16:51:33.914831 26542 window_data_layer.cpp:345] num: 100 /home/xuxiangrui/Pascal_voc/celeba/VOCdevkit/VOC2007/JPEGImages/000189.jpg 3 400 496 windows to process: 2304 I0406 16:51:34.066776 26542 window_data_layer.cpp:345] num: 200 /home/xuxiangrui/Pascal_voc/celeba/VOCdevkit/VOC2007/JPEGImages/000393.jpg 3 640 408 windows to process: 2301 I0406 16:51:34.247323 26542 window_data_layer.cpp:345] num: 300 /home/xuxiangrui/Pascal_voc/celeba/VOCdevkit/VOC2007/JPEGImages/000595.jpg 3 360 640 windows to process: 1185 I0406 16:51:34.380002 26542 window_data_layer.cpp:345] num: 400 /home/xuxiangrui/Pascal_voc/celeba/VOCdevkit/VOC2007/JPEGImages/000808.jpg 3 337 500 windows to process: 2281 I0406 16:51:34.591991 26542 window_data_layer.cpp:354] Number of images: 500 I0406 16:51:34.592015 26542 window_data_layer.cpp:358] class 0 has 1261791 samples I0406 16:51:34.592025 26542 window_data_layer.cpp:362] Amount of context padding: 16 I0406 16:51:34.592027 26542 window_data_layer.cpp:365] Crop mode: warp I0406 16:51:34.592031 26542 window_data_layer.cpp:376] output data size: 64,3,227,227 I0406 16:51:34.592034 26542 window_data_layer.cpp:388] Loading mean file from../../data/ilsvrc12/imagenet_mean.binaryproto I0406 16:51:36.851869 26542 net.cpp:126] Top shape: 64 3 227 227 (9893568) I0406 16:51:36.851935 26542 net.cpp:126] Top shape: 64 1 1 1 (64) I0406 16:51:36.851958 26542 net.cpp:157] data does not need backward computation. I0406 16:51:36.851992 26542 net.cpp:75] Creating Layer conv1 I0406 16:51:36.852007 26542 net.cpp:85] conv1 <- data I0406 16:51:36.852041 26542 net.cpp:111] conv1 -> conv1 I0406 16:51:36.866598 26542 net.cpp:126] Top shape: 64 96 55 55 (18585600) I0406 16:51:36.866847 26542 net.cpp:152] conv1 needs backward computation. I0406 16:51:36.867000 26542 net.cpp:75] Creating Layer relu1 I0406 16:51:36.867081 26542 net.cpp:85] relu1 <- conv1 I0406 16:51:36.867144 26542 net.cpp:99] relu1 -> conv1 (in-place) I0406 16:51:36.867208 26542 net.cpp:126] Top shape: 64 96 55 55 (18585600) I0406 16:51:36.867270 26542 net.cpp:152] relu1 needs backward computation. I0406 16:51:36.867333 26542 net.cpp:75] Creating Layer pool1 I0406 16:51:36.867391 26542 net.cpp:85] pool1 <- conv1 I0406 16:51:36.867465 26542 net.cpp:111] pool1 -> pool1 I0406 16:51:36.867533 26542 net.cpp:126] Top shape: 64 96 27 27 (4478976) I0406 16:51:36.867555 26542 net.cpp:152] pool1 needs backward computation. I0406 16:51:36.867578 26542 net.cpp:75] Creating Layer norm1 I0406 16:51:36.867590 26542 net.cpp:85] norm1 <- pool1 I0406 16:51:36.867606 26542 net.cpp:111] norm1 -> norm1 I0406 16:51:36.867746 26542 net.cpp:126] Top shape: 64 96 27 27 (4478976) I0406 16:51:36.867774 26542 net.cpp:152] norm1 needs backward computation. I0406 16:51:36.867817 26542 net.cpp:75] Creating Layer conv2 I0406 16:51:36.867832 26542 net.cpp:85] conv2 <- norm1 I0406 16:51:36.867864 26542 net.cpp:111] conv2 -> conv2 I0406 16:51:36.885260 26542 net.cpp:126] Top shape: 64 256 27 27 (11943936) I0406 16:51:36.885288 26542 net.cpp:152] conv2 needs backward computation. I0406 16:51:36.885303 26542 net.cpp:75] Creating Layer relu2 I0406 16:51:36.885310 26542 net.cpp:85] relu2 <- conv2 I0406 16:51:36.885320 26542 net.cpp:99] relu2 -> conv2 (in-place) I0406 16:51:36.885331 26542 net.cpp:126] Top shape: 64 256 27 27 (11943936) I0406 16:51:36.885340 26542 net.cpp:152] relu2 needs backward computation. I0406 16:51:36.885351 26542 net.cpp:75] Creating Layer pool2 I0406 16:51:36.885360 26542 net.cpp:85] pool2 <- conv2 I0406 16:51:36.885370 26542 net.cpp:111] pool2 -> pool2 I0406 16:51:36.885381 26542 net.cpp:126] Top shape: 64 256 13 13 (2768896) I0406 16:51:36.885390 26542 net.cpp:152] pool2 needs backward computation. I0406 16:51:36.885404 26542 net.cpp:75] Creating Layer norm2 I0406 16:51:36.885413 26542 net.cpp:85] norm2 <- pool2 I0406 16:51:36.885422 26542 net.cpp:111] norm2 -> norm2 I0406 16:51:36.885435 26542 net.cpp:126] Top shape: 64 256 13 13 (2768896) I0406 16:51:36.885448 26542 net.cpp:152] norm2 needs backward computation. I0406 16:51:36.885460 26542 net.cpp:75] Creating Layer conv3 I0406 16:51:36.885469 26542 net.cpp:85] conv3 <- norm2 I0406 16:51:36.885479 26542 net.cpp:111] conv3 -> conv3 I0406 16:51:36.906862 26542 net.cpp:126] Top shape: 64 384 13 13 (4153344) I0406 16:51:36.906883 26542 net.cpp:152] conv3 needs backward computation. I0406 16:51:36.906889 26542 net.cpp:75] Creating Layer relu3 I0406 16:51:36.906893 26542 net.cpp:85] relu3 <- conv3 I0406 16:51:36.906898 26542 net.cpp:99] relu3 -> conv3 (in-place) I0406 16:51:36.906903 26542 net.cpp:126] Top shape: 64 384 13 13 (4153344) I0406 16:51:36.906905 26542 net.cpp:152] relu3 needs backward computation. I0406 16:51:36.906910 26542 net.cpp:75] Creating Layer conv4 I0406 16:51:36.906913 26542 net.cpp:85] conv4 <- conv3 I0406 16:51:36.906916 26542 net.cpp:111] conv4 -> conv4 I0406 16:51:36.917457 26542 net.cpp:126] Top shape: 64 384 13 13 (4153344) I0406 16:51:36.917470 26542 net.cpp:152] conv4 needs backward computation. I0406 16:51:36.917475 26542 net.cpp:75] Creating Layer relu4 I0406 16:51:36.917479 26542 net.cpp:85] relu4 <- conv4 I0406 16:51:36.917482 26542 net.cpp:99] relu4 -> conv4 (in-place) I0406 16:51:36.917486 26542 net.cpp:126] Top shape: 64 384 13 13 (4153344) I0406 16:51:36.917490 26542 net.cpp:152] relu4 needs backward computation. I0406 16:51:36.917495 26542 net.cpp:75] Creating Layer conv5 I0406 16:51:36.917497 26542 net.cpp:85] conv5 <- conv4 I0406 16:51:36.917503 26542 net.cpp:111] conv5 -> conv5 I0406 16:51:36.927706 26542 net.cpp:126] Top shape: 64 256 13 13 (2768896) I0406 16:51:36.927819 26542 net.cpp:152] conv5 needs backward computation. I0406 16:51:36.927845 26542 net.cpp:75] Creating Layer relu5 I0406 16:51:36.927855 26542 net.cpp:85] relu5 <- conv5 I0406 16:51:36.927867 26542 net.cpp:99] relu5 -> conv5 (in-place) I0406 16:51:36.927880 26542 net.cpp:126] Top shape: 64 256 13 13 (2768896) I0406 16:51:36.927886 26542 net.cpp:152] relu5 needs backward computation. I0406 16:51:36.927898 26542 net.cpp:75] Creating Layer pool5 I0406 16:51:36.927903 26542 net.cpp:85] pool5 <- conv5 I0406 16:51:36.927911 26542 net.cpp:111] pool5 -> pool5 I0406 16:51:36.927924 26542 net.cpp:126] Top shape: 64 256 6 6 (589824) I0406 16:51:36.927928 26542 net.cpp:152] pool5 needs backward computation. I0406 16:51:36.927949 26542 net.cpp:75] Creating Layer fc6 I0406 16:51:36.927958 26542 net.cpp:85] fc6 <- pool5 I0406 16:51:36.927968 26542 net.cpp:111] fc6 -> fc6
Can anyone help me with this problem?