Error when running the Train_Caffenet.sh

Question

Error when running the Train_Caffenet.sh

Closed this issue 7 years ago · 6 comments

Hello,

I am using caffe and GPU Tesla K80 to train your model. This is what I get every time I run the train script. Any idea what could be the issue? Do I need more than one GPU to train?

I0529 11:25:57.548988 73168 blocking_queue.cpp:49] Waiting for data
I0529 11:28:21.023334 73174 data_layer.cpp:73] Restarting data prefetching from start.
I0529 11:28:23.880893 73168 solver.cpp:418] Test net output #0: accuracy = 0.000996492
I0529 11:28:23.880945 73168 solver.cpp:418] Test net output #1: accuracy_top5 = 0.00476323
I0529 11:28:23.880959 73168 solver.cpp:418] Test net output #2: loss = 6.93271 (* 1 = 6.93271 loss)
F0529 11:28:24.432736 73168 syncedmem.cpp:71] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x2aaaac012e6d (unknown)
@ 0x2aaaac014ced (unknown)
@ 0x2aaaac012a5c (unknown)
@ 0x2aaaac01563e (unknown)
@ 0x2aaaaaf41140 caffe::SyncedMemory::mutable_gpu_data()
@ 0x2aaaaade3382 caffe::Blob<>::mutable_gpu_data()
@ 0x2aaaaaf84ef0 caffe::ConvolutionLayer<>::Forward_gpu()
@ 0x2aaaaaf0dfac caffe::Net<>::ForwardFromTo()
@ 0x2aaaaaf0e387 caffe::Net<>::Forward()
@ 0x2aaaaaf2bc4f caffe::Solver<>::Step()
@ 0x2aaaaaf2c44f caffe::Solver<>::Solve()
@ 0x40a727 train()
@ 0x407ebc main
@ 0x2aaabcc76c05 __libc_start_main
@ 0x408703 (unknown)
./examples/imagenet/train_caffenet.sh: line 5: 73168 Aborted (core dumped) ./build/tools/caffe train --solver=/home-new/aup019/zynqnet/_TRAINED_MODEL/solver.prototxt $@

Answer 1 · 2018-05-30T06:00:05.000Z

„Out of memory“ usually means that your GPU does not have enough (free) memory, but the K80 should actually have more memory than the Titan X I used (24GB vs. 12GB). Are there other processes using the K80 at the same time (shared machine)? Can you run „nvidia-smi“ to check the current status of the GPU?

…

Am 29.05.2018 um 19:32 schrieb Sarah ***@***.***>: Hello, I am using caffe and GPU Tesla K80 to train your model. This is what I get every time I run the train script. Any idea what could be the issue? Do I need more than one GPU to train? I0529 11:25:57.548988 73168 blocking_queue.cpp:49] Waiting for data I0529 11:28:21.023334 73174 data_layer.cpp:73] Restarting data prefetching from start. I0529 11:28:23.880893 73168 solver.cpp:418] Test net output #0: accuracy = 0.000996492 I0529 11:28:23.880945 73168 solver.cpp:418] Test net output #1: accuracy_top5 = 0.00476323 I0529 11:28:23.880959 73168 solver.cpp:418] Test net output #2: loss = 6.93271 (* 1 = 6.93271 loss) F0529 11:28:24.432736 73168 syncedmem.cpp:71] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** @ 0x2aaaac012e6d (unknown) @ 0x2aaaac014ced (unknown) @ 0x2aaaac012a5c (unknown) @ 0x2aaaac01563e (unknown) @ 0x2aaaaaf41140 caffe::SyncedMemory::mutable_gpu_data() @ 0x2aaaaade3382 caffe::Blob<>::mutable_gpu_data() @ 0x2aaaaaf84ef0 caffe::ConvolutionLayer<>::Forward_gpu() @ 0x2aaaaaf0dfac caffe::Net<>::ForwardFromTo() @ 0x2aaaaaf0e387 caffe::Net<>::Forward() @ 0x2aaaaaf2bc4f caffe::Solver<>::Step() @ 0x2aaaaaf2c44f caffe::Solver<>::Solve() @ 0x40a727 train() @ 0x407ebc main @ 0x2aaabcc76c05 __libc_start_main @ 0x408703 (unknown) ./examples/imagenet/train_caffenet.sh: line 5: 73168 Aborted (core dumped) ./build/tools/caffe train --solver=/home-new/aup019/zynqnet/_TRAINED_MODEL/solver.prototxt $@ — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 2 · 2018-05-30T14:00:08.000Z

It is a shared machine but when I checked the status is says no running process found.
I reduced the batch size and could get it to run. But I'm guessing that would affect the accuracy?

Answer 3 · 2018-05-30T15:36:41.000Z

Weird... what’s the output from nvidia-smi? It looks like the K80 can be „split“ into 2 virtual GPUs with 12GB RAM each, maybe that‘s what’s happened. Regarding the accuracy: You’re right. How to balance a reduction in batch-size with the learning rate should be mentioned in the report.

Answer 4 · 2018-05-30T15:50:46.000Z

This is the output from nvidia-smi. I obtained this while training the model.

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 115307 C ./build/tools/caffe 8734MiB |
+-----------------------------------------------------------------------------+

Answer 5 · 2018-05-30T16:56:56.000Z

It seems your machine includes 4 Tesla K80 GPUs, each virtually split into 2 cards with approx. 11.2 GB of memory each. So the original training settings (which use almost 12.0GB) are slightly too big. With your smaller batch size you got it down to 8.7 GB. Now you could try multi-GPU training together with the other 7 virtual GPUs! :-)

…

Answer 6 · 2018-05-30T17:06:19.000Z

Right! Thank you for your help! :)