Hi, jianshu, thanks for your excellent codes, yet when re-producing your experiments, I find that the newest version of libgpuarray is not compatible with theano 0.10.0, so could you tell me which libgpuarray version do you use in your experment? thanks a lot!

Question

Hi, jianshu, thanks for your excellent codes, yet when re-producing your experiments, I find that the newest version of libgpuarray is not compatible with theano 0.10.0, so could you tell me which libgpuarray version do you use in your experment? thanks a lot!

Closed this issue 7 years ago · 8 comments

Answer 1 · 2018-03-15T01:46:33.000Z

Dear, the theano version is '0.10.0beta1.dev' and the libgpuarray version is '0.6.9'. If you still find the libgpuarray is not compatible with theano 0.10.0, you can also try theano 0.9.0 but the cudnn need to be v-5.1. Thank you for your interest in WAP ! xysszjs@mail.ustc.edu.cn From: Wenji Wang Date: 2018-03-15 07:38 To: JianshuZhang/WAP CC: Subscribed Subject: [JianshuZhang/WAP] Hi, jianshu, thanks for your excellent codes, yet when re-producing your experiments, I find that the newest version of libgpuarray is not compatible with theano 0.10.0, so could you tell me which libgpuarray version do you use in your experment? thanks a lot! (#1) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 2 · 2018-03-15T11:00:19.000Z

Thanks for prompt reply, I tried to run on ubuntu16.04 with theano 0.10.0beta1 libgpuarray 0.6.9, yet it fails. I try to install theano 0.10.0dev, but the version seems gone, maybe I shall try theano 0.9.0?

Answer 3 · 2018-03-15T11:15:05.000Z

Yes, you can try theano 0.9.0. You can install theano 0.9.0 by using conda, I believe the libgpuarray can also be installed if you install theano by using conda. xysszjs@mail.ustc.edu.cn From: Wenji Wang Date: 2018-03-15 19:00 To: JianshuZhang/WAP CC: Jianshu Zhang; Comment Subject: Re: [JianshuZhang/WAP] Hi, jianshu, thanks for your excellent codes, yet when re-producing your experiments, I find that the newest version of libgpuarray is not compatible with theano 0.10.0, so could you tell me which libgpuarray version do you use in your experment? thanks a lot! (#1) Thanks for prompt reply, I tried to run on ubuntu16.04 with theano 0.10.0beta1 libgpuarray 0.6.9, yet it fails. I try to install theano 0.10.0dev, but the version seems gone, maybe I shall try theano 0.9.0? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 4 · 2018-03-15T13:01:36.000Z

Fine, I manage to compile the code with theano 0.9.0 and pygpu 0.6.9 cudnn7.0. But it runs into pygpu.gpuarray.GpuArrayException: b'cuMemAlloc: CUDA_ERROR_OUT_OF_MEMORY: out of memory your batch_size is 16, however, the error remains even I change the batch_size to 2 on 12GB Tesla K80 GPU, that makes me confused

Answer 5 · 2018-03-15T13:31:12.000Z

I believe the batch_size is 8 not 16 ? I once updated the open source code. Besides the batch_size, the batch_Imagesize and maxImagesize also affect GPU memory use. You can reduce the batch_Imagesize from 500000 to 400000.

Answer 6 · 2018-03-15T13:33:57.000Z

And this is the error screenshot, as we can see, it starts with several iterations yet crashes because of out_of_memory error, and as the training process goes the memory of GPU seems to be increasingly ocuppied :(

Answer 7 · 2018-03-15T13:39:47.000Z

'gpuarray.preallocate=0.95' in THEANO_FLAGS is important, make sure you didn't remove it

Answer 8 · 2018-03-15T13:55:03.000Z

Ok, after resetting the batchsize and maxImagesize, the GPU occupation seems to be steady, thanks !