dmlc/mshadow

Use GPU To Random a tensor when shape is odd error

Closed this issue · 9 comments

reyoung commented

Error Code Shows like that

InitTensorEngine(0);
TensorContainer<gpu,1> tmp;
tmp.Resize(Shape1(1));  // 1 Or any odd value
Random<mshadow::gpu> rnd(0);
rnd.SampleGaussian(tmp, 0, 0.1);
ShutdownTensorEngine();

The error code is CURAND_STATUS_LENGTH_NOT_MULTIPLE. The error line maybe is here.

reyoung commented

Is this a bug or feature? I can write a patch if you want.

antinucleon commented

The param for SampleGaussain and SampleUniform is Tensor<xpu,dim>.
You are using TensorContainer, so you should use uniform or gaussian expression like rnd.gaussian(tmp, mu, sigma)

tqchen commented

This seems to be a BUG, TensorContainer should work here. I guess there are some restrictions in CURAND that we overlooked...

reyoung commented

I made a pull request to fix it.

tqchen commented

Thanks. @reyoung!! I make some further changes based on your patch . Can you check if it works correctly now?
I would like to know the reason why this happens, my guess is that when ever we put odd size into curand's gaussian random number generator, it will hits that error(due to fact that gaussian rnd must be generated in pairs), it would be great if we could test that.

reyoung commented

It is not correctly now. The error shows 'Error:an illegal memory access was encountered'. It seems the buffer_ is already freed.

reyoung commented

The code that will make this error occurred shows as follow.

    InitTensorEngine(0);
    TensorContainer<gpu,1> tmp;
    tmp.Resize(Shape1(3));  // 1 Or any odd value
    Random<mshadow::gpu> rnd(0);
    rnd.SampleGaussian(tmp, 0, 0.1);
    TensorContainer<cpu,1> b;
    b.Resize(tmp.shape);
    Copy(b,tmp);
    cout<<b[0]<<endl;
    ShutdownTensorEngine();
tqchen commented

Thanks.. seems I made a wrong fix, I pushed another patch to the code.

tqchen commented

This issue is fixed by newest patch.