BatchSize

[nitesh1813]

Hello,
Increasing the batch size seems to increase the time taken for processing the batch almost by a factor of batchsize.
The only thing which I feel which influences it is the n_grids parameter.
I calculated the memory used by the output octree at each step and I found that it should take around 1gb all combined for the forward pass per input. So considering backward and network parameters, the whole thing should not take more than 2.5gb. But changing n_grids it to -1 causes it to go out of memory.
Can you please help me out?

Thanks.

[griegler]

The convolution is implemented via a matrix multiplication, which means that the data in the octree is first transformed into a matrix. The parameter n_grids influences the number of shallow octrees that are transformed simultaneously. The matrix for the transformation needs extra memory and for large octree-grids you might run out of memory. Hence, there is a trade-off between speed and memory consumption.

[nitesh1813]

Ah. I was just curious because the same model with a tensorflow implementation was able to fit the uncompressed array in memory, So I felt like maybe I was doing something wrong.

[griegler]

Yes, with cudnn you do not have to explicitly build the matrix.

[nitesh1813]

Thanks a lot!