How to solve the problem of Loss NaN

[lumosity4tpj]

I wanted to implement sparse VD in convolution, but loss was NaN. Would anyone like to provide the code in the case of convolution? I am not very familiar with theano in the original code of the paper, and I still have problems after rewriting.
This is my code:

[senya-ashukha]

Hey,

Thank you for checking out at the assignment code!

1. lrt_std should not depend on the bias
2. make sure you are using adam or something adaptive

You can also look at tf implementation here https://github.com/google-research/google-research/blob/master/state_of_sparsity/layers/variational_dropout/nn.py#L347.

[lumosity4tpj]

Thanks a lot!!! Because of the problem, I tried a lot. Removing bias seems to work.