Sampler gives NaN values
jacobbieker opened this issue · 4 comments
Describe the bug
The Sampler gives NaN values when being used in the generator.
To Reproduce
Steps to reproduce the behavior:
The unit tests have it happen in test_generator, and test_sampler
Expected behavior
No NaN values should exist in the output
Additional context
From unit tests, each of the layers in the Sampler don't return NaNs, so not sure why it does so. It seems to happen usually after g3
or ConvGRU3
layers.
I don't understand why the authors transform the radar echo dBZ to rain rate using Z-R relationship? Why don't they directly use radar echo reflectivity?
Thanks for your implementation of dgmr! It's organization is clear,and easy to read.
When I run the test_model.py,I also found the nan values in sampler's output. I check the ConvGRU.py, found that the bug may be aroused by spectral_norm
function.
According to PyTorch spectral_norm doc , I use torch.nn.utils.parametrizations.spectral_norm()
instead, also spectral_norm
in common.py, and it works fine now. I Think you should have a try.
Though I don't know why, hopefully it helps. Thanks.
Oh, thanks for figuring that out! I'll update it to use that now then too
It worked! Thanks so much