openclimatefix/skillful_nowcasting

Sampler gives NaN values

jacobbieker opened this issue · 4 comments

Describe the bug
The Sampler gives NaN values when being used in the generator.

To Reproduce
Steps to reproduce the behavior:
The unit tests have it happen in test_generator, and test_sampler

Expected behavior
No NaN values should exist in the output

Additional context
From unit tests, each of the layers in the Sampler don't return NaNs, so not sure why it does so. It seems to happen usually after g3 or ConvGRU3 layers.

sjjlr commented

I don't understand why the authors transform the radar echo dBZ to rain rate using Z-R relationship? Why don't they directly use radar echo reflectivity?

Thanks for your implementation of dgmr! It's organization is clear,and easy to read.

When I run the test_model.py,I also found the nan values in sampler's output. I check the ConvGRU.py, found that the bug may be aroused by spectral_norm function.

According to PyTorch spectral_norm doc , I use torch.nn.utils.parametrizations.spectral_norm() instead, also spectral_norm in common.py, and it works fine now. I Think you should have a try.

Though I don't know why, hopefully it helps. Thanks.

Oh, thanks for figuring that out! I'll update it to use that now then too

It worked! Thanks so much