ELIFE-ASU/INNLab

`kaiming_uniform_` will lead to `nan` for RealNVP

Closed this issue · 1 comments

Zhangyanbo commented

The output of RealNVP can be very high. This is because it have torch.exp is the forward process:

y = x * torch.exp(self.logs(x)) + self.t(s)

Is there any method that do all requirement below at the same time:

  1. avoid very large number
  2. avoid very small number
Zhangyanbo commented

This seems be a issue off RealNVP, instead of a code bug. To solve this problem, I've added a soft clip option for RealNVP. Any number x that |x|>clip will become clip by using a function like clip * tanh(x / clip). However, this will not perfectly solve the problem for very deep models.