Adjust the value of sigma
Closed this issue · 3 comments
Hi,
As mentioned in my another issue, after scale the pixel intensity from [0,255] to [0, 1], I think there is a need to adjust the sigma value here.
I tried this one a small subset of data (less than 1%), with sigma set to be 0.7 to 2, the network won't converge, even after 1000+ epochs; after I divided the sigma by 16 (around the square root of 255), it converged within 10 epochs.
I am not totally sure., as 0.7/16 ≈ 0.04 the pixel variance will be much too small and the generator distribution collapses to a Dirac distribution around the mean, i.e. we assume that there is no noise in the data.
It would makes sense for it to converge under this value, but can you verify that not only reconstruction, but also samples look good under this training regime?
Good point, I did not try sampling, but based on the high KL value (around 200), it won't be good. I'm not sure if that will converge too with more epoches with this parameter.
I have added a pretrained model and updated the example notebooks. For this I used about 10% of the dataset and trained for 200 epochs with the 2.0 to 0.7 sigma scheme over 800.000 minibatch iterations. This seems to work well - therefore I believe this scheme is correct, however, the number of iterations to go to the value of 0.7 depends on the size of the dataset.