beta2 value reasoning
s1ddok opened this issue · 4 comments
Hey!
Could you explain the reasoning behind beta2 being equal to 0.999 instead of 0.9 (like in SPADE)? Didn't find any mention of this neither in the paper nor in code
Hi,
During the ablations, we observed that this optimizer setting yields the best final results with the N+1 loss of our segmentation-based discriminator. Otherwise, I don't think there was any other reasoning ;)
Vadim
thanks for the answer! is there any intuition you can share?
Generally, when I tune GAN hyperparameters, I mostly just look at the learning rate, the Adam betas, and the hyperparameters that are specific to the GAN architecture at hand. For OASIS, we looked at several combinations of learning rates and betas.
I just glanced at the notes in our lab journal, and it seems like for some learning rate, the choice of beta2 had an influence on the FID learning curve in the early stages of training, for Cityscapes. We picked 0.999 at some early point in the experimentation, but looking at later experiments it is clear that it did not make a discernable difference. Even if the FID curves look different in the early training stages, they all converge to the same final values eventually.
However, if you are applying OASIS to a completely different dataset, it might still be worth testing out different beta values.
thank you!