nv-tlabs/GENIE

GENIE Training Diverged in LSUN Church

ZichenMiao opened this issue · 1 comments

ZichenMiao commented

Hi,

It's a great work and thank you for sharing the code!

Yet I met some problems in reproducing training GENIE on LSUN Church. I followed the instructions, used 8 x A5000, changed the training batch_size from 32 to 8, and lowered the learning rate from 1e-4 to 1e-5. And the model diverged. Below are samples from 1000, 2000, and 5000 iterations.

iter_1000
iter_2000
iter_5000

I tried both lr=1e-4 and 1e-5. The results are the same.

timudk commented

Hi Zichen,

Thank you for opening the issue. The latest commit should fix the issue. I tested GENIE training on CIFAR-10 and it works again as expected. Let me know if you have any other issues.