LTH14/mage

slightly different performance result of un-conditional generation on ImageNet

Closed this issue · 2 comments

ChangyaoTian commented

Hi, my reproduction result (using the very eval code and ckpt in this repo) of un-conditional image generation is slightly different from your official results. As for ViT-B, the FID and IS scores are 13.74 and 81.16 (1.76), respectively. Is such performance fluctuation normal and acceptable?

LTH14 commented

This should not be the case, as the random seed for the generation is fixed in the script. I found the IS is consistent with my result while the FID is not. Do you use prepare_imgnet_val.py provided in this repo to generate the imagenet validation set?

ChangyaoTian commented

Thanks for your advice! We later found that the bug is caused by the saving format of images, we saved the imagenet validation set as '.jpeg' format rather than '.png', which caused this difference.
The corrected result of our reproduction is FID 11.18 & IS 80.57 (1.22), nearly the same as your official results.