Fid Score does not match the paper
Closed this issue · 5 comments
Hey when I try to run the code the fid scores calculated does not match the fid score in the paper. I am getting :
| fid_w | fid_no_w | |
|---|---|---|
| zero | 26.518 | 25.57 |
| ring | 26.007 | 25.57 |
| rand | 25.676 | 25.57 |
| ImageNet ring | 66.342 | Cell |
while the paper states this:
Hi, thanks for letting me know! I think one issue I found is that I copied the wrong config for imagenet, and it should use --w_radius 3 --w_channel 1.
Can you try it again with: python run_tree_ring_watermark_imagenet_fid.py --run_name imgnet_fid_run --gt_data imagenet --model_id 256x256_diffusion --w_radius 3 --w_channel 1 --w_pattern ring --start 0 --end 10000 --with_tracking
for : python run_tree_ring_watermark_imagenet_fid.py --run_name imgnet_fid_run --gt_data imagenet --model_id 256x256_diffusion --w_radius 3 --w_channel 1 --w_pattern ring --start 0 --end 10000 --with_tracking
it's coming 29.855
That's weird. I lost all my wandb logs, so I'm gonna rerun it over the weekend to see what's the problem.
Thank you. will be waiting for the good news.
Hi @unforgettablexD , I have ran python run_tree_ring_watermark_imagenet_fid.py --run_name imgnet_fid --gt_data imagenet --model_id 256x256_diffusion --w_radius 2 --w_channel 2 --w_pattern ring --start 0 --end 10000 --with_tracking --run_no_w, where I used channel 2 for watermarking. I got 19.40 for non-watermarked case and 18.11 for watermarked case.
I think there are two things you want to try to get even lower FID scores (I learned from how other people do FID in diffusion papers):
- One thing I remember I did is to add more images to the ground truth set. In the paper, I think I used the whole imagenet as the ground truth set, and the experiment I did above used 50000 images. I think having more generated images also helps.
- You may also want to resize/crop the ground truth images to 256x256, since we are generating 256x265 images.
Hope this helps, and let me know if you have further questions!