How to calculate FID? Any bash scripts?

[sumorday]

The celebahq_stat.npy file for FID testing is already in the pytorch_fid folder, and the generated images are in ../saved_info/latent_flow/celeba_256/celeb256_f8_adm/image_epoch_20.png.

Should I run !bash ../bash_scripts/run_test_ddp.sh ../test_args/celeb256_adm.txt

or just !bash ../bash_scripts/run_test_ddp.sh?

any bash scripts examples for running FID calculation? Thank you in advance.

[sumorday]

!bash ./bash_scripts/run_test_ddp.sh test_args/celeb256_adm.txt

[sumorday]

Traceback (most recent call last):
File "/content/drive/MyDrive/LFM/test_flow_latent_ddp.py", line 263, in
main(args)
File "/content/drive/MyDrive/LFM/test_flow_latent_ddp.py", line 59, in main
ckpt = torch.load("./saved_info/latent_flow/{}/{}/model_{}.pth".format(args.dataset, args.exp, args.epoch_id))
File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 986, in load
with _open_file_like(f, 'rb') as opened_file:
File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 435, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 416, in init
super().init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: './saved_info/latent_flow/celeba_256/celeb256_f8_adm/model_450.pth'

[sumorday]

CelebA 256x256
maybe run_test.sh ,this file? not run_test_ddp.sh ?

!bash ./bash_scripts/run_test.sh test_args/celeb256_adm.txt

FileNotFoundError: [Errno 2] No such file or directory: './saved_info/latent_flow/celeba_256/celeb256_f8_adm/model_450.pth'

[sumorday]

should I change the EPOCH_ID=450 to EPOCH_ID=30 if i only running it 30 times?
otherwise the test.sh can not find the 450.pth file?

[hao-pt]

Hi,

Basically, run_test.sh & run_test_ddp.sh are the same where both can be used to generate output samples and calculate FID. The main difference is that run_test_ddp.sh support multi-gpu running for faster image generation.

The error happened due to the mismatch of arg EPOCH_ID. You should adjust it to corresponding epoch id of your trained model such as EPOCH_ID=30 in your case.

If you already generated images in a particular folder, you could use this script for direct FID calculation:
python pytorch_fid/fid_score.py ./pytorch_fid/celebahq_stat.npy path_to_output_dir

[sumorday]

Thank you so much!

!python pytorch_fid/fid_score.py ./pytorch_fid/celebahq_stat.npy ./saved_info/latent_flow/celeba_256/celeb256_f8_adm_saved

_pickle.UnpicklingError: pickle data was truncated

Do I have to finish running, for example, 600 or 450 times, before I can test FID?
My folder is as shown in the picture. For the CelebA 256x256 dataset, it seems that a new generated image is recorded every 5 steps.
I can't seem to test FID.

[sumorday]

Downgrade scipy to 1.11.1 solved the problem for me.
pip install scipy==1.7.3

Not so sure if the FID value is correctly or not.

!python pytorch_fid/fid_score.py ./pytorch_fid/celebahq_stat.npy ./saved_info/latent_flow/celeba_256/celeb256_f8_adm

/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=None.
warnings.warn(msg)
Warning: batch size is bigger than the data size. Setting batch size to data size
100% 1/1 [00:01<00:00, 1.04s/it]
FID: 510.65202745674986