How to lower FID

Question

How to lower FID

Closed this issue 4 months ago · 8 comments

Thanks for providing the code. But, I think an error occured during evaluation.

I downloaded your checkpoint model and tried to evaluate using this code: CUDA_VISIBLE_DEVICES=0 python main.py --eval --num_images 50000 --flagfile ./logs/DDPM_CIFAR10_EPS/flagfile.txt. And the output script was this:

Files already downloaded and verified
Model params: 34.21 M
generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:53:24<00:00, 72.65s/it]
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.????????????????????????????????????????| 391/391 [7:53:24<00:00, 64.47s/it]
  warnings.warn(
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:51:47<00:00, 72.40s/it]
0/800000 IS:1.174, IS_std:0.001, FID:460.166, IS_EMA:1.176, IS_std_EMA:0.002, FID_EMA:460.171                                                                                                                                                                                                                       
  7%|???????????????????                                                                                                                                                                                                                                   | 59259/800000 [20:24:53<49:49:44,  4.13it/s, loss=0.029]
7%|???????????????????                                                                                                                                                                                                                                  | 59259/800000 [20:24:53<255:11:17,  1.24s/it, loss=0.029]

At this, I have some questions.
First, what do those warning messages mean? Is it normal for those messages to be printed out?
Second, why does the FID print out as 460.166? Does this relate to the first question?
Third, what is the progress bar that appears after the performance indicators?

Answer 1 · 2024-06-09T05:12:08.000Z

I reuploaded the checkpoint model and retried to evaluate using this code: CUDA_VISIBLE_DEVICES=3 python main.py --flagfile ./logs/DDPM_CIFAR10_EPS/flagfile.txt --notrain --eval. And the output script was this:

generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:52:59<00:00, 72.58s/it]
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
Model     : IS: 8.988(0.143), FID:  7.729 
...

I solved the problems by changing sample size and sample step. But your FID (3.249) and IS (9.475) are better than mine. How can I lower the FID from 7.xx to 3.xx through evaluation?

Answer 2 · 2024-06-10T10:48:00.000Z

Thanks for providing the code. But, I think an error occured during evaluation.

I downloaded your checkpoint model and tried to evaluate using this code: CUDA_VISIBLE_DEVICES=0 python main.py --eval --num_images 50000 --flagfile ./logs/DDPM_CIFAR10_EPS/flagfile.txt. And the output script was this:

Files already downloaded and verified
Model params: 34.21 M
generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:53:24<00:00, 72.65s/it]
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.????????????????????????????????????????| 391/391 [7:53:24<00:00, 64.47s/it]
  warnings.warn(
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:51:47<00:00, 72.40s/it]
0/800000 IS:1.174, IS_std:0.001, FID:460.166, IS_EMA:1.176, IS_std_EMA:0.002, FID_EMA:460.171                                                                                                                                                                                                                       
  7%|???????????????????                                                                                                                                                                                                                                   | 59259/800000 [20:24:53<49:49:44,  4.13it/s, loss=0.029]
7%|???????????????????                                                                                                                                                                                                                                  | 59259/800000 [20:24:53<255:11:17,  1.24s/it, loss=0.029]

At this, I have some questions. First, what do those warning messages mean? Is it normal for those messages to be printed out? Second, why does the FID print out as 460.166? Does this relate to the first question? Third, what is the progress bar that appears after the performance indicators?

The pre-trained model API of torchvision has some changes since version 0.13. The legacy usage is no longer updated, but it does not affect past results.

Answer 3 · 2024-06-10T10:49:35.000Z

I reuploaded the checkpoint model and retried to evaluate using this code: CUDA_VISIBLE_DEVICES=3 python main.py --flagfile ./logs/DDPM_CIFAR10_EPS/flagfile.txt --notrain --eval. And the output script was this:

generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:52:59<00:00, 72.58s/it]
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
Model     : IS: 8.988(0.143), FID:  7.729 
...

I solved the problems by changing sample size and sample step. But your FID (3.249) and IS (9.475) are better than mine. How can I lower the FID from 7.xx to 3.xx through evaluation?

Can you provide the versions of PyTorch and CUDA you are using? I will try to fix it as soon as possible.

Thanks.

Answer 4 · 2024-06-11T01:45:03.000Z

I reuploaded the checkpoint model and retried to evaluate using this code: CUDA_VISIBLE_DEVICES=3 python main.py --flagfile ./logs/DDPM_CIFAR10_EPS/flagfile.txt --notrain --eval. And the output script was this:
generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:52:59<00:00, 72.58s/it]
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
Model     : IS: 8.988(0.143), FID:  7.729 
...
I solved the problems by changing sample size and sample step. But your FID (3.249) and IS (9.475) are better than mine. How can I lower the FID from 7.xx to 3.xx through evaluation?
Can you provide the versions of PyTorch and CUDA you are using? I will try to fix it as soon as possible.

Thanks.

Here are the versions of PyTorch and CUDA.

PyTorch: 2.2.2
CUDA: 11.7

Answer 5 · 2024-06-11T03:34:21.000Z

I have tested the pretrained model with the following versions:

torch==2.3.1
CUDA 12.1

The results are as follows:

Model     : IS: 8.882(0.115), FID:  8.060
Model(EMA): IS: 9.493(0.154), FID:  3.210

I believe that evaluating the model twice is the main source of confusion. The second evaluation (EMA) is the primary result for DDPM, so I will make a commit that only evaluates the EMA model for clarity.
The UserWarning will also be fixed.

See the latest commit f804ccb for details.
Thanks.

Answer 6 · 2024-06-11T05:13:31.000Z

Thanks for your reply! In DDPM, the FID came out to be 3.17, but your FID came out to be 3.210 when applying EMA. Was EMA applied in DDPM as well? When I searched for DDPM paper, I cannot find such content.

Answer 7 · 2024-06-11T05:26:45.000Z

3.17 and 3.21 are close enough. It is possible to get the FID of 3.17 if you run the evaluation several times.

You can find the EMA operation here in the official implementation. As far as i know, they did not mention it in the paper, but it is a common trick for image generative models.

Answer 8 · 2024-06-11T05:33:19.000Z

My questions has been resolved. Thank you! I will close this issue.