Query Regarding the evaluation code for FFHOQ512

Script for evaluating :
bash {GMPI_ROOT}/gmpi/eval/eval.sh {GMPI_ROOT} FFHQ512 20220727_104512601146 {Deep3DFaceRecon_PATH} nodebug

Can you Please help me in fixing the issue for evaluating the code for FFHQ512....

Error:
Load weights from /proj/cvl/users/x_fahkh/mn/ml-gmpi/ckpts/gmpi_pretrained/20220727_104512601146/generator.pth
Load weights from /proj/cvl/users/x_fahkh/mn/ml-gmpi/ckpts/gmpi_pretrained/20220727_104512601146/ema.pth
save_dir: /proj/cvl/users/x_fahkh/mn/ml-gmpi/ckpts/gmpi_pretrained/20220727_104512601146/planes_96_n_50000/psi_1.0/fid_kid/rgb
0%| | 0/50000 [00:00<?, ?it/s]
log: torch.Size([1, 96, 4, 512, 512]) (1, 512, 512, 3) (1, 512, 512, 1)
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 50000/50000 [3:22:49<00:00, 4.11it/s]

Compute FID/KID from
real: /proj/cvl/users/x_fahkh/mn/ml-gmpi/runtime_dataset/real_data/FFHQ512/FFHQ_real_res_512_n_50000
fake: /proj/cvl/users/x_fahkh/mn/ml-gmpi/ckpts/gmpi_pretrained/20220727_104512601146/planes_96_n_50000/psi_1.0/fid_kid/rgb

Creating feature extractor "inception-v3-compat" with features ['2048']
Extracting features from input1
Looking for samples non-recursivelty in "/proj/cvl/users/x_fahkh/mn/ml-gmpi/ckpts/gmpi_pretrained/20220727_104512601146/planes_96_n_50000/psi_1.0/fid_kid/rgb" with extensions png,jpg,jpeg
Found 50000 samples
Processing samples
Extracting features from input2
Looking for samples non-recursivelty in "/proj/cvl/users/x_fahkh/mn/ml-gmpi/runtime_dataset/real_data/FFHQ512/FFHQ_real_res_512_n_50000" with extensions png,jpg,jpeg
Found 71 samples
Traceback (most recent call last):
File "/proj/cvl/users/x_fahkh/mn/ml-gmpi/gmpi/eval/compute_fid_kid.py", line 29, in
verbose=True,
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/torch_fidelity/metrics.py", line 243, in calculate_metrics
featuresdict_2 = extract_featuresdict_from_input_id_cached(2, feat_extractor, **kwargs)
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/torch_fidelity/utils.py", line 372, in extract_featuresdict_from_input_id_cached
featuresdict = fn_recompute()
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/torch_fidelity/utils.py", line 360, in fn_recompute
return extract_featuresdict_from_input_id(input_id, feat_extractor, **kwargs)
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/torch_fidelity/utils.py", line 345, in extract_featuresdict_from_input_id
featuresdict = get_featuresdict_from_dataset(input, feat_extractor, batch_size, cuda, save_cpu_ram, verbose)
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/torch_fidelity/utils.py", line 111, in get_featuresdict_from_dataset
for bid, batch in enumerate(dataloader):
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1183, in _next_data
return self._process_data(data)
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
data.reraise()
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/torch/_utils.py", line 425, in reraise
raise self.exc_type(msg)
OSError: Caught OSError in DataLoader worker process 1.
Original Traceback (most recent call last):
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/torch_fidelity/datasets.py", line 31, in getitem
img = Image.open(path).convert('RGB')
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/PIL/Image.py", line 2822, in open
raise IOError("cannot identify image file %r" % (filename if filename else fp))
OSError: cannot identify image file '/proj/cvl/users/x_fahkh/mn/ml-gmpi/runtime_dataset/real_data/FFHQ512/FFHQ_real_res_512_n_50000/00070.png'

Can you Please have a look to this

I re-run the evaluation on my side and cannot reproduce this error.

Looking for samples non-recursivelty in "/proj/cvl/users/x_fahkh/mn/ml-gmpi/runtime_dataset/real_data/FFHQ512/FFHQ_real_res_512_n_50000" with extensions png,jpg,jpeg Found 71 samples

It seems like you only have 71 images under the path /proj/cvl/users/x_fahkh/mn/ml-gmpi/runtime_dataset/real_data/FFHQ512/FFHQ_real_res_512_n_50000. However, you are trying to evaluate 50k images.

Maybe you can delete /proj/cvl/users/x_fahkh/mn/ml-gmpi/runtime_dataset/real_data/FFHQ512/FFHQ_real_res_512_n_50000 and re-run the command to generate real images.

Note, if the real_path exists, the code will not re-generate real images since generating 50k images is costly. So please double check whether you interrupt the running such that the real-image generation is not complete:

ml-gmpi/gmpi/eval/prepare_real_data.py

Line 43 in c809703

if not os.path.exists(real_dir):

Some suggestions:

You may set the flag from nodebug to debug. In this way, you will only generate 2048 images for FID/KID, which will be much quicker and you can use this to check the path setup. However, keep in mind, 2048 is too small to provide a reasonable FID/KID evaluation.
I recommend not running the whole eval.sh at first. I have split the script in a command-by-command style. Just run one command at a time. After each command, double-check the anticipated path's content. I think in this way, you will easily pinpoint the issue if there are any.

@VIROBO-15 FYI - the readability of your issues would improve, if proper code formatting is applied.

Thank You for replying @fangchangma @Xiaoming-Zhao...

I am still facing some issues in evaluation
Script : # for Depth and pose

eval "$(conda shell.bash hook)"
conda activate gmpi
export MKL_NUM_THREADS=5 && export NUMEXPR_NUM_THREADS=5 &&
python ${REPO_DIR}/gmpi/eval/prepare_fake_data.py
--ckpt_path ${CKPT_DIR}/generator.pth
--seed ${SEED}
--save_dir ${GEO_DIR}
--nplanes ${N_PLANES}
--n_imgs ${N_GEO}
--task geometry
--exp_config ${CKPT_DIR}/config.pth
--save_depth 1
--dataset ${DATASET}
--truncation_psi ${TRUNCATION_PSI}
--stylegan2_sanity_check ${SANITY_CHECK}

eval "$(conda shell.bash hook)"
conda activate mtcnn_env
export MKL_NUM_THREADS=5 && export NUMEXPR_NUM_THREADS=5 &&
python ${REPO_DIR}/gmpi/eval/prepare_face_landmarks.py
--data_dir ${GEO_DIR}/geometry/rgb

eval "$(conda shell.bash hook)"
conda activate deep3d_pytorch
cd ${Deep3DFaceRecon_PATH}
python ${Deep3DFaceRecon_PATH}/estimate_pose_gmpi.py
--name=pretrained
--epoch=20
--gmpi_img_root ${GEO_DIR}/geometry/rgb
--gmpi_depth_root ${GEO_DIR}/geometry/depth
--gmpi_detect_root ${GEO_DIR}/geometry/rgb/detections

eval "$(conda shell.bash hook)"
conda activate gmpi
python ${REPO_DIR}/gmpi/eval/compute_geometry.py
--geo_dir ${GEO_DIR}/geometry
--angle_err 1

Error
sorted_f_list: 1024 ['/proj/cvl/users/x_fahkh/mn/ml-gmpi/ckpts/gmpi_pretrained/20220727_102024290372/planes_96_n_1024/psi_1.0/geometry/rgb/000000.png', '/proj/cvl/users/x_fahkh/mn/ml-gmpi/ckpts/gmpi_pretrained/20220727_102024290372/planes_96_n_1024/psi_1.0/geometry/rgb/000001.png', '/proj/cvl/users/x_fahkh/mn/ml-gmpi/ckpts/gmpi_pretrained/20220727_102024290372/planes_96_n_1024/psi_1.0/geometry/rgb/000002.png', '/proj/cvl/users/x_fahkh/mn/ml-gmpi/ckpts/gmpi_pretrained/20220727_102024290372/planes_96_n_1024/psi_1.0/geometry/rgb/000003.png', '/proj/cvl/users/x_fahkh/mn/ml-gmpi/ckpts/gmpi_pretrained/20220727_102024290372/planes_96_n_1024/psi_1.0/geometry/rgb/000004.png']

100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 1024/1024 [00:46<00:00, 22.09it/s]
/proj/cvl/users/x_fahkh/mn/ml-gmpi/gmpi/eval/eval.sh: line 158: cd: nodebug: No such file or directory
python: can't open file 'nodebug/estimate_pose_gmpi.py': [Errno 2] No such file or directory
0%| | 0/1023 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/proj/cvl/users/x_fahkh/mn/ml-gmpi/gmpi/eval/compute_geometry.py", line 93, in
tmp_depth_err = compute_depth_err(tmp_aligned_depth_f, tmp_pred_depth_f, tmp_pred_mask_f)
File "/proj/cvl/users/x_fahkh/mn/ml-gmpi/gmpi/eval/compute_geometry.py", line 26, in compute_depth_err
depth = np.load(alinged_depth_f)
File "/home/x_fahkh/.conda/envs/gmpi/lib/python3.7/site-packages/numpy/lib/npyio.py", line 417, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/proj/cvl/users/x_fahkh/mn/ml-gmpi/ckpts/gmpi_pretrained/20220727_102024290372/planes_96_n_1024/psi_1.0/geometry/recon/aligned_depth/000000.npy'

@VIROBO-15 I recommend reading the trace log if possible:

python: can't open file 'nodebug/estimate_pose_gmpi.py': [Errno 2] No such file or directory

I guess estimate_pose_gmpi.py is not under nodebug. Instead, it should be in the folder for Deep3DFaceRecon_PATH:

ml-gmpi/gmpi/eval/eval.sh

Line 159 in c809703

python ${Deep3DFaceRecon_PATH}/estimate_pose_gmpi.py \

Thank you for reply......

I am Still facing some issues for calculating the depth and pose score..

Error:
Traceback (most recent call last):
File "/proj/cvl/users/x_fahkh/mn/ml-gmpi/Deep3DFaceRecon_pytorch/estimate_pose_gmpi.py", line 6, in
from options.test_options import TestOptions
File "/proj/cvl/users/x_fahkh/mn/ml-gmpi/Deep3DFaceRecon_pytorch/options/test_options.py", line 4, in
from .base_options import BaseOptions
File "/proj/cvl/users/x_fahkh/mn/ml-gmpi/Deep3DFaceRecon_pytorch/options/base_options.py", line 9, in
import models
File "/proj/cvl/users/x_fahkh/mn/ml-gmpi/Deep3DFaceRecon_pytorch/models/init.py", line 22, in
from models.base_model import BaseModel
File "/proj/cvl/users/x_fahkh/mn/ml-gmpi/Deep3DFaceRecon_pytorch/models/base_model.py", line 9, in
from . import networks
File "/proj/cvl/users/x_fahkh/mn/ml-gmpi/Deep3DFaceRecon_pytorch/models/networks.py", line 18, in
from .arcface_torch.backbones import get_model
ModuleNotFoundError: No module named 'models.arcface_torch'

Have you completed the setup of Deep3DFaceRecon_pytorch following the steps of its README?