Script Reports Lack of GPU Support Despite Correct Environment Setup
Opened this issue · 1 comments
Environment:
Operating System: Ubuntu on GCP
Python Version: 3.9.19
PyTorch Version: 2.0.1
CUDA Version: 11.7
GPU Model: Tesla T4 (as verified by torch.cuda.get_device_name(0))
Issue Description:
I'm encountering an issue where the fit_3dmm_landmark.py script reports that it is "Not compiled with GPU support" despite having a fully supported and accessible GPU environment. My CUDA installation is verified, and PyTorch recognizes the GPU correctly (as evidenced by torch.cuda.is_available() returning True and torch.cuda.get_device_name(0) successfully reporting the GPU model).
I have ensured that the environment variables are correctly set, including CUDA_VISIBLE_DEVICES, and my PyTorch and related libraries are aligned with the CUDA version installed.
Steps to Reproduce:
Set environment variables (including VIDEO_ID, PYTHONPATH, and CUDA_VISIBLE_DEVICES).
Execute the script using: python data_gen/utils/process_video/fit_3dmm_landmark.py --ds_name=nerf --vid_dir=data/raw/videos/${VIDEO_ID}.mp4 --reset --debug --id_mode=global.
The process initiates but eventually reports that it was not compiled with GPU support, falling back to CPU processing.
Expected Behavior:
The script utilizes the GPU for processing, as the environment is correctly configured for CUDA and PyTorch GPU operations.
Actual Behavior:
The script fails to recognize GPU support and defaults to CPU processing, significantly affecting performance and feasibility for my use case.
Additional Context:
All diagnostic checks for CUDA and PyTorch GPU capabilities pass without issue.
Environment variables are set to explicitly target GPU usage.
A restart of the system/server has been performed to ensure all environment variables and configurations are correctly loaded.
Is there a specific compilation step or configuration that might be missing for enabling GPU support within this script? Any guidance or assistance on how to resolve this would be greatly appreciated.
Terminal Command:
export VIDEO_ID=Robert
export PYTHONPATH=./
export CUDA_VISIBLE_DEVICES=0
python data_gen/utils/process_video/fit_3dmm_landmark.py --ds_name=nerf --vid_dir=data/raw/videos/${VIDEO_ID}.mp4 --reset --debug --id_mode=global
Terminal Output:
args Namespace(vid_dir='data/raw/videos/Robert.mp4', ds_name='nerf', seed=0, process_id=0, total_process=1, id_mode='global', keypoint_mode='mediapipe', large_yaw_threshold=9999999.9, debug=True, reset=True, load_names=False)
['data/raw/videos/Robert.mp4']
process 0: fitting 3dmm ...: 0%| | 0/1 [00:00<?, ?it/s]batch 0 | loss_lan: 0.93, loss_reg_id: 0.75,loss_reg_exp: 0.27,loss_lap_ldm:0.0041
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0680, loss_vel_euler: 0.0047, loss_vel_trans: 0.0171
batch 1 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0028
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0550, loss_vel_euler: 0.0030, loss_vel_trans: 0.0135
batch 2 | loss_lan: 0.76, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0027
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0521, loss_vel_euler: 0.0032, loss_vel_trans: 0.0133
batch 3 | loss_lan: 0.77, loss_reg_id: 0.75,loss_reg_exp: 0.17,loss_lap_ldm:0.0015
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0447, loss_vel_euler: 0.0035, loss_vel_trans: 0.0176
batch 4 | loss_lan: 0.84, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0034
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0635, loss_vel_euler: 0.0042, loss_vel_trans: 0.0166
batch 5 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0042
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0437, loss_vel_euler: 0.0022, loss_vel_trans: 0.0112
batch 6 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0010
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0359, loss_vel_euler: 0.0028, loss_vel_trans: 0.0093
batch 7 | loss_lan: 0.91, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0041
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0727, loss_vel_euler: 0.0055, loss_vel_trans: 0.0154
batch 8 | loss_lan: 0.82, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0020
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0478, loss_vel_euler: 0.0036, loss_vel_trans: 0.0081
batch 9 | loss_lan: 0.88, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0039
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0630, loss_vel_euler: 0.0037, loss_vel_trans: 0.0128
batch 10 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0011
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0390, loss_vel_euler: 0.0039, loss_vel_trans: 0.0118
batch 11 | loss_lan: 0.82, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0031
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0543, loss_vel_euler: 0.0044, loss_vel_trans: 0.0120
batch 12 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.17,loss_lap_ldm:0.0041
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0667, loss_vel_euler: 0.0054, loss_vel_trans: 0.0192
batch 13 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0033
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0478, loss_vel_euler: 0.0021, loss_vel_trans: 0.0118
batch 14 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0030
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0589, loss_vel_euler: 0.0043, loss_vel_trans: 0.0139
batch 15 | loss_lan: 0.87, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0017
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0377, loss_vel_euler: 0.0021, loss_vel_trans: 0.0081
batch 16 | loss_lan: 0.93, loss_reg_id: 0.75,loss_reg_exp: 0.18,loss_lap_ldm:0.0031
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0583, loss_vel_euler: 0.0052, loss_vel_trans: 0.0158
batch 17 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.17,loss_lap_ldm:0.0042
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0566, loss_vel_euler: 0.0022, loss_vel_trans: 0.0151
batch 18 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0048
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0777, loss_vel_euler: 0.0046, loss_vel_trans: 0.0163
batch 19 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0024
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0507, loss_vel_euler: 0.0050, loss_vel_trans: 0.0100
batch 20 | loss_lan: 0.91, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0032
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0534, loss_vel_euler: 0.0044, loss_vel_trans: 0.0145
batch 21 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0033
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0604, loss_vel_euler: 0.0034, loss_vel_trans: 0.0147
batch 22 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0053
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0659, loss_vel_euler: 0.0035, loss_vel_trans: 0.0163
batch 23 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0036
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0595, loss_vel_euler: 0.0019, loss_vel_trans: 0.0113
batch 24 | loss_lan: 0.79, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0044
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0600, loss_vel_euler: 0.0023, loss_vel_trans: 0.0137
batch 25 | loss_lan: 0.85, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0039
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0645, loss_vel_euler: 0.0048, loss_vel_trans: 0.0142
batch 26 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.16,loss_lap_ldm:0.0042
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0624, loss_vel_euler: 0.0051, loss_vel_trans: 0.0133
batch 27 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0018
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0447, loss_vel_euler: 0.0029, loss_vel_trans: 0.0063
batch 28 | loss_lan: 0.71, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0011
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0354, loss_vel_euler: 0.0017, loss_vel_trans: 0.0067
batch 29 | loss_lan: 0.84, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0017
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0435, loss_vel_euler: 0.0042, loss_vel_trans: 0.0149
batch 30 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0027
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0487, loss_vel_euler: 0.0022, loss_vel_trans: 0.0107
batch 31 | loss_lan: 0.82, loss_reg_id: 0.75,loss_reg_exp: 0.16,loss_lap_ldm:0.0024
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0587, loss_vel_euler: 0.0055, loss_vel_trans: 0.0169
batch 32 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0067
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0752, loss_vel_euler: 0.0048, loss_vel_trans: 0.0172
batch 33 | loss_lan: 0.69, loss_reg_id: 0.75,loss_reg_exp: 0.09,loss_lap_ldm:0.0007
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0271, loss_vel_euler: 0.0021, loss_vel_trans: 0.0065
batch 34 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0029
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0569, loss_vel_euler: 0.0034, loss_vel_trans: 0.0098
batch 35 | loss_lan: 0.69, loss_reg_id: 0.75,loss_reg_exp: 0.09,loss_lap_ldm:0.0006
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0273, loss_vel_euler: 0.0018, loss_vel_trans: 0.0057
batch 36 | loss_lan: 0.77, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0016
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0496, loss_vel_euler: 0.0031, loss_vel_trans: 0.0092
batch 37 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.16,loss_lap_ldm:0.0027
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0551, loss_vel_euler: 0.0039, loss_vel_trans: 0.0187
batch 38 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0027
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0586, loss_vel_euler: 0.0045, loss_vel_trans: 0.0145
batch 39 | loss_lan: 0.82, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0032
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0591, loss_vel_euler: 0.0048, loss_vel_trans: 0.0142
batch 40 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0030
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0550, loss_vel_euler: 0.0047, loss_vel_trans: 0.0134
batch 41 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0007
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0303, loss_vel_euler: 0.0018, loss_vel_trans: 0.0077
batch 42 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0015
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0385, loss_vel_euler: 0.0029, loss_vel_trans: 0.0087
batch 43 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0013
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0423, loss_vel_euler: 0.0034, loss_vel_trans: 0.0116
batch 44 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0021
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0456, loss_vel_euler: 0.0024, loss_vel_trans: 0.0082
batch 45 | loss_lan: 0.86, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0024
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0575, loss_vel_euler: 0.0053, loss_vel_trans: 0.0123
batch 46 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0009
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0328, loss_vel_euler: 0.0027, loss_vel_trans: 0.0072
batch 47 | loss_lan: 0.85, loss_reg_id: 0.75,loss_reg_exp: 0.16,loss_lap_ldm:0.0023
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0559, loss_vel_euler: 0.0051, loss_vel_trans: 0.0145
batch 48 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0027
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0414, loss_vel_euler: 0.0022, loss_vel_trans: 0.0079
batch 49 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.09,loss_lap_ldm:0.0017
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0333, loss_vel_euler: 0.0022, loss_vel_trans: 0.0070
batch 50 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0034
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0552, loss_vel_euler: 0.0045, loss_vel_trans: 0.0127
batch 51 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0025
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0535, loss_vel_euler: 0.0037, loss_vel_trans: 0.0107
batch 52 | loss_lan: 0.82, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0045
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0713, loss_vel_euler: 0.0039, loss_vel_trans: 0.0135
batch 53 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0028
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0566, loss_vel_euler: 0.0041, loss_vel_trans: 0.0139
batch 54 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0012
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0390, loss_vel_euler: 0.0016, loss_vel_trans: 0.0089
batch 55 | loss_lan: 0.67, loss_reg_id: 0.75,loss_reg_exp: 0.09,loss_lap_ldm:0.0009
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0265, loss_vel_euler: 0.0014, loss_vel_trans: 0.0071
batch 56 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0027
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0558, loss_vel_euler: 0.0025, loss_vel_trans: 0.0145
batch 57 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0018
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0459, loss_vel_euler: 0.0030, loss_vel_trans: 0.0138
batch 58 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0023
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0505, loss_vel_euler: 0.0028, loss_vel_trans: 0.0149
batch 59 | loss_lan: 0.84, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0028
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0602, loss_vel_euler: 0.0058, loss_vel_trans: 0.0208
batch 60 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0010
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0365, loss_vel_euler: 0.0020, loss_vel_trans: 0.0093
batch 61 | loss_lan: 0.69, loss_reg_id: 0.75,loss_reg_exp: 0.09,loss_lap_ldm:0.0023
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0489, loss_vel_euler: 0.0031, loss_vel_trans: 0.0103
batch 62 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0018
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0537, loss_vel_euler: 0.0040, loss_vel_trans: 0.0195
batch 63 | loss_lan: 0.70, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0022
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0461, loss_vel_euler: 0.0015, loss_vel_trans: 0.0086
batch 64 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0020
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0375, loss_vel_euler: 0.0009, loss_vel_trans: 0.0075
batch 65 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.16,loss_lap_ldm:0.0042
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0630, loss_vel_euler: 0.0044, loss_vel_trans: 0.0158
batch 66 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0032
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0653, loss_vel_euler: 0.0045, loss_vel_trans: 0.0138
batch 67 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0038
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0692, loss_vel_euler: 0.0067, loss_vel_trans: 0.0137
batch 68 | loss_lan: 0.71, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0016
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0417, loss_vel_euler: 0.0023, loss_vel_trans: 0.0091
batch 69 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0028
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0441, loss_vel_euler: 0.0017, loss_vel_trans: 0.0078
batch 70 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0013
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0411, loss_vel_euler: 0.0027, loss_vel_trans: 0.0099
batch 71 | loss_lan: 0.88, loss_reg_id: 0.75,loss_reg_exp: 0.19,loss_lap_ldm:0.0064
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0808, loss_vel_euler: 0.0078, loss_vel_trans: 0.0201
batch 72 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0021
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0484, loss_vel_euler: 0.0032, loss_vel_trans: 0.0109
batch 73 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0022
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0469, loss_vel_euler: 0.0048, loss_vel_trans: 0.0107
batch 74 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0020
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0516, loss_vel_euler: 0.0023, loss_vel_trans: 0.0106
batch 75 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0018
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0506, loss_vel_euler: 0.0044, loss_vel_trans: 0.0167
batch 76 | loss_lan: 0.76, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0028
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0518, loss_vel_euler: 0.0032, loss_vel_trans: 0.0089
batch 77 | loss_lan: 0.71, loss_reg_id: 0.75,loss_reg_exp: 0.09,loss_lap_ldm:0.0002
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0163, loss_vel_euler: 0.0006, loss_vel_trans: 0.0033
batch 78 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0031
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0602, loss_vel_euler: 0.0046, loss_vel_trans: 0.0129
batch 79 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0038
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0684, loss_vel_euler: 0.0068, loss_vel_trans: 0.0191
batch 80 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0020
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0460, loss_vel_euler: 0.0028, loss_vel_trans: 0.0104
batch 81 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0023
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0481, loss_vel_euler: 0.0045, loss_vel_trans: 0.0115
batch 82 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.16,loss_lap_ldm:0.0027
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0579, loss_vel_euler: 0.0040, loss_vel_trans: 0.0115
batch 83 | loss_lan: 0.69, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0029
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0521, loss_vel_euler: 0.0026, loss_vel_trans: 0.0089
batch 84 | loss_lan: 0.67, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0032
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0453, loss_vel_euler: 0.0018, loss_vel_trans: 0.0109
batch 85 | loss_lan: 0.90, loss_reg_id: 0.75,loss_reg_exp: 0.20,loss_lap_ldm:0.0034
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0728, loss_vel_euler: 0.0059, loss_vel_trans: 0.0183
batch 86 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0025
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0535, loss_vel_euler: 0.0027, loss_vel_trans: 0.0081
batch 87 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0024
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0502, loss_vel_euler: 0.0015, loss_vel_trans: 0.0083
batch 88 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.17,loss_lap_ldm:0.0024
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0599, loss_vel_euler: 0.0055, loss_vel_trans: 0.0165
batch 89 | loss_lan: 0.88, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0041
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0695, loss_vel_euler: 0.0050, loss_vel_trans: 0.0186
batch 90 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0024
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0595, loss_vel_euler: 0.0075, loss_vel_trans: 0.0147
batch 91 | loss_lan: 0.76, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0026
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0544, loss_vel_euler: 0.0038, loss_vel_trans: 0.0108
batch 92 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0032
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0642, loss_vel_euler: 0.0060, loss_vel_trans: 0.0175
batch 93 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0015
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0389, loss_vel_euler: 0.0028, loss_vel_trans: 0.0095
batch 94 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0030
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0597, loss_vel_euler: 0.0053, loss_vel_trans: 0.0123
batch 95 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0017
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0441, loss_vel_euler: 0.0017, loss_vel_trans: 0.0065
batch 96 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0017
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0435, loss_vel_euler: 0.0021, loss_vel_trans: 0.0119
batch 97 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0020
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0437, loss_vel_euler: 0.0018, loss_vel_trans: 0.0108
batch 98 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0022
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0509, loss_vel_euler: 0.0027, loss_vel_trans: 0.0097
batch 99 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0025
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0423, loss_vel_euler: 0.0019, loss_vel_trans: 0.0084
batch 100 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0055
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0742, loss_vel_euler: 0.0038, loss_vel_trans: 0.0121
batch 101 | loss_lan: 0.79, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0026
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0540, loss_vel_euler: 0.0042, loss_vel_trans: 0.0101
batch 102 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0023
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0530, loss_vel_euler: 0.0039, loss_vel_trans: 0.0099
batch 103 | loss_lan: 0.79, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0017
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0465, loss_vel_euler: 0.0018, loss_vel_trans: 0.0086
batch 104 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0043
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0665, loss_vel_euler: 0.0043, loss_vel_trans: 0.0105
batch 105 | loss_lan: 0.79, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0043
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0561, loss_vel_euler: 0.0036, loss_vel_trans: 0.0155
batch 106 | loss_lan: 0.77, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0031
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0535, loss_vel_euler: 0.0021, loss_vel_trans: 0.0100
batch 107 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0021
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0506, loss_vel_euler: 0.0048, loss_vel_trans: 0.0153
batch 108 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.24,loss_lap_ldm:0.0021
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0517, loss_vel_euler: 0.0046, loss_vel_trans: 0.0103
batch 109 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.20,loss_lap_ldm:0.0019
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0419, loss_vel_euler: 0.0025, loss_vel_trans: 0.0074
batch 110 | loss_lan: 0.79, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0020
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0576, loss_vel_euler: 0.0038, loss_vel_trans: 0.0110
batch 111 | loss_lan: 0.76, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0028
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0562, loss_vel_euler: 0.0027, loss_vel_trans: 0.0105
batch 112 | loss_lan: 0.77, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0023
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0503, loss_vel_euler: 0.0033, loss_vel_trans: 0.0108
batch 113 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0016
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0432, loss_vel_euler: 0.0017, loss_vel_trans: 0.0058
batch 114 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0018
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0506, loss_vel_euler: 0.0046, loss_vel_trans: 0.0159
batch 115 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0054
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0729, loss_vel_euler: 0.0034, loss_vel_trans: 0.0143
batch 116 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0025
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0480, loss_vel_euler: 0.0042, loss_vel_trans: 0.0132
batch 117 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0016
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0444, loss_vel_euler: 0.0028, loss_vel_trans: 0.0099
batch 118 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0036
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0596, loss_vel_euler: 0.0051, loss_vel_trans: 0.0155
batch 119 | loss_lan: 0.84, loss_reg_id: 0.75,loss_reg_exp: 0.28,loss_lap_ldm:0.0014
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0432, loss_vel_euler: 0.0030, loss_vel_trans: 0.0138
batch 120 | loss_lan: 0.85, loss_reg_id: 0.75,loss_reg_exp: 0.31,loss_lap_ldm:0.0011
|--------loss_vel_id: 0.0000, loss_vel_exp: 0.0413, loss_vel_euler: 0.0025, loss_vel_trans: 0.0137
create rasterizer on device cuda:0cessed/videos/Robert/debug_fit_3dmm.mp4..: 0%| | 0/250 [00:00<?, ?it/s]
rendering debug video to data/processed/videos/Robert/debug_fit_3dmm.mp4..: 0%| | 0/250 [00:00<?, ?it/s]
data/raw/videos/Robert.mp4 Not compiled with GPU support
finished 1 / 1 = 1.0000, failed 1 / 1 = 1.0000
process 0: fitting 3dmm ...: 100%|██████████████████████████████████████████████████████████████████████████| 1/1 [00:59<00:00, 59.58s/it]
all failed image names: ['data/raw/videos/Robert.mp4']
All finished!
please check the memory space available, try to remove the files which were created in the failed attempts of precprocessing data/training, also remove the trash files so that you can make the space available
Also another thing for the error of failed images please check that if you have done the complete preprocessing of data step by step rightly.