CUDA OUT OF MEMORY
Lxzzzzzzzzzzz opened this issue · 2 comments
Can 4090 run the demo?
(lart) lxz@a4061-MS-7E06:~/projects/LART$ python scripts/demo.py video.source="assets/jump.mp4" +half=True
[02/26 14:54:37] INFO No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate' acceleratesupport.py:17
[2024-02-26 14:54:38,556][pytorch_lightning.utilities.migration.utils][INFO] - Lightning automatically upgraded your loaded checkpoint from v1.8.1 to v2.2.0.post0. To apply the upgrade to your files permanently, run python -m pytorch_lightning.utilities.upgrade_checkpoint ../../.cache/4DHumans/logs/train/multiruns/hmr2/0/checkpoints/epoch=35-step=1000000.ckpt
WARNING: You are using a SMPL model, with only 10 shape coefficients.
[2024-02-26 14:54:40,603][phalp.trackers.PHALP][INFO] - Loading Predictor model...
[2024-02-26 14:54:40,730][phalp.trackers.PHALP][INFO] - Loading Detection model...
[2024-02-26 14:54:44,187][detectron2.checkpoint.detection_checkpoint][INFO] - [DetectionCheckpointer] Loading from https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/cascade_mask_rcnn_vitdet_h/f328730692/model_final_f05665.pkl ...
[2024-02-26 14:54:45,620][detectron2.checkpoint.detection_checkpoint][INFO] - [DetectionCheckpointer] Loading from https://dl.fbaipublicfiles.com/detectron2/COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x/139653917/model_final_2d9806.pkl ...
[2024-02-26 14:54:45,709][phalp.trackers.PHALP][INFO] - Setting up Visualizer...
[2024-02-26 14:54:45,954][phalp.utils.io][INFO] - Number of frames: 171
[2024-02-26 14:54:46,181][phalp.trackers.PHALP][INFO] - Setting up DeepSort...
[2024-02-26 14:54:46,181][phalp.trackers.PHALP][INFO] - Saving tracks at : outputs//results/jump
Tracking : jump ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% eta : 0:00:00 time elapsed : 0:00:49
[2024-02-26 14:55:36,588][pytorch_lightning.utilities.migration.utils][INFO] - Lightning automatically upgraded your loaded checkpoint from v1.8.1 to v2.2.0.post0. To apply the upgrade to your files permanently, run python -m pytorch_lightning.utilities.upgrade_checkpoint ../../.cache/4DHumans/logs/train/multiruns/hmr2/0/checkpoints/epoch=35-step=1000000.ckpt
WARNING: You are using a SMPL model, with only 10 shape coefficients.
[2024-02-26 14:55:38,351][main][INFO] - Loading Predictor model...
[2024-02-26 14:55:38,651][phalp.trackers.PHALP][INFO] - Loading Detection model...
[2024-02-26 14:55:41,734][detectron2.checkpoint.detection_checkpoint][INFO] - [DetectionCheckpointer] Loading from https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/cascade_mask_rcnn_vitdet_h/f328730692/model_final_f05665.pkl ...
[2024-02-26 14:55:43,150][detectron2.checkpoint.detection_checkpoint][INFO] - [DetectionCheckpointer] Loading from https://dl.fbaipublicfiles.com/detectron2/COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x/139653917/model_final_2d9806.pkl ...
[2024-02-26 14:55:43,309][phalp.trackers.PHALP][INFO] - Setting up Visualizer...
[2024-02-26 14:55:44,810][slowfast.visualization.predictor][INFO] - Start loading model weights.
[2024-02-26 14:55:44,810][slowfast.utils.checkpoint][INFO] - Loading network weights from /home/lxz/.cache/phalp/ava/mvit.pyth.
missing keys: []
unexpected keys: []
[2024-02-26 14:55:45,121][slowfast.visualization.predictor][INFO] - Finish loading model weights
Error executing job with overrides: ['video.source=assets/jump.mp4', '+half=True']
Traceback (most recent call last):
File "/home/lxz/projects/LART/scripts/demo.py", line 103, in main
lart_model.postprocessor.run_lart(pkl_path)
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/phalp/visualize/postprocessor.py", line 102, in run_lart
final_visuals_dic = self.post_process(final_visuals_dic, save_fast_tracks=self.cfg.post_process.save_fast_tracks, video_pkl_name=video_pkl_name)
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/phalp/visualize/postprocessor.py", line 36, in post_process
smoothed_fast_track_ = self.phalp_tracker.pose_predictor.smooth_tracks(fast_track_, moving_window=True, step=32, window=32)
File "/home/lxz/projects/LART/lart/utils/wrapper_phalp.py", line 226, in smooth_tracks
fast_track = self.add_slowfast_features(fast_track)
File "/home/lxz/projects/LART/lart/utils/wrapper_phalp.py", line 203, in add_slowfast_features
task_ = SlowFastWrapper(t_, cfg, list_of_all_frames, mid_bbox_, video_model, center_crop=center_crop)
File "/home/lxz/projects/LART/lart/utils/wrapper_pyslowfast.py", line 61, in SlowFastWrapper
task = video_model(task)
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/slowfast/visualization/predictor.py", line 110, in call
preds, feats = self.model(inputs, bboxes)
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/slowfast/models/video_model_builder.py", line 1239, in forward
x, thw = blk(x, thw)
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/fairscale/nn/checkpoint/checkpoint_activations.py", line 171, in _checkpointed_forward
return original_forward(module, *args, **kwargs)
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/slowfast/models/attention.py", line 547, in forward
x_block, thw_shape_new = self.attn(x_norm, thw_shape)
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/slowfast/models/attention.py", line 407, in forward
attn = cal_rel_pos_spatial(
File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/slowfast/models/attention.py", line 112, in cal_rel_pos_spatial
attn[:, :, sp_idx:, sp_idx:].view(B, -1, q_t, q_h, q_w, k_t, k_h, k_w)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.25 GiB. GPU 0 has a total capacty of 23.62 GiB of which 4.13 GiB is free. Including non-PyTorch memory, this process has 18.56 GiB memory in use. Of the allocated memory 17.68 GiB is allocated by PyTorch, and 428.70 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Hi @Lxzzzzzzzzzzz , did you find a solution to this?
Hi @Lxzzzzzzzzzzz , did you find a solution to this?
换个视频就行，别用那个例子