torch.nn.parallel.DistributedDataParallel hang on

Question

torch.nn.parallel.DistributedDataParallel hang on

Crd1140234468 opened this issue a year ago · 5 comments

I encountered ‘torch.nn.parallel.DistributedDataParallel hang on’ problemwhen I run the train_depth.py. I found that the program cannot enter the statement "dist._verify_model_across_ranks"

How to solve this problem

Answer 1 · 2023-09-21T08:10:41.000Z

This is a function inside torch

Answer 2 · 2023-09-21T08:12:27.000Z

Also, here's the problem I'm having with multiple GPUs

Answer 3 · 2023-09-22T14:13:39.000Z

what's the command you run?

Answer 4 · 2023-09-25T05:29:20.000Z

what's the command you run?

CUDA_VISIBLE_DEVICES=1,3 python -m torch.distributed.launch --nproc_per_node=2 --master_port 8888 test11.py --bsize=8

Answer 5 · 2023-09-25T05:34:02.000Z

what's the command you run?
Currently, model_ad can be loaded into torch.nn.parallel.DistributedDataParallel, but when the model is set to sd-v1-4.ckpt, it cannot be loaded into torch.nn.parallel.DistributedDataParallel, and it will get stuck.