timeout when training

Question

timeout when training

Closed this issue 3 years ago · 3 comments

Hi, when I want to training model with aishell1, I meet the problem that connect() timeout . Can you help me?

Answer 1 · 2022-03-04T07:00:23.000Z

Hi, could you post some logs here so I can check the problem?

Answer 2 · 2022-03-04T07:32:26.000Z

Sorry，I can't download the logs directly , The errors show as follow: " RuntimeError : connetct () timed out " ; in launch configs : rdzv_configs: { 'rank': 1, 'timeout': 900}

Answer 3 · 2022-03-04T09:21:37.000Z

I'm a bit confused about the connect() function. Is that the k2.connect(), or some communication operation in DDP? I suppose we don't need a dictionary called rdzv_configs?