Wikikgv2 Model Training is Hanging

Question

Wikikgv2 Model Training is Hanging

HarryShomer opened this issue 2 years ago · 15 comments

Hi,

I'm having an issue where the model gets stuck while training. It typically happens early in the first epoch. Below is an example when running the smore/training/vec_scripts/train_shallow_wikikgv2.sh (unmodified sans the GPUs) on 4 NVIDIA RTX A6000 50GB GPUs.

It hangs forever unless I stop it with a keyboard interrupt. Doing so yields the following traceback (I only post a portion because it's very long and repetitive).

It seems like something is happening in the multiprocessing as it's hanging when sharing messages between processes.

Any help would be appreciated! @hyren

Thanks,
Harry

Answer 1 · 2022-10-09T02:30:30.000Z

Hi, this does not seem to be related to the smore repo. Here is an instruction I found, can you please check?
https://www.rdmamojo.com/2012/05/18/libibverbs/

Answer 2 · 2022-10-11T20:07:29.000Z

That doesn't seem to be the issue.

I tried seting up libibverbs, and while the warnings went away, the code is still hanging.

Answer 3 · 2022-10-11T21:14:10.000Z

Hi, can you say more about the details of the environment?

Answer 4 · 2022-10-11T21:43:50.000Z

Sure. Here's some basic info. Let me know if you would like anything else.

OS: Ubuntu 20.04.5 LTS
CUDA: 11.6.124
GPU(s): NVIDIA RTX A6000
Python: 3.9.12
PyTorch: 1.12.1

Answer 5 · 2022-10-11T21:49:20.000Z

Hi there, sorry for the inconvenience, could you please quickly try with only 1 gpu, and see if it still hangs? We are trying to figure out whether it is due to gpu-gpu communication, or the c++ sampler we used.

Answer 6 · 2022-10-11T21:57:27.000Z

Sorry, I should have been clearer in my initial comment. The code doesn't hang when running with just one GPU. This only occurs when using multiple GPUs.

Answer 7 · 2022-10-12T05:58:08.000Z

We tried one gpu again, actually it still hangs. Sorry for the inconvenience.

Answer 8 · 2022-10-12T07:26:35.000Z

Hi there,

Could you please kindly pull the latest code in the wikikgv2 branch, and add --train_async_rw=False to your script and try again? Since I'm unable to reproduce your issue, I'd like to see whether this would temporarily resolve the issue.

Answer 9 · 2022-10-12T22:06:19.000Z

Hi,

We tried your suggestion and now the training doesn't hang! But we got the similar issue in the evaluation. And we noticed your suggestion in another issue and follow the suggestion, but it still hangs.

I really appreciate your help!

Answer 10 · 2022-10-12T22:21:07.000Z

hi, what's the script you used?

Answer 11 · 2022-10-12T22:22:16.000Z

Hi, train_shallow_wikikgv2.sh in the training/vec_scripts folder.

Answer 12 · 2022-10-12T22:34:16.000Z

Just to make sure, you used the scripts in train_shallow_wikikgv2.sh and add the --train_async_rw=False flag?

Answer 13 · 2022-10-12T23:54:48.000Z

Actually no, since if we add it we got the unrecognized arguments error.

Answer 14 · 2022-10-12T23:57:30.000Z

have you pulled the most recent commits?

Answer 15 · 2022-10-13T03:16:09.000Z

Sorry we made some mistakes when we merged the code. Now we pulled the the latest code and it works for both training and evaluation!

Thank you so much!