IPython is great for debugging but using it in multi-process environment is awkward. mpdb
is designed to make multi-process debugging (esp. PyTorch multi-GPU) less frustrating.
pip install git+https://github.com/RalphMao/mpdb
- Add
mpdb.embed()
(similar toIPython.embed()
) to the target location of the python script. - Initially only Rank 0 will be activated with a IPython shell.
- To switch to other ranks, use line magic
%switch <rank_id>
.
- Add MPI and Slurm examples
- Support multi-process equivalent of
ipdb.set_trace
- More general sync mechanism without torch dependency
- Support multi-node debugging on Slurm