kakaobrain/torchgpipe

About the debugger for running torchgpipe

Real-ZeminJiang opened this issue · 3 comments

Real-ZeminJiang commented

Hey authors @sublee @ummae @kimdwkimdw @huntrax11 @ildoonet :

I am wondering how to debug the async pytorch in the pipeline.py. There should be some proper debugger that can trace all the code both for CPU and GPU. Thank you very much for the help!

Real-ZeminJiang commented

And when I trying to step into

            output = model(input)

inside main.py,

The program just stuck there and have no response for a long time. Can anyone give me some advice? Thank you very much!

sublee commented

This question does not seem to be related to torchgpipe. However, when I develop torchgpipe, NVIDIA Nsight Systems which is a GPU profiler was very useful to me.

To advise the program in stuck, I need more context. Which main.py did you run? Which GPUs do you use? Was any GPU utilized?

sublee commented

I close this issue because it's not related to torchgpipe and there's no response for a long while.