Port NVIDIA Nsight profiling to gpt-neox
Quentin-Anthony opened this issue · 0 comments
Quentin-Anthony commented
Megatron supports an easy way to profile a few steps in training: https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/arguments.py#L834
Currently I'm just manually running nsight, but this would make things easier.
- Port Megatron-LM nsight changes to gpt-neox (done: #1153)
- Document the change in the README