NOTE: Our forward-backward kernel correlation heuristics are not working correctly with PyTorch v1.6. We are working to resolve it. In the mean time, please use Pytorch 1.5 or the 20.03 NGC container, which you can retrieve via:
docker pull http://nvcr.io/nvidia/pytorch:20.03-py3LATEST RELEASE: You are currently working on the master branch which tracks under-development progress towards the next release. The latest release of the PyProf is 3.5.0 and is available on branch r20.10.
PyProf is a tool that profiles and analyzes the GPU performance of PyTorch models. PyProf aggregates kernel performance from Nsight Systems or NvProf and provides the following additional features:
- Identifies the layer that launched a kernel: e.g. the association of ComputeOffsetsKernel with a concrete PyTorch layer or API is not obvious.
- Identifies the tensor dimensions and precision: without knowing the tensor dimensions and precision, it's impossible to reason about whether the actual (silicon) kernel time is close to maximum performance of such a kernel on the GPU. Knowing the tensor dimensions and precision, we can figure out the FLOPs and bandwidth required by a layer, and then determine how close to maximum performance the kernel is for that operation.
- Forward-backward correlation: PyProf determines what the forward pass step is that resulted in the particular weight and data gradients (wgrad, dgrad), which makes it possible to determine the tensor dimensions required by these backprop steps to assess their performance.
- Determines Tensor Core usage: PyProf can highlight the kernels that use Tensor Cores.
- Correlate the line in the user's code that launched a particular kernel (program trace).
The current release of PyProf is 3.5.0 and is available in the 20.10 release of the PyTorch container on NVIDIA GPU Cloud (NGC). The branch for this release is r20.10.
Clone the git repository
$ git clone https://github.com/NVIDIA/PyProf.git
Navigate to the top level PyProf directory
Install PyProf
$ pip install .
Verify installation is complete with pip list
$ pip list | grep pyprof
Should display
pyprof 3.7.0.dev0
Add the following lines to the PyTorch network you want to profile:
import torch.cuda.profiler as profiler import pyprof pyprof.init()
Profile with NVProf or Nsight Systems to generate a SQL file.
$ nsys profile -f true -o net --export sqlite python net.py
Run the parse.py script to generate the dictionary.
$ python -m pyprof.parse net.sqlite > net.dict
Run the prof.py script to generate the reports.
$ python -m pyprof.prof --csv net.dict
The User Guide can be found in the documentation for current release, and provides instructions on how to install and profile with PyProf.
A complete Quick Start Guide provides step-by-step instructions to get you quickly started using PyProf.
An FAQ provides answers for frequently asked questions.
The Release Notes indicate the required versions of the NVIDIA Driver and CUDA, and also describe which GPUs are supported by PyProf
Contributions to PyProf are more than welcome. To contribute make a pull request and follow the guidelines outlined in the Contributing document.
We appreciate any feedback, questions or bug reporting regarding this project. When help with code is needed, follow the process outlined in the Stack Overflow (https://stackoverflow.com/help/mcve) document. Ensure posted examples are:
- minimal – use as little code as possible that still produces the same problem
- complete – provide all parts needed to reproduce the problem. Check if you can strip external dependency and still show the problem. The less time we spend on reproducing problems the more time we have to fix it
- verifiable – test the code you're about to provide to make sure it reproduces the problem. Remove all other problems that are not related to your request/question.