/nnsmith

Automatic DNN generation for fuzzing and more.

Primary LanguagePythonApache License 2.0Apache-2.0

Support table

Model\Engine TVM ORT TensorRT TFLite XLA Torch-JIT
ONNX ✅ ✅ ✅
TensorFlow 🔨 ⚠️ ⚠️
PyTorch 🔨 🔨 🔨

✅: Supported; ⚠️: Experimental support; 🔨: Coming soon;

Setup

Install latest stable release:

pip install "nnsmith[torch,onnx]" --upgrade
Install GitHub HEAD: [click to expand] 
pip install "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade
# or pip install "git+ssh://git@github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade
Install latest pre-release: [click to expand] 
pip install "nnsmith[torch,onnx]"                     \
            --pre --upgrade                           \
            --index-url https://test.pypi.org/simple/ \
            --extra-index-url https://pypi.org/simple/

Quick Start

Setting up graphviz for debugging [click to expand]

Graphviz provides dot for visualizing graphs in nice pictures. But it needs to be installed via the following methods:

sudo apt-get install graphviz graphviz-dev      # Linux
brew install graphviz                           # MacOS
conda install --channel conda-forge pygraphviz  # Conda
choco install graphviz                          # Windows

pip install pygraphviz  # Final step.

Also see pygraphviz install guidance.

# Generate a random model in "nnsmith_outputs/*"
nnsmith.model_gen model.type=onnx debug.viz=true

See other commands under doc/cli. We use hydra to manage configurations. See nnsmith/config/main.yaml.

Developer Notes

  • pip install -r requirements/core.txt to run generation and fuzzing;
  • pip install --upgrade --pre -r requirements/sys/[system].txt to allow generating and running specific frameworks;
    • Why "--upgrade --pre"? In fact, all the sources under requirements/sys/ are nightly release (except tvm) as we want to "save the world" by catching new bugs;
Pre-commits [click to expand]

You can use pre-commit to simpify development:

  • pip install -r requirements/dev.txt;
  • pre-commit install;
  • pre-commit will run upon a commit; To explicitly run pre-commit for all files: pre-commit run --all-files.
Local development [click to expand]
  • Develop locally by setting export PYTHONPATH=$PYTHONPATH:$(pwd) (pwd should be this git folder.)
  • Set PYTHONPATH="" when doing pip install nnsmith from online version.
Simplify the code [click to expand]

Simplicity is prerequisite for reliability. --Edsger W. Dijkstra

We want code simplicity: keeping minimal dependencies and focusing on a small set of simple APIs to make NNSmith maintainable to developers and reliable to users.

Test before commit [click to expand] 
# env of torch & tf will conflict so split their unit tests.
pytest tests/core -s
pytest tests/torch -s
pytest tests/tensorflow -s

Notes

  • NNSmith is modularized and can be extended as a 3rd-party library, which allows you to patch your own backend and do fuzzing without modifying NNSmith's source code.
  • Meanwhile, feel free to request a backend support: the project maintainer is happy to support DL systems that care about software reliability and quality to benefit the whole DL software stack.
  • It would be great if you can let us know if you find new bugs with NNSmith or build a new system inspired by NNSmith.

Paper

ASPLOS'23 | NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers. [click to expand citation] 
@article{liu2022finding,
  title={Finding Deep-Learning Compilation Bugs with NNSmith},
  author={Liu, Jiawei and Lin, Jinkun and Ruffy, Fabian and Tan, Cheng and Li, Jinyang and Panda, Aurojit and Zhang, Lingming},
  journal={arXiv preprint arXiv:2207.13066},
  year={2022}
}