/EMPIR

Source code for the paper EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness against Adversarial Attacks

Primary LanguagePythonMIT LicenseMIT

EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness against Adversarial Attacks

Build Status

This repository contains the source code for the paper EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness against Adversarial Attacks (Accepted at ICLR 2020)

It is based on CleverHans 1.0.0, a Python library to benchmark machine learning systems' vulnerability to adversarial examples. You can learn more about such vulnerabilities on the accompanying blog.

Setting up

  • Install TensorFlow
  • Install Keras
  • Git clone this repository
  • For ImageNet results, download ImageNet dataset and convert the data into TFRecords using this script.

We tested this setup using tensorflow-gpu 1.10, keras 2.2.4, python 3.5, CUDA 9.2 and Ubuntu 18.04 on a single RTX 2080 Ti GPU. Tensorflow was installed using anaconda.

Example commands

  • python examples/mnist_attack.py --wbits=$model1_weight_prec --abits=$model1_activation_prec --wbits2=$model2_weight_prec --abits2=$model2_activation_prec --ensembleThree --model_path1=/path/to/model1/ckpt --model_path2=/path/to/model2/ckpt --model_path3=/path/to/model3/ckpt - White-Box CW attack on MNISTconv EMPIR model
  • python examples/mnist_attack.py --model_path=/path/to/baseline/model/ckpt - White-Box CW attack on MNISTconv baseline model
  • python examples/cifar10_attack.py --abits=$model1_activation_prec --wbits=$model1_weight_prec --abits2=$model2_activation_prec --wbits2=$model2_weight_prec --model_path1=/path/to/model1/ckpt --model_path2=/path/to/model2/ckpt --model_path3=/path/to/model3/ckpt --ensembleThree - White-Box CW attack on CIFARconv EMPIR model
  • python examples/cifar10_attack.py --model_path=/path/to/baseline/model/ckpt - White-Box CW attack on CIFARconv baseline model
  • python examples/alexnet_attack.py --batch_size=100 --imagenet_path=/path/to/imagenet/tf_records --ensembleThree --abits=$model1_activation_prec --wbits=$model1_weight_prec --abits2=$model2_activation_prec --wbits2=$model2_weight_prec --model_path1=/path/to/model1/ckpt --model_path2=/path/to/model2/ckpt --model_path3=/path/to/model3/ckpt - White-Box CW attack on AlexNet EMPIR model
  • python examples/alexnet_attack.py --batch_size=100 --imagenet_path=/path/to/imagenet/tf_records --model_path=/path/to/baseline/model/ckpt - White-Box CW attack on AlexNet baseline model

Results

  • EMPIR models
Dataset Precisions Unperturbed Accuracy (%) Adversarial Accuracy (%)
Model 1 Model 2 Model 3 CW FGSM BIM PGD Average
MNIST abits=4, wbits=2 Download abits=4, wbits=2 Download Full-precision (32 bits) Download 98.89 86.73 67.06 18.61 17.51 47.48
CIFAR-10 abits=2, wbits=4 Download abits=2, wbits=2 Download Full-precision (32 bits) Download 72.56 48.51 20.45 24.59 13.55 26.78
ImageNet abits=2, wbits=2 Download abits=4, wbits=4 Download Full-precision (32 bits) Download 55.09 29.36 21.65 20.67 11.76 20.86
  • Baseline models
Dataset Models Unperturbed Accuracy (%) Adversarial Accuracy (%)
CW FGSM BIM PGD Average
MNIST MNISTconv Download 98.87 3.69 14.32 0.9 0.77 4.92
CIFAR-10 CIFARconv Download 74.54 13.38 10.28 11.97 10.69 11.58
ImageNet AlexNet Download 53.23 9.94 10.29 10.81 10.30 10.34

Citing this work

@inproceedings{
sen2020empir,
title={{\{}EMPIR{\}}: Ensembles of Mixed Precision Deep Networks for Increased Robustness Against Adversarial Attacks},
author={Sanchari Sen and Balaraman Ravindran and Anand Raghunathan},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=HJem3yHKwH}
}

Copyright

Copyright 2017 - Google Inc., OpenAI and Pennsylvania State University.