/Super-Fast-Adversarial-Training

Official PyTorch Implementation Code for Developing Super Fast Adversarial Training with Distributed Data Parallel, Channel Last Memory Format, Mixed Precision Training + Adversarial Attack, Faster Adversarial Training, and Fast Forward Computer Vision (FFCV).

Primary LanguagePythonMIT LicenseMIT

PyTorch Git

Super-Fast-Adversarial-Training

Generic badge Generic badge Generic badge License: MIT

This is an Official PyTorch Implementation code for developing super fast adversarial training. This code is combined with below state-of-the-art technologies for accelerating adversarial attacks and defenses with Deep Neural Networks on Volta GPU architecture.

  • Distributed Data Parallel [link]
  • Channel Last Memory Format [link]
  • Mixed Precision Training [link]
  • Mixed Precision + Adversarial Attack (based on torchattacks [link])
  • Faster Adversarial Training for Large Dataset [link]
  • Fast Forward Computer Vision (FFCV) [link]

Citation

If you find this work helpful, please cite it as:

@software{SuperFastAT_ByungKwanLee_2022,
  author = {Byung-Kwan Lee},
  title = {Super Fast Adversarial Training with Distributed Data Parallel and Mixed Precision},
  howpublished = {\url{https://github.com/ByungKwanLee/Super-Fast-Adversarial-Training}},
  year = {2022}
}

Library for Fast Adversarial Attacks

This library is developed based on the well-known package of torchattacks [link] due to its simple scalability.

Current Available Attacks Below

  • Fast Gradient Sign Method (FGSM)
  • Basic Iterative Method (BIM)
  • Projected Gradient Descent (PGD)
  • Momentum Iterative Method (MIM)
  • Carlini & Wagner (CW)
  • Fast Adaptive Boundary (FAB)
  • Auto-PGD (AP)
  • Difference of Logits Ratio (DLR)
  • Auto-Attack (AA)

Environment Setting

Please check below settings to successfully run this code. If not, follow step by step during filling the checklist in.

  • To utilize FFCV [link], you should install it on conda virtual environment. I use python version 3.8, pytorch 1.7.1, torchvision 0.8.2, and cuda 10.1. For more different version, you can refer to PyTorch official site [link].

conda create -y -n ffcv python=3.8 cupy pkg-config compilers libjpeg-turbo opencv pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.1 numba -c pytorch -c conda-forge

  • Activate the created environment by conda

conda activate ffcv

  • And, it would be better to install cudnn to more accelerate GPU. (Optional)

conda install cudnn -c conda-forge

  • To install FFCV, you should download it in pip and install torchattacks [link] to run adversarial attack.

pip install ffcv torchattacks==3.1.0

  • To guarantee the execution of this code, please additionally install library in requirements.txt (matplotlib, tqdm)

pip install -r requirements.txt

Available Datasets

Available Baseline Models

How to run

After making completion of environment settings, then you can follow how to run below.

  • First, run fast_dataset_converter.py to generate dataset with .betson extension, instead of using original dataset [FFCV].
# Future import build
from __future__ import print_function

# Import built-in module
import os
import argparse

# fetch args
parser = argparse.ArgumentParser()

# parameter
parser.add_argument('--dataset', default='imagenet', type=str)
parser.add_argument('--gpu', default='0', type=str)
args = parser.parse_args()

# GPU configurations
os.environ["CUDA_VISIBLE_DEVICES"]=args.gpu

# init fast dataloader
from utils.fast_data_utils import save_data_for_beton
save_data_for_beton(dataset=args.dataset)
  • Second, run fast_pretrain_standard.py(Standard Training) or fast_pretrain_adv.py (Adversarial Training)
# model parameter
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--dataset', default='imagenet', type=str)
parser.add_argument('--network', default='resnet', type=str)
parser.add_argument('--depth', default=50, type=int)
parser.add_argument('--gpu', default='0,1,2,3,4', type=str)

# learning parameter
parser.add_argument('--learning_rate', default=0.1, type=float)
parser.add_argument('--weight_decay', default=0.0002, type=float)
parser.add_argument('--batch_size', default=512, type=float)
parser.add_argument('--test_batch_size', default=128, type=float)
parser.add_argument('--epoch', default=100, type=int)

or

# model parameter
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--dataset', default='imagenet', type=str)
parser.add_argument('--network', default='resnet', type=str)
parser.add_argument('--depth', default=18, type=int)
parser.add_argument('--gpu', default='0,1,2,3,4', type=str)

# learning parameter
parser.add_argument('--learning_rate', default=0.1, type=float)
parser.add_argument('--weight_decay', default=0.0002, type=float)
parser.add_argument('--batch_size', default=1024, type=float)
parser.add_argument('--test_batch_size', default=512, type=float)
parser.add_argument('--epoch', default=60, type=int)

# attack parameter
parser.add_argument('--attack', default='pgd', type=str)
parser.add_argument('--eps', default=0.03, type=float)
parser.add_argument('--steps', default=10, type=int)

To-do

I have plans to make a variety of functions to be a standard framework for adversarial training.

  • Many Compatible Adversarial Attacks
  • Many Compatible Adversarial Defenses