datasec-lab/CertifiedAttack

Certifiable Black-box Attack

This is the official implementation of the paper "Certifiable Black-Box Attacks with Randomized Adversarial Examples: Breaking Defenses with Provable Confidence".

This project includes the ready-to-run models/attacks/defenses and the corresponding tutorials:

• Certifiable Attack

• Empirical Attacks:

  • 14 black-box attacks from BlackboxBench repository: NES, ZOSignSGD, Bandit-prior, ECO attack, SimBA, SignHunter, Sqaure attack, Boundary attack, OPT attack, Sign-OPT, GeoDA, HSJA, Sign Flip, RayS.

  • 2 additional unrestricted black-box attacks: PointWise, SparseEvo

• Defense methods:

  • Blacklight Black-box Attack Detection

  • Randomized Pre-processing Defense

  • Randomized Post-processing Defense

  • TRADES Adversarial Training

• Image classification models and datasets from this repository:

  • 6 Datasets: MNIST, FashionMNIST, KMNIST, CIFAR10, CIFAR100, ImageNet

  • 9 Models: VGG, ResNet, ResNet-preact, WRN, DenseNet, PyramidNet, ResNeXt, shake-shake, SENet

To set up the environment for this program, follow these steps:

1. Install Anaconda & Pytorch

   conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
   

2. Install required packages:

   pip install fvcore thop tensorboard 
   

3. For Certified Attacks

   pip install matplotlib scipy==1.11 statsmodels einops transformers accelerate pytorch_fid ema-pytorch torchviz
   

Run the code

Note: All the experimental settings used in the paper are stored in './configs', you can also customize your config files in './configs' to meet your requirements. The default configuration for all the attacks/defenses and models are stored in './pytorch_image_classification/config/defaults.py'.

Example 1: Run the Certifiable Attack against VGG model on CIFAR10

python attack.py --config "./configs/attack/cifar10/untargeted/unrestricted/vgg_CertifiedAttack.yaml" device cuda:0

Example 2: Run the Certifiable Attack against ResNet model on CIFAR100 under Blacklight

python attack.py --config "./configs/attack/cifar100_blacklight/untargeted/unrestricted/resnet_CertifiedAttack.yaml" device cuda:0

Example 3: Run the Certifiable Attack against ResNet model on ImageNet under RAND Pre-processing defense

python attack.py --config "./configs/attack/imagenet_RAND/untargeted/unrestricted/resnet_CertifiedAttack.yaml" device cuda:0

Example 4: Run the Certifiable Attack against WRN model on CIFAR10 under RAND Post-processing defense

python attack.py --config "./configs/attack/cifar10_post_RAND/untargeted/unrestricted/wrn_CertifiedAttack.yaml" device cuda:0

Example 5: Run the Certified Attack against Adversarial Trained VGG model on CIFAR10

python attack.py --config "./configs/attack/cifar10_AT/untargeted/unrestricted/resnet_CertifiedAttack_l2.yaml" device cuda:0

Example 6: Train the ResNext model on CIFAR10

python train.py --config "./configs/cifar10/resnext.yaml"

Example 7: Evaluate the trained VGG model on CIFAR10

python evaluate.py --config "./configs/evaluate/vgg.yaml"

Example 8: Train the ResNet model on CIFAR100 using Adversarial Training

python train.py --config "./configs/AT/cifar100/resnet_linf.yaml" device "cuda:2"