wielandbrendel/foolbox-native

Foolbox Native brings native performance to Foolbox

Foolbox Native

Foolbox Native is an extension for Foolbox that tries to bring native performance to Foolbox. This extension is a prototype with the goal of ultimately becoming part of Foolbox itself. Please be aware of the the differences to Foolbox listed below.

Foolbox Native currently provides full support for:

• PyTorch
• TensorFlow
• JAX

Other frameworks can be used as well by falling back to Foolbox for the model API, see below.

Installation

pip install --upgrade foolbox-native

PyTorch Example

import foolbox.ext.native as fbn
import torchvision.models as models

# instantiate a model
model = models.resnet18(pretrained=True).eval()
preprocessing = dict(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], axis=-3)
fmodel = fbn.models.PyTorchModel(model, bounds=(0, 1), preprocessing=preprocessing)

# get data and test the model
images, labels = fbn.utils.samples(fmodel, dataset='imagenet', batchsize=16)
print(fbn.utils.accuracy(fmodel, images, labels))
# -> 0.9375

# apply the attack
attack = fbn.attacks.LinfinityBasicIterativeAttack(fmodel)
adversarials = attack(images, labels, epsilon=0.03, step_size=0.005)  # L-inf norm
print(fbn.utils.accuracy(fmodel, adversarials, labels))
# -> 0.0

# apply another attack
attack = fbn.attacks.L2BasicIterativeAttack(fmodel)
adversarials = attack(images, labels, epsilon=2.0, step_size=0.2)  # L2 norm
print(fbn.utils.accuracy(fmodel, adversarials, labels))
# -> 0.0

TensorFlow Example

import foolbox.ext.native as fbn
import tensorflow as tf

# instantiate a model
model = tf.keras.applications.ResNet50(weights='imagenet')
preprocessing = dict(flip_axis=-1, mean=[104., 116., 123.])  # RGB to BGR
fmodel = fbn.models.TensorFlowModel(model, bounds=(0, 255), preprocessing=preprocessing)

# get data and test the model
images, labels = fbn.utils.samples(fmodel, dataset='imagenet', batchsize=16)
print(fbn.utils.accuracy(fmodel, images, labels))

# apply the attack
attack = fbn.attacks.LinfinityBasicIterativeAttack(fmodel)
adversarials = attack(images, labels, epsilon=0.03 * 255., step_size=0.005 * 255.)  # L-inf norm
print(fbn.utils.accuracy(fmodel, adversarials, labels))

# apply another attack
attack = fbn.attacks.L2BasicIterativeAttack(fmodel)
adversarials = attack(images, labels, epsilon=2.0 * 255., step_size=0.2 * 255.)  # L2 norm
print(fbn.utils.accuracy(fmodel, adversarials, labels))

Robust Accuracy Evaluation

import foolbox.ext.native as fbn

# get fmodel, images, labels like above
fmodel = ...
images, labels = ...

attacks = [
    L2BasicIterativeAttack,
    L2CarliniWagnerAttack,
    L2ContrastReductionAttack,
    BinarySearchContrastReductionAttack,
    LinearSearchContrastReductionAttack,
]
epsilons = [0.0, 1.0, 2.0, 4.0, 8.0, 16.0, 32.0, 64.0, 128.0]

_, robust_accuracy = fbn.evaluate_l2(fmodel, x, y, attacks=attacks, epsilons=epsilons)
print(robust_accuracy)

# Plot an accuracy-distortion curve
plt.plot(epsilons, robust_accuracy)

Other Frameworks

Foolbox Native supports all frameworks supported by the standard Foolbox by simply wrapping the foolbox.models.* classes using fbn.model.FoolboxModel. This, however, comes with a performance penalty. Nevertheless, it still allows one to profit from the manually batched attack reimplementations that come with Foolbox Native.

Important differences to Foolbox

Unlike Foolbox:

• Foolbox Native does not yet guarantee API stability (expect breaking changes)
• Foolbox Native is currently limited to very few attacks
• Foolbox Native does not make any guarantees about the output of an attack
  • The user is responsible for checking if the returned samples are adversarial
  • Whether the size of the perturbations is guaranteed depends on the attack
  • Foolbox, on the other hand, searches for the smallest perturbation while guaranteeing that the returned samples are adversarial