/realistic_adversarial_hardening

This repository contains the data and the code used for S&P 2023 paper "On The Empirical Effectiveness of Unrealistic Adversarial Hardening Against Realistic Adversarial Attacks"

Primary LanguagePython

Realistic adversarial hardening

This repository contains the data and the code used for S&P 2023 paper On The Empirical Effectiveness of Unrealistic Adversarial Hardening Against Realistic Adversarial Attacks. The paper investigates whether "cheap" unrealistic adversarial attacks can be used to harden ML models against computationally expensive realistic attacks. The paper studies 3 use cases (text, botnet, malware) and at least one realistic and unrealistic attack for each of them.

Downloading the data

All the data used for the results analysis can be downloaded in this link. To download use-case specific data please continue reading below.

1. Text classification

We consider two attacks: Deepwordbug(unrealistic) and TextFooler(realistic).

We use Textattack library from the command line to train the models and evaluate their adversarial robustness. You will find all the models presented in the paper in this link. In the config files of each model you can see the training parameters, including the attack used for the hardening. This file provides an easy way to identify the models.

2. Botnet detection

As an unrealistic attack we use the PGD implementation of ART library.
For the realistic attack we use the FENCE attack by Chernikova et al. Our implementation in this repo includes updates from TensorFlow 1 to TensorFlow 2, code refactoring and as well as bug fixes. However you can access the original repository by authors here if needed.

If you need to do feature engineering for new datasets not included in this repo check the tool from Talha Ongun.

3. Malware detection

We use 3 attacks: PGD and MoEvA2 (unrealistic) and AIMED (realistic).

Again for PGD we use the ART implementation.
For MoEvA2 we use and include in this repo an early version called Coeva2. If you are interested, the latest release by the authors can be found here.
For AIMED we use and include in this repo a lighter version with only the necessary functionalities needed for this study. For the full code and functionalities check the authors original version in here

Accessing data for individual use cases

Follow the links to get general malware data, pgd_data, moeva2 data, AIMED data. You can download and place them in their respective folders.

We have received the original Windows PE files from the authors of When Malware is Packin’ Heat; Limits of Machine Learning Classifiers Based on Static Analysis Features.