RelaxLoss

This repository contains the implementation for "RelaxLoss: Defending Membership Inference Attacks without Losing Utility" (ICLR 2022).

Contact: Dingfan Chen (dingfan.chen@cispa.de)

Requirements

This implementation is based on PyTorch (tested for version 1.7.0). Please refer to requirements.txt for the other required packages and version.

Datasets

The implementation supports the following datasets:

CIFAR10, CIFAR100 (corresponding files are in folder "source/cifar")
Texas, Purchase (corresponding files are in folder "source/nonimage")

The datasets will be automatically downloaded to folder "data" once you run the program.

Running Experiments

API (Run experiments using the default configurations).

cd source
python main.py \
--dataset "Dataset name" \
--method "Defense method" \
--mode "Experiment mode" \
[--model "Model architecture"(used only for CIFAR datasets)] \

Set the "Defense methods" to be 'relaxloss' to run our method.
The following defense methods are supported: 'vanilla', 'relaxloss', 'advreg', 'dpsgd', 'confidence_penalty', 'distillation', 'dropout', 'early_stopping', 'label_smoothing'. And 'vanilla' corresponds to no defense baseline.
The following "Experiment modes" are supported:
- 'defense': run the defense method to train a target model.
- 'attack': evaluate the target model under different MIAs (it requires to run defense first).
- 'shadow': run the defense method to train a shadow model.
- 'defense_attack': run the defense method to train a target model and directly evaluate the trained target model under different MIAs.
For CIFAR datasets, default configurations support 'resnet20' and 'vgg11_bn' models.

Run defense with specific configurations.

Go into folder source/cifar/defense or source/nonimage/defense depends on the dataset.
You can run each defense with command
```
python 'Defense_name'.py \
-name "Experiment name (output folder name)" \
--dataset "Dataset name" \
[--model "Model architecture"(for CIFAR datasets)] \ 
```
and specify the arguments in command-lines. Please refer to the parse_arguments() function in each python file for more details.

Run attacks.

Go into folder source/cifar or source/nonimage depends on the dataset.

Run attacks by providing the directory that contains the 'model.pt' file.

python run_attacks.py -target "Target_MODELDIR" -shadow "Shadow_MODELDIR"

In the current implementation, the attack ACC(accuracy) and AUC(area under the ROC curve) are evaluated.
The following attacks are supported. Please refer to 'source\utils\attack.py for more details.
- 'NN','Confidence','Loss','Entropy','Modified-Entorpy (Black-box)
- 'grad_wrt_x_l1', 'grad_wrt_x_l2', 'grad_wrt_w_l1', 'grad_wrt_w_l2' (White-box)

Citation

@inproceedings{chen2022relaxloss,
  title={RelaxLoss: Defending Membership Inference Attacks without Losing Utility},
  author={Chen, Dingfan and Yu, Ning and Fritz, Mario},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2022}
}

Acknowledgements

Our implementation uses the source code from the following repositories:

DingfanChen/RelaxLoss