/CSAW-HackML-2020

Starting Point for the CSAW HackML 2020 competition is here: https://www.csaw.io/hackml

Primary LanguagePython

CSAW-HackML-2020

├── data
    └── clean_validation_data.h5 // this is clean data used to evaluate the BadNet and design the backdoor defense
├── models
    └── bd_net.h5
    └── bd_weights.h5
├── architecture.py
└── eval.py // this is the evaluation script

I. Dependencies

  1. Python 3.6.9
  2. Keras 2.3.1
  3. Numpy 1.16.3
  4. Matplotlib 2.2.2
  5. H5py 2.9.0
  6. TensorFlow-gpu 1.15.2

II. Validation Data

  1. Download the clean validation data from here and store the data in data/clean_validation_data.h5 directory.
  2. The dataset contains images from YouTube Aligned Face Dataset. We retrieve 883 individuals each containing 10 images in the validation dataset.

III. Evaluating the Backdoored Model

  1. The DNN architecture used to train the face recognition model is the state-of-the-art DeepID network. This DNN is backdoored with multiple triggers. Each trigger is associated with its own target label. Essentially, the provided BadNet is a Multi-Trigger Multi-Target setting.

  2. To evaluate the backdoored model, execute eval.py by running:
    python3 eval.py <clean validation data directory> <model directory>.

    E.g., python3 eval.py data/clean_validation_data.h5 models/bd_net.h5.

  3. Clean data classification accuracy on the provided validation dataset is 96.07 %.

IV. Evaluating the Submissions

To aid teams in designing their defense, here are a few guidelines to keep in mind to get maximum points for the submission:

  1. Defense should generalize well to other backdoored networks. To verify the defense generalizability, the organizers will evaluate the submission on a specially curated BadNet with different trigger properties.
  2. Teams gain maximum points if the defense greatly reduces attack success rate on the trigger(s) while maintaining high clean classification accuracy.
  3. Points will also be given to teams that identify poisoned images in the online test stream of images.
  4. Fewer points will be allocated to teams that only detect the network as clean or backdoored.
  5. Report should contain a description of the defense performance on adaptive attackers.