This repository contains code to evaluate attacks against CNN-based and LBP-based models [1], as well as scripts to evaluate two defense mechanisms for CNN training (Madry defense [2] and Ensemble adversarial Training [3]).
[1] Hafemann, Luiz G., Robert Sabourin, and Luiz S. Oliveira. "Characterizing and evaluating adversarial examples for Offline Handwritten Signature Verification" (preprint)
[2] Madry, A., Makelov, A., Schmidt, L., Tsipras, D. and Vladu, A., 2017. Towards deep learning models resistant to adversarial attacks. preprint
[3] Tramèr, F., Kurakin, A., Papernot, N., Goodfellow, I., Boneh, D. and McDaniel, P., 2017. Ensemble adversarial training: Attacks and defenses. preprint
First install the package sigver as follows:
pip install git+https://github.com/luizgh/sigver.git --process-dependency-linksDownload (or clone) this repository and install its requirements:
pip install requirements.txt
Download this repository: https://github.com/carlini/nn_robust_attacks. Add the path where this is downloaded to the PYTHONPATH environment variable.
For the LBP experiments, you need to have Matlab installed, and install the python/matlab integration:
https://www.mathworks.com/help/matlab/matlab_external/install-the-matlab-engine-for-python.html
An interactive example can be found in the example.ipynb file, that can also be viewed in this link. This example considers the following tasks:
- Extracting features and training a WD classifier for a user
- Perform a type-I attack (change a genuine signature so that it is rejected)
- Perform a type-II attack (change a skilled forgery so that it is accepted)
To reproduce all steps in the paper you need access to the four datasets (GPDS, MCYT, CEDAR, Brazilian PUC-PR). To reproduce the result for a single dataset, you need access to this dataset, and the models trained on GPDS (see the trained models section). Please note that I cannot share the datasets, so please directly contact the groups that proposed the datasets to have access to them (MCYT and CEDAR have public access).
To reproduce the paper, the following steps are necessary:
- Train the CNNs on GPDS:
scripts/train_cnns.sh(or use the trained models below) - Process the datasets - extract lbp and signet features:
scripts/process_datasets.sh - Train classifiers and select the images to be attacked (only images that are correctly classified by all models):
scripts/choose_files.sh - Run LBP attacks:
scripts/lbp_attacks.sh - Run CNN attacks:
scripts/cnn_attacks.sh
The scripts scripts/process_results.sh and scripts/process_results_[defense/otsu].py process the results and generate latex tables
Below are the trained CNN models that were used in this paper:
- SigNet (from [4])
- SigNet 350-615: trained with GPDS users [350, 615)
- SigNet Madry 350-615: trained with GPDS users [350, 615) using the Madry defense [2]
- SigNet Ens Adv 350-615: trained with GPDS users [350, 615) using Ensemble Adversarial Training [3]
- SigNet 615-881: trained with GPDS users [615, 881)
[4] Hafemann, Luiz G., Robert Sabourin, and Luiz S. Oliveira. "Learning Features for Offline Handwritten Signature Verification using Deep Convolutional Neural Networks" http://dx.doi.org/10.1016/j.patcog.2017.05.012 (preprint)
If you use our code, please consider citing the following papers:
Hafemann, Luiz G., Robert Sabourin, and Luiz S. Oliveira. "Learning Features for Offline Handwritten Signature Verification using Deep Convolutional Neural Networks" http://dx.doi.org/10.1016/j.patcog.2017.05.012 (preprint)
Hafemann, Luiz G., Robert Sabourin, and Luiz S. Oliveira. "Characterizing and evaluating adversarial examples for Offline Handwritten Signature Verification" (preprint)
The source code on the project root and the "scripts" folder is released under the BSD 3-clause license. The code under "clbp" is copyrighted by the authors Guo, Zhenhua, Lei Zhang, and David Zhang [5]. Note that the trained models used the GPDS dataset for training, which is restricted for non-commercial use.
[5] Guo, Zhenhua, Lei Zhang, and David Zhang. "A completed modeling of local binary pattern operator for texture classification." IEEE Transactions on Image Processing 19.6 (2010): 1657-1663.