PAT

Core implementation of Paper: "Order-Disorder: Imitation Adversarial Attacks for Black-box Neural Ranking Models".

Requirements

Python 3.8
Pytorch==1.10.0
transformers==4.6.1
sentence-transformers==2.1.0
apex==0.1
tqdm
nltk
pytrec_eval
trectools

Environment

Tesla V100 32GB GPU x 8
CUDA 11.2
Memory 256GB

Datasets

Model Imitation

Data Processing Build Dev data into pickle file to speedup the evaluation.
1. MSMARCO Passage Ranking Download MSMARCO Passage Ranking dataset. Download sub small dev set of MSMARCO, used for accerlating evaluation during training process
```
python ./bert_ranker/dataloader/preprocess_pr.py
```
1. TREC DL2019
```
python ./bert_ranker/dataloader/preprocess_trec_dl.py
```
1. TREC MB2014
```
python ./bert_ranker/dataloader/preprocess_mb.py
```
1. Natural Question
```
python ./bert_ranker/dataloader.preprocess_nq.py
```

Train Pairwise-BERT Ranker from scratch

python ./bert_ranker/run_pairwise_ranker.py

Get runs file (TREC Format) from the publicly available ranking model.
```
python ./bert_ranker/dev_public_bert_ranker.py
```
--mode determines which dataset to evaluate on --transformer_model determines which open source model to verify on. The default setting is "bert-large-uncased". There is an option, and we will also use the verification result later, which sets as "ms-marco-MiniLM-L-12-v2".
Sample training data from runs file of public model to train imitation model.
```
python ./bert_ranker/dataloader/sample_runs.py
```

Train imitation model using sampled data.

python ./bert_ranker/run_imitation_pairwise.py

Evaluate the similarity between imitation model and victim model using runs file.
```
python imitation_agreement.py
```
Evaluate ranking performance using runs file Note that the evaluation metrics during training and development are not consistent with the official evaluation method. We get the standard ranking performance by official trec tools, which are implemented in trec_eval_tools.py

Text ranking attack via PAT

The data preprocessing is implemented in ./adv_ir/data_utils.py. We need extract the query, query id, scores (imitation model), and target candidate passages from runs file.
The Pairwise Anchor-based Trigger generation is implemented in ./adv_ir/attack_methods.py function name: pairwise_anchor_trigger()

For generating adversarial triggers for ranking attack.

python pat_attack.py --target=mini --imitation_model=imitate.v2  --nsp --lamba_1=0.6 --lambda_2=0.1 --num_beams=10 --topk=128 --mode=train

Note that we adopted the fine-tuned BERT LM from Song et al.(2020)

Test the transferability of triggers

python pat_attack.py --target=mini --imitation_model=imitate.v2  --nsp --lamba_1=0.6 --lambda_2=0.1 --num_beams=10 --topk=128 --mode=test

Reference

@inproceedings{liu2022order,
  title={Order-Disorder: Imitation Adversarial Attacks for Black-box Neural Ranking Models},
  author={Liu, Jiawei and Kang, Yangyang and Tang, Di and Song, Kaisong and Sun, Changlong and Wang, Xiaofeng and Lu, Wei and Liu, Xiaozhong},
  booktitle={Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security},
  pages={2025--2039},
  year={2022}
}

trumpyla/PAT

PAT

Requirements

Environment

Datasets

Model Imitation

Text ranking attack via PAT

Reference