Install all dependencies using conda
:
conda env create -f environment.yml
conda activate lang-patching
pip install -e .
Note that this repository uses hydra
for managing hyperparameters and experiments. The configs we use for training can be found under config
. hydra
creates unique outputs for every experiment under the directory output
.
To start, create synthetic data for patch finetuning using the yaml format (some examples are in the PATCH_DIR
folder), and then use convert_yaml_to_data.py
to create json files. The JSON files used in our experiments can be found in the PATCH_DIR
folder.
The entry script for training patchable models is train_models.py
. Run it as:
python train_models.py train.save_path={SAVE_PATH} +protocol={protocol} +patch_type={SUB_FOLDER} +multitask_sst=True +train.load_path={TASK_FINETUNED_MODEL} +learnt_interpreter={True/False}
- {SAVE_PATH}: path where the patchable model will be saved
- {protocol}: can be one of
simple
: If you want to train a model on just the task ("Task Finetuning")patch_finetuning_conds
: train a patchable model for Sentiment Classificationpatch_re
: to train a patchable model for Relation Extraction
- {SUB_FOLDER}: one of the folders in the PATCH_DIR directory. To train models with override patches, use
override_patch_data
and to train a model with feature based patches, usefeature_based_patch_data
. - learnt_interpreter: set this to
True
to train feature based patches.
Checkpoints for models used in this work can be found at this link. We also provide notebooks to reproduce various Tables in the paper. To reproduce results:
- For Table-2, see the notebooks with
checklist
in the name - For Table-3,4,5 please follow the instructions in the notebook
override_patches_sentiment.ipynb
andorig_model_results.ipynb
- For Figure-4, use
finetuning_experiments.py
To cite this paper, use:
@inproceedings{murty2022patches,
title = "Fixing Model Bugs with Natural Language Patches",
author = "Murty, Shikhar and
Manning, Christopher and
Lundberg, Scott and
Ribeiro, Marco Tulio",
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
year = "2022",
}