PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models

This repo contains the PyTorch implementation of Rabeeh Karimi Mahabadi, Luke Zettlemoyer, james Henderson, Marzieh Saeidi, Lambert Mathias, Veselin ‪Stoyano, and Majid Yazdani PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models, ACL 2022.

For any questions, please contact the first author(email) or leave issues.

Installation

conda create --name perfect python=3.8
python setup.py develop 
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
pip install -r requirements.txt

data pre-processing

For SST-2, SST-5, CR, MR, Subj, TREC datasets, we used the datasets from Gao et al. ACL 2021 (paper), which are processed as below, for other datasets we used the huggingface datasets, which automatically get downloaded:

wget https://nlp.cs.princeton.edu/projects/lm-bff/datasets.tar
tar xvf datasets.tar
rm datasets.tar 
mv original/ datasets
python process_datasets.py
rm -r datasets

How to run the models

We provide the example scripts to run each model in the paper in fewshot/scripts folder with their config files in fewshot/configs. To run the models, please do cd fewshot and run: Please note on top of each script, I wrote how we modified the hyper-parameters

Reproducing results in table 1

Perfect results

bash scripts/perfect.sh

Finetune results:

bash scripts/finetune.sh

PET

bash scripts/pet.sh

Logan IV et al's results

bash scripts/loganIV.sh

Prompt+mte ablation

bash scripts/prompt_mte_ablation.sh

bitfit+mte ablation results:

bash scripts/bitfit_mte_ablation.sh

perfect+init ablation results:

scripts/perfect_init_ablation.sh

Reproducing results in table 3

Pattern-Free ablation results

bash scripts/pattern_free.sh

Reproducing results in table 4

bash scripts/perfect_without_adapters_ablation.sh

Reproducing results in table 5

bash scripts/perfect_num_masks_ablation.sh

Reproducing results in table 7

bash scripts/perfect_mask_position_ablation.sh

Reproducing results in table 8

bash scripts/perfect_init_range_ablation.sh

Reproducing results in table 9

Hinge loss ablation

bash scripts/perfect_hinge_loss_ablation.sh

+Label Embed ablation

bash scripts/perfect_label_embed_ablation.sh

-Prototypical ablation

bash scripts/perfect_prototypical_ablation.sh

Bibliography

If you find this repo useful, please cite our paper.

@inproceedings{karimi2022perfect,
  title={PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models},
  author={Karimi Mahabadi, Rabeeh and Zettlemoyer, Luke and Henderson, James and Saeidi, Marzieh and Mathias, Lambert and ‪Stoyano, Veselin and Yazdani, Majid},
  booktitle={Annual Meeting of the Association for Computational Linguistics},
  year={2022}
}

License

The code in this repository is released under the Apache 2.0 license

probe2/perfect