Don't Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling
Paper link: https://aclanthology.org/2022.coling-1.405/
To be published in Coling 2022
Our code is mainly based on the code of SimCSE. Please refer to their repository for more detailed information.
Requirements
- Python 3.8
Install other packages
pip install -r requirements.txt
Download the pretraining dataset
cd data
bash download_nli.sh
Download the downstream dataset
cd SentEval/data/downstream/
bash download_dataset.sh
Training
(Using Multi-GPU run_sup_layerattnpooler.sh
)
python train.py \
--model_name_or_path bert-base-uncased \
--train_file data/nli_for_simcse.csv \
--output_dir result/bert-base-uncased-cl-layerattnpooler \
--num_train_epochs 3 \
--per_device_train_batch_size 64 \
--learning_rate 2e-5 \
--max_seq_length 64 \
--evaluation_strategy steps \
--metric_for_best_model stsb_spearman \
--load_best_model_at_end \
--eval_steps 100 \
--pooler_type cls \
--overwrite_output_dir \
--temp 0.05 \
--do_train \
--do_eval \
--fp16 \
"$@"
Citations
Please cite our paper if they are helpful to your work!
@inproceedings{oh2022don,
title={Don’t Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling},
author={Oh, Dongsuk and Kim, Yejin and Lee, Hodong and Huang, H Howie and Lim, Heui-Seok},
booktitle={Proceedings of the 29th International Conference on Computational Linguistics},
pages={4585--4592},
year={2022}
}