Source code and dataset for Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models.
In this work we propose AutoSeq, a general-purpose, fully automatic prompting method that depends only on few-shot annotations.
Figure: Illustration of AutoSeq. We first use T5 to generate label sequence candidates given each label's training instances; we then use contrastive re-ranking to get label sequences that are more label-specific; in the end we enumerate all the combinations and re-rank by the fine-tuning performance.
To run our code, please install all the dependencies:
pip install -r requirements.txt
Please download the official GLUE and SuperGLUE data and extract the files to ./data/GLUE/TASK_NAME and ./data/SuperGLUE/TASK_NAME.
Then use the following command to generate the few-shot data:
python tools/generate_k_shot_data_for_GLUE.py
python tools/generate_k_shot_data_for_SuperGLUE.pyTo reproduce our results, please use the scripts we provide. For convenience, we use SST-2 as an example below.
AutoSeq:
task=SST-2 bash scripts/t5_seq.shAutoWord:
task=SST-2 bash scripts/t5_word.shManual:
task=SST-2 bash scripts/t5_man.shManual with eng.:
task=SST-2 bash scripts/t5_man_eng.shFine-tuning:
task=SST-2 bash scripts/t5_fine_tuning.shFine-tuning (Full train set):
task=SST-2 bash scripts/t5_full_fine_tuning.shPrompt tuning:
task=SST-2 bash scripts/t5_prompt_tuning.shAll the results will be stored in ./log. To gather all the results, run the following command:
python tools/gather_result.py --condition "{'tag': 'exp-autoseq-SST-2', 'task_name': 'sst-2', 'few_shot_type': 'prompt'}"We also release the original log of our experiments to facilitate the reproduction.
You can find our generated mappings in ./auto_label_sequences and ./auto_label_words.
We first generate candidate label sequence and label word mappings by running:
python tools/generate_label_sequences.py
python tools/generate_label_words.pyThe generated candidates will be saved in ./my_auto_label_sequences and ./my_auto_label_words.
Then we do prompt-based fine-tuning of all the mappings by:
task=SST-2 bash scripts/sort_mapping.shFinally we sort them based on dev set performance:
python tools/sort_mapping.py --condition "{'tag': 'exp-mapping-SST-2', 'task_name': 'sst-2'}" --mapping_dir auto_label_sequencesPlease cite our paper if you use AutoSeq in your work:
@inproceedings{yu2022automatic,
title={Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models},
author={Zichun Yu and Tianyu Gao and Zhengyan Zhang and Yankai Lin and Zhiyuan Liu and Maosong Sun and Jie Zhou},
booktitle={International Conference on Computational Linguistics (COLING)},
year={2022}
}