This code for the paper: "Adjective Scale Probe: Can Language Models Encode Formal Semantics Information?", presented at AAAI 2023 (oral). See the paper, the corresponding slides and the appendix file.
Directory data contains the NLI-style samples used in the paper.
see detailed descriptions in data/readme.md
Directory training contains the code for fine-tuning pre-trained models on the MNLI or the our Adjective Scale Probe (ASP) dataset.
Training codes are forked from Transformers
Run bash run.sh to fine-tuning pre-trained models on the ASP.
Change the configuration of run.sh to reproduce other fine-tuning procedures.
Directory evaluation contains the code for evaluating the models on the ASP.
Run bash evaluation.sh to test the ASP models on the leaveout testing sets.
Run python NLI_ASP.py to test the MNLI models on the ASP.
Run python zs_ASP.py to test the zero-shot models on the ASP.
zs_ASP.py is forked from T-zero
Directory human contains the questions and results for the human experiment.
cloze.csv: Cloze-style questions for human annotations. We change the unit to United States customary units, since all annotators are American.
result.csv: Raw results of human.
Directory pkl: processed human results for most tests of the degree estimation task.
If you make use of the code in this repository, please cite the following papers:
@article{Liu_Xiang_Ding_2023,
title={Adjective Scale Probe: Can Language Models Encode Formal Semantics Information?},
author={Liu, Wei and Xiang, Ming and Ding, Nai},
journal={Proceedings of the AAAI Conference on Artificial Intelligence},
volume={37}, number={11}, pages={13282-13290},
url={https://ojs.aaai.org/index.php/AAAI/article/view/26559},
month={Jun.}, year={2023},
}