/Verifiable-Coherent-NLU

Shared repository for TRIP dataset for verifiable NLU and coherence measurement for text classifiers.

Primary LanguageJupyter Notebook

Verifiable-Coherent-NLU

Shared repository for TRIP dataset for verifiable NLU and coherence measurement for text classifiers. Covers the following upcoming publications in Findings of EMNLP 2021:

  1. Shane Storks, Qiaozi Gao, Yichi Zhang, and Joyce Chai. (2021). Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding. In Findings of EMNLP 2021.
  2. Shane Storks and Joyce Chai. (2021). Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers. In Findings of EMNLP 2021.

Please contact Shane Storks with any questions.

Getting Started

Our results can be reproduced using the Python notebook file Verifiable-Coherent-NLU.ipynb, which we ran in Colab (may require some adaptation for use in Jupyter).

Clone the repo:

git clone https://github.com/sled-group/Verifiable-Coherent-NLU.git

You will then need to upload the contents of this folder to Google Drive.

From the Verifiable-Coherent-NLU directory in your Google Drive, open Verifiable-Coherent-NLU.ipynb using Google Colab.

Reproducing Results

Configure the cells below the first heading of Verifiable-Coherent-NLU.ipynb as needed and run the Setup block to prepare the notebook for reproducing a specific set of results. Then navigate to the appropriate block to reproduce results on TRIP, Conversational Entailment, or ART. Each block will have sub-blocks for preparing the data (run every time), and for training and testing models.

You may either re-train the models from the papers, or use our pre-trained model instances (see below).

Pre-Trained Model Instances

Pre-trained model instances from the papers are available here. Each sub-directory indicates a model and (if applicable) a loss function configuration, while the archive files within are for each type of LM trained, e.g., BERT, RoBERTa, or DeBERTa.

Copy the desired archive file(s) within these directories to your own Google Drive, and unzip them into a new directory ./saved_models. Run inference on them as needed using the appropriate blocks in the notebook. The names of the provided pre-trained model directories are already listed in the configuration area for convenience.

Cite

If you use our code or models in your work, please cite one of our following papers from Findings of EMNLP 2021:

  @misc{storks2021tiered,
        title={Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding}, 
        author={Shane Storks and Qiaozi Gao and Yichi Zhang and Joyce Chai},
        year={2021},
        booktitle={Findings of the Association for Computational Linguistics: EMNLP 2021},
        location={Punta Cana, Dominican Republic},
        publisher={Association for Computational Linguistics},
  }
  @misc{storks2021tip,
        title={Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers}, 
        author={Shane Storks and Joyce Chai},
        year={2021},
        booktitle={Findings of the Association for Computational Linguistics: EMNLP 2021},
        location={Punta Cana, Dominican Republic},
        publisher={Association for Computational Linguistics},
  }

Additionally, please consider citing Conversational Entailment and ART, which are used in experiments from the latter paper (and included in this repo):

  @inproceedings{zhang-chai-2010-towards,
      title = "Towards Conversation Entailment: An Empirical Investigation",
      author = "Zhang, Chen  and
        Chai, Joyce",
      booktitle = "Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing",
      month = oct,
      year = "2010",
      address = "Cambridge, MA",
      publisher = "Association for Computational Linguistics",
      url = "https://aclanthology.org/D10-1074",
      pages = "756--766",
  }
  @inproceedings{
      bhagavatula2020abductive,
      title={Abductive Commonsense Reasoning},
      author={Chandra Bhagavatula and Ronan Le Bras and Chaitanya Malaviya and Keisuke Sakaguchi and Ari Holtzman and Hannah Rashkin and Doug Downey and Wen-tau Yih and Yejin Choi},
      booktitle={International Conference on Learning Representations},
      year={2020},
      url={https://openreview.net/forum?id=Byg1v1HKDB}
  }