/Order_As_Supervision

NAACL 2024 Paper on how using order as supervision can benefit procedural reasoning

Primary LanguagePython

Code for the NAACL 2024 Paper - 'Order-Based Pre-training Strategies for Procedural Text Understanding'

image

Required dependencies

Please run pip install -r requirements.txt.

Hyperparameters for commands

  • CONTEXT_STEPS: Number of steps to be used as the context.
  • EMBEDDING_TYPE: Type of embedding ('hamming', 'kenny', 'lehmer').
  • MODEL_NAME: Name of the base transformer model ('bert', 'roberta') or a HuggingFace model which has the transformer model name as a substring ('contra-4-2_bert').
  • MODEL_PATH: Path to the base transformer model ('bert-base-uncased', 'roberta-base').
  • NUM_RANDOM_STEPS: Number of random steps to be sampled.
  • RECIPES_STEPS: Number of steps of recipes, or text about any process, in the training data.
  • TEST: Set 'True' to test code on your local machine (True, False).
  • TOTAL_NUM_PERMUTATIONS: Size of the permutation set.

Before pre-training store your training data as a list of recipes with each recipe in turn as a list of steps in a pickle file called full_corpus.pickle. Running the pretraining code for Permutation Classification or Embedding Regression should generate permutation_hamming_<RECIPES_STEPS>_<TOTAL_NUM_PERMUTATIONS>.pickle file which contains TOTAL_NUM_PERMUTATIONS permutations of length RECIPES_STEPS.

Pretraining using Permutation Classification

Run python3 classification.py <MODEL_NAME> <MODEL_PATH> <RECIPES_STEPS> <TEST> <TOTAL_NUM_PERMUTATIONS>. Running the command should generate a folder models/pc_<RECIPES_STEPS>_<TOTAL_NUM_PERMUTATIONS>__<MODEL_NAME> containing model and log files. Link to pretrained model pc_6_100__roberta.

Pretraining using Embedding Regression

Run python3 embedding.py <EMBEDDING_TYPE> <MODEL_NAME> <MODEL_PATH> <RECIPES_STEPS> <TEST> <TOTAL_NUM_PERMUTATIONS>. Running the command should generate a folder models/embeddings_<EMBEDDING_TYPE>_<RECIPES_STEPS>_<TOTAL_NUM_PERMUTATIONS>__<MODEL_NAME> containing model and log files.

Pretraining using Skip-Clip

Run python3 skipclip.py <CONTEXT_STEPS> <MODEL_NAME> <MODEL_PATH> <NUM_RANDOM_STEPS> <TEST>. Running the command should generate a folder models/skipclip_<CONTEXT_STEPS>_<NUM_RANDOM_STEPS>__<MODEL_NAME> containing model and log files.

Fine-tuning on SQuAD 2.0

  • Pretrained models need to be finetuned on SQuAD 2.0 dataset before training on the entity-tracking downstream task.
  • To download the training set, run wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json.
  • Run python3 finetune_squad.py <MODEL_TYPE> <MODEL_PATH>.
  • To get the models fine-tuned on SQuAD 2.0, follow the following format to get the link - https://huggingface.co/AnonymousSub/<SUBSTRING AFTER THE LAST '/' IN PRE-TRAINED MODEL LINK>_squad2.0.

Links to Pre-trained models Fine-tuned on SQuAD 2.0

  • Permutation Classification
  1. pc-6-50-roberta_squad2.0
  2. pc-6-200-roberta_squad2.0
  • Embedding Regression
  1. embeddings-lehmer-6-50-roberta_squad2.0
  2. embeddings-hamming-6-50-roberta_squad2.0
  3. embeddings-hamming-6-100-roberta_squad2.0
  • Skip-Clip
  1. skipclip-4-4-roberta_squad2.0