/CAST_public

Codes for Inferring the Reader: Guiding Automated Story Generation with Commonsense Reasoning

Primary LanguagePython

Pls contact me if you need processed data and model! xpeng62 at gatech dot edu.

CAST

This code accompanies the paper Inferring the Reader: Guiding Automated Story Generation with Commonsense Reasoning made-with-python

drawing

COMeT Filtering on GPT-2

alt text

🏗️ Installation

  • There are some extra packages you need to install here.
conda install -c pytorch pytorch
pip install ftfy

  • First, download the pretrained models from here.

  • Then untar the file:

tar -xvzf pretrained_models.tar.gz
  • Then run the following script to interactively generate arbitrary ATOMIC event effects:
python scripts/interactive/atomic_single_example.py --model_file pretrained_models/atomic_pretrained_model.pickle

How to use COMeT2020?

Relations Human Readable Template
AtLocation located or found at/in/on
CapableOf is/are capable of
Causes causes
CausesDesire makes someone want
CreatedBy is created by
Desires desires
HasA has, possesses or contains
HasFirstSubevent BEGINS with the event/action
HasLastSubevent ENDS with the event/action
HasPrerequisite to do this, one requires
HasProperty can be characterized by being/having
HasSubEvent includes the event/action
HinderedBy can be hindered by
InstanceOf is an example/instance of
isAfter happens after
isBefore happens before
isFilledBy blank can be filled by
MadeOf is made of
MadeUpOf made (up) of
MotivatedByGoal is a step towards accomplishing the goal
NotDesires do(es) NOT desire
ObjectUse, UsedFor used for
oEffect as a result, Y or others will
oReact as a result, Y or others feels
oWant as a result, Y or others want
PartOf is a part of
ReceivesAction can receive or be affected by the action
xAttr X is seen as
xEffect as a result, PersonX will
xIntent because PersonX wanted
xNeed but before, PersonX needed
xReact as a result, PersonX feels
xReason because
xWant as a result, PersonX wants

How to use GPT-2?

  • Run the code to use gpt-2
python ./examples/text-generation/run_generation.py --model_type=gpt2 --length=20 --model_name_or_path=gpt2 --num_return_sequences 5

How to use them together to do COMeT filtering with GPT-2?

  • All the related file is in /cometFilter/ folder.

  • Remember to add path in comet-commonsense/src/interactive/functions.py in your machine

  • Run the baseline of COMeT filtering: No filtering, two characters' interaction story

  • Run codes with new decoder, new prompt system, single char. Add -t to use two chars. Remove '-d' to remove decoder.

    • --model_name_or_path: str; the language model path.
    • --prompt_file: str; the first story sentence file. Each line is one prompt. txt file.
    • --comet_use: store_true; whether to use our technique. Remove it to use baseline.
    • -f: str; filter level. We use strong_new in inlg paper.
    • --coref_use: store_true; use coreference resoluation to remove the 3rd char
    • --use_cond: store_true; use new prompt style. EX. * [Char_1] * [Char_1] loves ice cream.
    • -l: int; story length. We use 5 or 10.
    • -d: store_true; new decoding system.
    • --history_use: store_true; use history when generate story using LM.
    • --diverse:
python -W ignore cometFilter.py --model_name_or_path finetune_language_model/finetuned_model/roc_char/21 --prompt_file data_story/test.txt --comet_use -f strong_new --use_cond -l 5 -d --history_use --diverse
  • Run codes with new decoder, old prompt system. Add -t to use two chars.
python -W ignore cometFilter.py --model_name_or_path finetune_language_model/finetuned_model/roc_pure --prompt_file data_story/test.txt --comet_use -f strong_new -l 5 -d --history_use --diverse
  • Run baseline to compare with.
python -W ignore cometFilter.py --model_name_or_path finetune_language_model/finetuned_model/roc_char/21 --prompt_file data_story/test.txt --use_cond -l 5 --history_use --device_id 1

👩‍💻 Matching Criteria

  • Decide matching criteria pairs by the following code:
 python comet_2020_use.py --check

--check: print all the matching criteria pairs' max score

  • Decide criteria matching criteria with files.
 python comet_2020_use.py --file_check

--file_check: summarize all the possible matching criteria pairs in one file.

--story_file_path: file path of story you use for decide matching criteria.

--save_file_path: file path of story you use for saving matching criteria.

  • Verify matching criteria by the following code:
 python comet_2020_use.py --verify --char 2

--verify: verify the matching criteria pairs

--char: int; number of characters

Single Character 👩 👨

Prompt Continuation
xWant xIntent
sentence xNeed
xEffect sentence
CausesDesire Desires
isBefore isAfter
AtLocation AtLocation

Multiple Character 👫 👬 👭

Prompt Continuation
oReact xAttr
oWant xIntent
oEffect sentence

💁‍♀️ Features

  • Filter level:

    • --comet_use: T or F.
    • --filter_level : str.
      • weak: As long as any 1st sentence's "effects on others" matches the 2nd sentence's "effects/ attr/causes for PersonX", return True
      • medium: As long as any 1st sentence's "effects on others" matches the 2nd sentence's "attr/causes for PersonX", return True
      • strong: oWant -> xIntent; oEffect -> xNeed; oReact -> xAttr
      • want: oWant -> xIntent
      • need: oEffect -> xNeed
      • effect: oReact -> xAttr
    • Relax the match condition to weak (--random_tech) when there is no match after 50 rounds
    • --num_matching: int, default is 1, indicating how many matching has to be reached.

  • Parser for extracting character names (spacy==2.3.2 and neuralcoref==4.0)

      pip install neuralcoref       
  python -m spacy download en
    • spacy.strings.StringStore size changed error: try installing from source

  • Similarity check for matching

      pip install sentence_transformers
    • The similarity file is located at CommonsenseStoryGen/cometFilter/similarity.py
    • It embeds the output from the COMET and calculate the similarity score for matching.
    • The default similarity score threshold is 0.8 (--sim_threshold).

  • Diverse beam search

    • download this weight to the /beam_search folder

        curl -O  "https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz"

    • Run the codes with diverse beam search (hamming)

      • --beam_search: diverse beam search type, i.e. hamming (#TODO for more options)
      • --beam_lamb: the coefficient for adding penalty, larger is more penalty

    • Modify the words prob

      • do NOT decrease the prob of high frequency words (TFIDF) - added
      • do NOT decrease the prob of characters
        • --exclude_chars: bool, to control if exclude chars names

    • Use similarity to penalize:

      • --glove_use: bool, to control whether to use glove embedding

  • Filter parser: add parser to filter process

    • Use the parser to check if the sentence candidate generate the third characters.
    • --filter_parser: T or F

  • Coreference resolution:

    • We use Coreference Resolution in spaCy with Neural Networks here.
    • If we find the third char or like 'them', some coref which is not among the two chars, we filter it out.
    • --coref_use : T or F
    • Only choose one between coref and filter parser.

  • ❣️ Use the whole history to generate text instead of one sentence!

    • --history_use: T or F, default is T

  • Baseline with no chars name.

    • We replace the name of characters with [MALE], [FEMALE], [NEURAL] and train the gpt2, located in /gpt2_models/roc_model_generalized
    • --char_name_use: bool, default=True. Use F/ False if you wanna use this model

  • Number of characters

    • We define this model is used to generate a 2-char or 1-char story generation.
    • --two_char_use: bool, default is True , if you only wanna consider one character, pls set it as False

  • Backtrack

    • We use backtrack to help find the matching?
    • --backtrack_use: bool. True indicates use the backtrack.

Fine-tune LM

  • RocStories with [CHAR1] and [CHAR2]:cometFilter/data_story/100KStories_dealed.txt
  • prompt dealed files: cometFilter/data_story/prompt_deal.txt
  • Fine-tune LM on roc dataset: 
    python cometFilter/finetune_language_model/finetune_gpt2_roc.py
  • Fine-tune LM on prompt_char: 
    python cometFilter/finetune_language_model/finetune_on_roc.py

  • Run the parser, you need to go to ./stanford-corenlp-4.1.0 and then run java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 30000