memray/llm_phrase_semantics

Code for EMNLP2024 paper: Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models

[EMNLP2024]Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models

This is the official repository for the EMNLP 2024 Findings paper Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models. This repo is built based on Phrase-BERT and PiC.

Full results can be seen in this Google Sheet.

Command examples

Note: the models to run are configured in config/model_path.py.

Run OpenAI models on Turney/BiRD/PiC (4-shot and Chain-of-Thought prompt)

python eval_turney.py \
  --input_path data/turney/data_test2170.txt \
  --output_dir output/turney_test2170/ \ 
  --k_shot 4 --cot true --openai_key YOUR_KEY

python eval_bird.py \
    --input_path data/bird/data_test3335.txt \
    --output_dir output/bird_test3335/ \
    --k_shot 4 --cot true --openai_key YOUR_KEY

python eval_PiC_PS.py \
  --input_path data/PiC-PS/test-hard-v2.0.1.json \
  --output_dir output/PiC-PS_20240609/ \
  --k_shot 4 --cot true --openai_key YOUR_KEY