YevhenKost/eval_causal_reasoning_pairs

Causal Reasoning Capability of LLMs. Evaluation System

This repository contains code for the evaluation of different deep learning models for the sentence pair causality detection task.

Usage

Install the requirements from requirements.txt

All the required settings are located in settings.py. See the docs for each setting field there.

To run MNLI experiment:

python run_mnli.py

To run LLM experiment:

python run_llms.py

Dataset

The dataset is COPA dataset with some manual preprocessing and filtering. The format example and dataset itself can be found in copa-processed-ds.json

Additional References

instruct_pipeline.py was taken from the oficial Databricks page on huggingface.co