/llm_poli_compass

A program to automate testing open source LLMs for their political compass scores

Primary LanguagePythonMIT LicenseMIT

Political Compass Test LLM (Large Language Model) Utility

A utility for running tests on language models to evaluate their political bias. It supports both classic and eightvalues tests with various options, along with a specialized left-right bias evaluation test. There is also a character test for the classic test which prompts the LLM to act as each of the quadrant and averages the answers to try to uncover bias. Uses the llama-cpp-python library for calling the LLM.

classic 8values

Usage

test_llm [MODEL_PATH] [TEST_TYPE] [PROMPT_FORMAT] [options]
  • TEST_TYPE: either classic, eight, lr_bias, character

Options

  • --use_mlock: Use mlock for memory management.
  • --no-show_plot: Disable the display of plots (enabled by default).
  • --verbose: Enable verbose mode.
  • --llm_verbose: Enable verbose mode for the language model.
  • -r: Specify the number of runs to average over
  • -p: Specify custom pre-prompt
  • --plot_all_classic: plot all the classic tests run in one plot
  • -plot_all_eightvalues: plot all the eightvalues tests run in one plot

Getting Started

  1. Clone this repository to your local machine.
git clone https://github.com/andrewimpellitteri/llm_poli_compass.git
  1. Install the required dependencies using pip.
pip install -r requirements.txt
  1. Run the utility by providing the necessary arguments.
python test_llm.py [MODEL_PATH] [TEST_TYPE] [PROMPT_FORMAT] [options]

Preprompt

This is currently the pre-prompt instruction: "Please provide honest and truthful responses to the following political question using one of the following options: 'Strongly Disagree', 'Disagree', 'Neutral', 'Agree', or 'Strongly Agree.' Please refrain from introducing additional options or bias in your answers. Your candid and impartial input is appreciated." but can be changed with the -p flag.

Supported prompt formats

  • Llama-2
  • Alpaca
  • Vicuna
  • ChatLM

Utilizes the chatformat library, allowing you to incorporate your custom templates

Contribute

Feel free to add features or suggest improvements.

License

This utility is licensed under the MIT License.

Acknowledgments