InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews

News

May. 2024: InCharacter got accepted to ACL 2024! See you in Bangkok.

Setup

Install necessary dependencies via:

pip install -r requirements.txt

No need to install ChatHaruhi separately; a fixed version of ChatHaruhi is already included in the code/ directory of this repository.

Enter the code folder.

cd code

Set your openai apikey in config.json. You can refer to config_template.json for its format.

Personality Assessment

To assess the personality of a specific role-playing agent (RPA), use the following commands:

Conduct personality test on a specific character:

python personality_tests.py --questionnaire_name BFI --character hutao --agent_type ChatHaruhi --agent_llm gpt-3.5 --evaluator_llm gpt-4 --eval_method interview_assess_batch_anonymous

Supported choices for eval_method include ['self_report' (SR), 'self_report_cot' (SR-CoT), 'expert_rating' (ER_batch), 'expert_rating_collective' (ER_all), 'option_conversion' (OC),'dimension_option_conversion' (d-OC)].

To reproduce our experiments on the 32 RPAs, please refer to code/run_experiments.py

BFI Personalities of Selected Characters/RPAs

Radar chart of BFI personalities of state-of-the-art RPAs (yellow) and the characters (blue). O, C, E, A, N stands for openness, consciousness, extroversion, agreeableness and neuroticism in the BFI.

Demo

[Online Demo]

English:

Interview Response:

Result:

Self-report Response:

(May give options inconsistent with character behaviors)

Chinese:

Interview Response:

Result:

PDB character extraction

PDB Character Search Script

This Python script facilitates automated searching of character profiles on the Personality Database (PDB) website using Selenium and BeautifulSoup. The script is designed to retrieve the ID of a given character by searching the website and extracting relevant profile information.

How to Use:

Install the necessary dependencies using the following command:
```
pip install requests beautifulsoup4 msedge-selenium-tools
```
Ensure you have Microsoft Edge and the corresponding WebDriver (msedgedriver.exe) installed.
Execute the script and input the desired character name. The script will return the ID of the character if found.

Example Usage:

character_id = get_character_id("Tony Stark")
print(character_id)

Neph0s/InCharacter