HalluciMAD

Interpreting and Mitigating Hallucination in MLLMs through Multi-agent Debate

Install

Clone this repository and navigate to HalluciMAD folder

git clone https://github.com/LZzz2000/HalluciMAD.git
cd HalluciMAD

Install Package

tqdm
time
google-generativeai
Pillow
random
json

Download Dataset

Please download val2014 and extract it to the data directory.

The annotation files have been downloaded.

./HalluciMAD/data/val2014
./HalluciMAD/data/coco_pope_random.json
./HalluciMAD/data/coco_pope_popular.json
./HalluciMAD/data/coco_pope_adversarial.json
./HalluciMAD/data/coco_pope_random_POPER&POPEC.json

Get API Key

Please get api_key from Google AI Studio.

google_api_key = '' # add your api_key

Run

Enter the code directory and fill in the out_file.
Run

python multi_eval_pope_gemini.py # Our approach
python sro_eval_pope_gemini.py # Self Reflection Only
python single_eval_pope_gemini.py # Baseline

Evaluation

Enter the eval directory and fill in the ans_file.
Set the out flag to "True" if you want to output the bad case.
Run

python evaluate.py