/HaELM

An automatic MLLM hallucination detection framework

Primary LanguagePython

HaELM

An automatic MLLM hallucination detection framework

1. Installing

Install peft

$ pip install git+https://gitclone.com/github.com/huggingface/peft.git -i https://pypi.mirrors.ustc.edu.cn/simple --trusted-host=pypi.mirrors.ustc.edu.cn

2. Preparing

Download the checkpoint of llama-7b-hf

3. Training

We provide the hallucination training dataset in "data/train_data.jsonl" and the manually labeled validation set in "data/eval_data.jsonl". If you want to:

  • Retrain
  • Use another scale of llama
  • Use llama-2
  • Use additional data

see here.

  • Modify the path in lines 19-21 of finetune.py
  • Run the command below
python finetune.py

4. Interface

We provide interface templates populated by the output of mPLUG-Owl in "LLM_output/mPLUG_caption.jsonl".

  • Modify the path in lines 14-16 of interface.py
  • Run the command below
python interface.py

5. Citation

@article{wang2023evaluation,
  title={Evaluation and Analysis of Hallucination in Large Vision-Language Models},
  author={Wang, Junyang and Zhou, Yiyang and Xu, Guohai and Shi, Pengcheng and Zhao, Chenlin and Xu, Haiyang and Ye, Qinghao and Yan, Ming and Zhang, Ji and Zhu, Jihua and others},
  journal={arXiv preprint arXiv:2308.15126},
  year={2023}
}