ChatCAD+: Towards a Reliable and Universal Interactive CAD using LLMs

by Zihao Zhao*, Sheng Wang*, Jinchen Gu*, Yitao Zhu*, Lanzhuju Mei, Zixu Zhuang, Zhiming Cui, Qian Wang, Dinggang Shen

Introduction

This repository provides the official implementation of some components of ChatCAD+:

Modality identification
Chinese version Interactive CAD of Chest X-rays
LLM-based knowledge retrieval
An easy-deploy local web ui based on Gradio
The online demo will be available soon

Resources

We would like to thank Merck Manual Professional who make all these medical knowledge public, we sorted their website for easier usage: here
A BART-based model that has the capability to translate chest X-ray reports into Chinese well [link]

Usage

weights&others

R2GenCMN: r2gcmn_mimic-cxr.pth and annotation.json
PCAM weights: JFchexpert.pth
Place annotation.json under ./r2g/ and pre-trained weights under ./weights/
For template retrieval system, please download MIMIC-CXR reports from official website and organize them into a dictionary, save as report_en_dict.json under the ./

You can either find them from original repository or dowload from Google Drive

Deploy local web ui

pip install -r requirements.txt
implement web.py and load your openai api-key

- Would like some diagnostic results? upload image via left panel --> wait for your report

- ChatCAD+ will answer your question with a reference from Merck Manucal Professional

即将到来的更新

Migrate the project to Gradio
Online demo with available dental and Knee MRI network

Citation

@article{wang2023chatcad,
  title={Chatcad: Interactive computer-aided diagnosis on medical image using large language models},
  author={Wang, Sheng and Zhao, Zihao and Ouyang, Xi and Wang, Qian and Shen, Dinggang},
  journal={arXiv preprint arXiv:2302.07257},
  year={2023}
}

@article{zhao2023chatcad,
      title={ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMs},
      author={Zihao Zhao and Sheng Wang and Jinchen Gu and Yitao Zhu and Lanzhuju Mei and Zixu Zhuang and Zhiming Cui and Qian Wang and Dinggang Shen},
      journal={arXiv preprint arXiv:2305.15964},
      year={2023},
}

Acknowledgment

Our implementation (including coming version) is based on the following codebases. We gratefully thank the authors for their wonderful works.

R2GenCMN, PCAM, CSNet.

ZhaoYongrui/ChatCAD