/MSAT

Source code for the paper "A Medical Semantic-Assisted Transformer for Radiographic Report Generation"

Primary LanguageJupyter Notebook

MSAT

Source code for the paper "A Medical Semantic-Assisted Transformer for Radiographic Report Generation"


overview of the proposed framework
Figure: A overview of the proposed framework

Requirements

  • Python 3
  • CUDA 11
  • tqdm
  • easydict
  • psutil
  • ftfy
  • regex
  • tqdm
  • PyTorch=1.7.1
  • torchvision

Data preparation

  1. Extract clip features: Download the images and our preprocessed annoatation.json file.
python tools/extract_clip_feature.py --annotation mimic_cxr/annotation.json --save_path ./data/feature/mimic_clip16_att_512
  1. Convert reports to tokens and save it to data/mimic folder using the following script. Or download from here
python tools/build_vocab.py --annotation mimic_cxr/annotation.json --save_path data/mimic --radgraph data/mimic/MIMIC-CXR_graphs.json
  1. Download metric package from here and unzip it into MSAT folder.

Training

Train MSAT model

python main.py --folder experiments/V1

Train MSAT model using reinforcement learning

python main.py --folder experiments/V1_rl --resume experiments/V1/snapshot/{best_model}.pth

Acknowledgements

Thanks the contribution of image-captioning, self-critical.pytorch and awesome PyTorch team.