We now provide a pretrained OmniFM-DR on June 9, 2023!
Click the image to have a try with OmniFM-DR around the chest DR images
This repository provides the official implementation of OmniFM-DR
key feature bulletin points here
- First foundation model for multi-task analysis of chest DR image
- The largest full labeled chest DR dataset
- Supoort 5 tpyes of downstream tasks
- Report Generation
- Disease Localization
- Segmentation
- Classification
- Visual Question Answering
We have built a multimodal multitask model for DR data, aiming to solve all tasks in this field with one model, such as report generation, disease detection, disease question answering, and even segmentation. Without any fine-tuning, our model has achieved satisfactory results in report generation, disease detection and question answering.
More intro text here.
We utilize 10 public and 6 private datasets for pre-training and provide the download via the following links:
Public dataset:
- MIMIC-CXR
- VinDR
- ChestX-Det-Dataset
- ChestX-ray14
- CheXpert
- TBX11K
- object-CXR
- JSRT Database
- Shenzhen chest X-ray Set
- Montgomery County chest X-ray Set
Main Requirements
- python 3.7.4
- pytorch 1.11.0
- torchvision 0.12.1
Installation
git clone https://github.com/MedHK23/OmniFM-DR.git
git clone --depth 1 --filter=blob:none https://github.com/OFA-Sys/OFA.git fairseq
pip install -e ./fairseq/
pip install -r requirements.txtDownload Model
You can download from Huggingface. Put it under folder checkpoints.
Testing
python demo.py # result can be checked in logsThis project is under the CC-BY-NC 4.0 license. See LICENSE for details.
A lot of code is modified from OFA-Sys. Thanks them for releasing their codes.
If you find this repository useful, please consider citing this paper:
@article{xu2023learning,
title={Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretation},
author={Xu, Lijian and Ni, Ziyu and Liu, Xinglong and Wang, Xiaosong and Li, Hongsheng and Zhang, Shaoting},
journal={arXiv preprint arXiv:2311.01092},
year={2023}
}