New!🔥 Checkout our live radiology report generation 📝 space on HuggingFace🤗
ViLMedic: a framework for research at the intersection of vision and language in medical AI
conda create --name vilmedic python==3.9 -y
git clone https://github.com/jbdel/vilmedic
python setup.py develop
Learn more about ViLMedic here.
ViLMedic hosts a zoo of pretrained models.
from vilmedic import AutoModel
model, processor = AutoModel.from_pretrained("selfsup/convirt-mimic")
batch = processor.inference(seq=["no acute cardiopulmonary process"],
image=["my_chest_xray.jpg"])
out = model(**batch)
print(out.keys())
# dict_keys(['loss', 'loss_l', 'loss_v', 'linguistic', 'visual'])
| Name | dataset | Report preprocessing |
|---|---|---|
| Radiology report generation | ||
| rrg/biomed-roberta-baseline-mimic | mimic-cxr | r2gen |
| rrg/biomed-roberta-baseline-indiana | indiana | r2gen |
| rrg/baseline-padchest | padchest | - |
| Radiology report summarization | ||
| rrs/biomed-roberta-baseline-mimic | mimic-cxr | rouge |
| rrs/biomed-roberta-baseline-indiana | indiana | r2gen |
| Self-supervision | ||
| selfsup/convirt-mimic | mimic-cxr | r2gen |
| selfsup/convirt-mimic-balanced | mimic-cxr | r2gen |
| selfsup/convirt-padchest-16 | padchest | gloria |
| selfsup/convirt-padchest-32 | padchest | gloria |
| selfsup/convirt-indiana-16 | indiana | r2gen |
| selfsup/convirt-indiana-32 | indiana | r2gen |
| selfsup/convirt-indiana-64 | indiana | r2gen |
| selfsup/gloria-chexpert | CheXpert | gloria |
| selfsup/gloria-mimic-48 | mimic-cxr | r2gen |
| selfsup/simclr-mimic-16 | mimic-cxr | |
| selfsup/simclr-mimic-32 | mimic-cxr | |
| selfsup/simclr-mimic-64 | mimic-cxr | |
| selfsup/vae-mimic | mimic-cxr | |
| selfsup/vae-indiana | indiana | |
| selfsup/vae-padchest | padchest | |
| Medical VQA | ||
| mvqa/mvqa-imageclef | ImageCLEF-VQAMed |
ViLMedic replicates solutions from the multimodal medical literature.
If you use ViLMedic in your work or use any models published in ViLMedic, please cite:
@inproceedings{delbrouck-etal-2022-vilmedic,
title = "{V}i{LM}edic: a framework for research at the intersection of vision and language in medical {AI}",
author = "Delbrouck, Jean-benoit and
Saab, Khaled and
Varma, Maya and
Eyuboglu, Sabri and
Chambon, Pierre and
Dunnmon, Jared and
Zambrano, Juan and
Chaudhari, Akshay and
Langlotz, Curtis",
booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
month = may,
year = "2022",
address = "Dublin, Ireland",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.acl-demo.3",
pages = "23--34",
}ViLMedic is MIT-licensed. The license applies to the pre-trained models as well.