🔥🔥 This is a collection of awesome articles about foundation models in medical imaging🔥🔥
Our survey paper on arXiv: Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision ❤️
If you find our work useful in your research, please consider citing:
@article{azad2023foundational,
title={Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision},
author={Azad, Bobby and Azad, Reza and Eskandari, Sania and Bozorgpour, Afshin and Kazerouni, Amirhossein and Rekik, Islem and Merhof, Dorit},
journal={arXiv preprint arXiv:2310.18689},
year={2023}
}Foundation models, large-scale pre-trained deep learning models adaptable to various tasks, have gained interest across deep learning applications. In the medical imaging field, they enable contextual reasoning, generalization, and prompt-based task adjustments. This survey provides an overview of foundation models in medical imaging, covering fundamental concepts, taxonomy based on training strategies, application domains, imaging modalities, and more. It highlights practical use cases, applications, future directions, and challenges, including interpretability, data management, computational needs, and contextual comprehension.
We strongly encourage authors of relevant works to make a pull request and add their paper's information.
Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision🔥
Bobby Azad, Reza Azad, Sania Eskandari, Afshin Bozorgpour, Amirhossein Kazerouni, Islem Rekik, Dorit Merhof
[28th Oct., 2023] [arXiv, 2023]
[Paper]
Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning
Weijian Huang, Cheng Li, Hao Yang, Jiarun Liu, Shanshan Wang
[12th Sep., 2023] [arXiv, 2023]
[Paper]
A visual-language foundation model for pathology image analysis using medical Twitter
Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J. Montine, James Zou
[17th Aug., 2023] [Nature Medicine, 2023]
[Paper] [GitHub]
ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders
Shawn Xu, Lin Yang, Christopher Kelly, Marcin Sieniek, Timo Kohlberger, Martin Ma, Wei-Hung Weng, Atilla Kiraly, Sahar Kazemzadeh, Zakkai Melamed, Jungyeon Park, Patricia Strachan, Yun Liu, Chuck Lau, Preeti Singh, Christina Chen, Mozziyar Etemadi, Sreenivasa Raju Kalidindi, Yossi Matias, Katherine Chou, Greg S. Corrado, Shravya Shetty, Daniel Tse, Shruthi Prabhakara, Daniel Golden, Rory Pilgrim, Krish Eswaran, Andrew Sellergren
[2nd Aug., 2023] [arXiv, 2023]
[Paper]
Knowledge Boosting: Rethinking Medical Contrastive Vision-Language Pre-Training
Xiaofei Chen, Yuting He, Cheng Xue, Rongjun Ge, Shuo Li, Guanyu Yang
[14th Jul., 2023] [MICCAI, 2023]
[Paper] [GitHub]
Text-guided Foundation Model Adaptation for Pathological Image Classification
Yunkun Zhang, Jin Gao, Mu Zhou, Xiaosong Wang, Yu Qiao, Shaoting Zhang, Dequan Wang
[27th Jul., 2023] [MICCAI, 2023]
[Paper] [GitHub]
Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images
Ming Y. Lu, Bowen Chen, Andrew Zhang, Drew F.K. Williamson, Richard J. Chen, Tong Ding, Long Phi Le, Yung-Sung Chuang, Faisal Mahmood
[13th Jun., 2023] [CVPR, 2023]
[Paper]
Large-Scale Domain-Specific Pretraining for Biomedical Vision-Language Processing
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Matthew P. Lungren, Tristan Naumann, Hoifung Poon
[2nd Mar., 2023] [arXiv, 2023]
[Paper] [GitHub]
Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts
Zhihong Chen, Shizhe Diao, Benyou Wang, Guanbin Li, Xiang Wan
[17th Feb., 2023] [arXiv, 2023]
[Paper] [GitHub]
Learning to Exploit Temporal Structure for Biomedical Vision Language Processing
Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Pérez-García, Maximilian Ilse, Daniel C. Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori, Javier Alvarez-Valle, Ozan Oktay
[11th Jan., 2023] [CVPR, 2023]
[Paper]
CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection
Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, Zongwei Zhou
[2nd Jan., 2023] [ICCV, 2023]
[Paper] [GitHub]
MedCLIP: Contrastive Learning from Unpaired Medical Images and Text
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, Jimeng Sun
[18th Oct., 2022] [EMNLP, 2022]
[Paper] [GitHub]
Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning
Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P. Langlotz, Andrew Y. Ng, Pranav Rajpurkar
[15th Sep., 2022] [Nature Biomedical Engineering, 2022]
[Paper]
Radiology-Llama2: Best-in-Class Large Language Model for Radiology
Zhengliang Liu, Yiwei Li, Peng Shu, Aoxiao Zhong, Longtao Yang, Chao Ju, Zihao Wu, Chong Ma, Jie Luo, Cheng Chen, Sekeun Kim, Jiang Hu, Haixing Dai, Lin Zhao, Dajiang Zhu, Jun Liu, Wei Liu, Dinggang Shen, Tianming Liu, Quanzheng Li, Xiang Li
[29th Aug., 2023] [arXiv, 2023]
[Paper]
ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation
Guangyu Wang, Guoxing Yang, Zongxin Du, Longjun Fan, Xiaohu Li
[16th Jun., 2023] [arXiv, 2023]
[Paper]
XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models
Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, Fahad Shahbaz Khan
[13th Jun., 2023] [arXiv, 2023]
[Paper] [GitHub]
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, Jianfeng Gao
[1st Jun., 2023] [arXiv, 2023]
[Paper] [GitHub]
PMC-LLaMA: Towards Building Open-source Language Models for Medicine
Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, Weidi Xie
[27th Apr., 2023] [arXiv, 2023]
[Paper] [GitHub]
Visual Med-Alpaca: A Parameter-Efficient Biomedical LLM with Visual Capabilities
Chang Shu, Baian Chen, Fangyu Liu, Zihao Fu, Ehsan Shareghi, Nigel Collier
[11th Apr., 2023] [GitHub, 2023]
[GitHub]
ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge
Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, You Zhang
[24th Mar., 2023] [Cureus, 2023]
[Paper] [GitHub]
DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4
Zhengliang Liu, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Wei Liu, Dinggang Shen, Quanzheng Li, Tianming Liu, Dajiang Zhu, Xiang Li
[20th Mar., 2023] [arXiv, 2023]
[Paper] [GitHub]
ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models
Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, Dinggang Shen
[14th Feb., 2023] [arXiv, 2023]
[Paper]
Med-Flamingo: a Multimodal Medical Few-shot Learner
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, Jure Leskovec
[27th Jul., 2023] [arXiv, 2023]
[Paper] [GitHub]
Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports Generation
none
[22nd Jun., 2022] [AAAI, 2022]
[Paper]
Towards Expert-Level Medical Question Answering with Large Language Models
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, Vivek Natarajan
[16th May., 2023] [arXiv, 2023]
[Paper]
MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts
Qiuhui Chen, Xinyue Hu, Zirui Wang, Yi Hong
[18th May., 2023] [arXiv, 2023]
[Paper] [GitHub]
Vision-Language Model for Visual Question Answering in Medical Imagery
none
[22nd Feb., 2023] [Bioengineering, 2023]
[Paper]
Virchow: A Million-Slide Digital Pathology Foundation Model
Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Siqi Liu, Philippe Mathieu, Alexander van Eck, Donghun Lee, Julian Viret, Eric Robert, Yi Kan Wang, Jeremy D. Kunz, Matthew C. H. Lee, Jan Bernhard, Ran A. Godrich, Gerard Oakley, Ewan Millar, Matthew Hanna, Juan Retamero, William A. Moye, Razik Yousfi, Christopher Kanan, David Klimstra, Brandon Rothrock, Thomas J. Fuchs
[14th Sep., 2023] [arXiv, 2023]
[Paper]
SAM-Med2D
Junlong Cheng, Jin Ye, Zhongying Deng, Jianpin Chen, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, Hui Sun, Junjun He, Shaoting Zhang, Min Zhu, Yu Qiao,
[30th Aug., 2023] [arXiv, 2023]
[Paper] [GitHub]
SAM-U: Multi-box prompts triggered uncertainty estimation for reliable SAM in medical image
Guoyao Deng, Ke Zou, Kai Ren, Meng Wang, Xuedong Yuan, Sancong Ying, Huazhu Fu
[11th Jul., 2023] [arXiv, 2023]
[Paper]
How to Efficiently Adapt Large Segmentation Model(SAM) to Medical Images
Xinrong Hu, Xiaowei Xu, Yiyu Shi
[23rd Jun., 2023] [arXiv, 2023]
[Paper] [GitHub]
LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching
Duy M. H. Nguyen, Hoang Nguyen, Nghiem T. Diep, Tan N. Pham, Tri Cao, Binh T. Nguyen, Paul Swoboda, Nhat Ho, Shadi Albarqouni, Pengtao Xie, Daniel Sonntag, Mathias Niepert
[20th Jun., 2023] [arXiv, 2023]
[Paper] [GitHub]
Customized Segment Anything Model for Medical Image Segmentation
Kaidong Zhang, Dong Liu
[26th Apr., 2023] [arXiv, 2023]
[Paper] [GitHub]
Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
[25th Apr., 2023] [arXiv, 2023]
[Paper] [GitHub]
Segment Anything in Medical Images
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, Bo Wang
[24th Apr., 2023] [arXiv, 2023]
[Paper] [GitHub]
A foundation model for generalizable disease detection from retinal images
Yukun Zhou, Mark A. Chia, Siegfried K. Wagner, Murat S. Ayhan, Dominic J. Williamson, Robbert R. Struyven, Timing Liu, Moucheng Xu, Mateo G. Lozano, Peter Woodward-Court, Yuka Kihara, Andre Altmann, Aaron Y. Lee, Eric J. Topol, Alastair K. Denniston, Daniel C. Alexander, Pearse A. Keane
[5th Oct., 2023] [Nature, 2023]
[Paper] [GitHub]
Towards Generalist Foundation Model for Radiology
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, Weidi Xie
[4th Aug., 2023] [arXiv, 2023]
[Paper] [GitHub]
Towards Generalist Biomedical AI
Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, Basil Mustafa, Aakanksha Chowdhery, Yun Liu, Simon Kornblith, David Fleet, Philip Mansfield, Sushant Prakash, Renee Wong, Sunny Virmani, Christopher Semturs, S Sara Mahdavi, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Karan Singhal, Pete Florence, Alan Karthikesalingam, Vivek Natarajan
[26th Jul., 2023] [arXiv, 2023]
[Paper]
BiomedGPT: A Unified and Generalist Biomedical Generative Pre trained Transformer for Vision, Language, and Multimodal Tasks
Kai Zhang, Jun Yu, Zhiling Yan, Yixin Liu, Eashan Adhikarla, Sunyang Fu, Xun Chen, Chen Chen, Yuyin Zhou, Xiang Li, Lifang He, Brian D. Davison, Quanzheng Li, Yong Chen, Hongfang Liu, Lichao Sun
[26th May., 2023] [arXiv, 2023]
[Paper] [GitHub]
Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation
Peilun Shi, Jianing Qiu, Sai Mu Dalike Abaxi, Hao Wei, Frank P.-W. Lo, Wu Yuan
[25th Apr., 2023] [Diagnostics, 2023]
[Paper] [GitHub]
Foundation models for generalist medical artificial intelligence
Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, Pranav Rajpurkar
[12th Apr., 2023] [Nature, 2023]
[Paper]