Awesome-Multimodal-Applications-In-Medical-Imaging
This repository includes resources on several applications of multi-modal learning in medical imaging.
Overview
Survey 
- [arXiv 2022] Visual Attention Methods in Deep Learning: An In-Depth Survey [pdf]
- [arXiv 2022] Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study [pdf]
- [arXiv 2022] Vision+X: A Survey on Multimodal Learning in the Light of Data [pdf]
Medical Report Generation 
- [NeurIPS 2021 Datasets and Benchmarks Track (Round 2)] FFA-IR: Towards an Explainable and Reliable Medical Report Generation Benchmark [pdf] [code]
- [arXiv 2020] Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation [pdf]
- [CVPR 2022] Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation [pdf]
- [EMNLP 2018] Automated Generation of Accurate & Fluent Medical X-ray Reports [pdf] [code]
- [ACL 2018] On the Automatic Generation of Medical Imaging Reports [pdf] [code]
- [ACL 2021] Competence-based Multimodal Curriculum Learning for Medical Report Generation [pdf]
- [NeurIPS 2018] Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation [pdf]
- [CVPR 2021] Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation [pdf]
- [MICCAI 2021] AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation [pdf]
- [NAACL-HLT 2021] Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation [pdf] [code]
- [MICCAI 2021] RATCHET: Medical Transformer for Chest X-ray Diagnosis and Reporting [pdf][code]
- [arXiv 2022] Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generation [pdf]
- [EMNLP 2020] Generating Radiology Reports via Memory-driven Transformer [pdf] [code]
- [ACCV 2020] Hierarchical X-Ray Report Generation via Pathology tags and Multi Head Attention [pdf] [code]
- [MICCAI 2021] Trust It or Not: Confidence-Guided Automatic Radiology Report Generation [pdf]
- [MICCAI 2021] Surgical Instruction Generation with Transformers [pdf]
- [MICCAI 2021] Class-Incremental Domain Adaptation with Smoothing and Calibration for Surgical Report Generation [pdf] [code]
- [Nature Machine Intelligence 2022] Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reports [pdf] [code]
- [MICCAI 2022] A Self-Guided Framework for Radiology Report Generation [pdf]
- [ACL-IJCNLP 2021] Cross-modal Memory Networks for Radiology Report Generation [pdf] [code]
- [MICCAI 2022] A Medical Semantic-Assisted Transformer for Radiographic Report Generation [pdf]
- [MIDL 2022] Representative Image Feature Extraction via Contrastive Learning Pretraining for Chest X-ray Report Generation [pdf]
- [MICCAI 2022] RepsNet: Combining Vision with Language for Automated Medical Reports [pdf] [code]
- [arXiv 2022] Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors [pdf]
- [IEEE TNNLS 2022] Hybrid Reinforced Medical Report Generation with M-Linear Attention and Repetition Penalty [pdf]
- [Medical Image Analysis 2022] CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generation [pdf]
- [arXiv 2022] Lesion Guided Explainable Few Weak-shot Medical Report Generation [pdf] [code]
- [arXiv 2022] Self adaptive global-local feature enhancement for radiology report generation [pdf]
- [arXiv 2022] On the Importance of Image Encoding in Automated Chest X-Ray Report Generation [pdf] [code]
- [arXiv 2022] RoentGen: Vision-Language Foundation Model for Chest X-ray Generation [pdf]
- [arXiv 2022] DeltaNet:Conditional Medical Report Generation for COVID-19 Diagnosis [pdf] [code]
- [arXiv 2023] Unified Chest X-ray and Radiology Report Generation Model with Multi-view Chest X-rays [pdf] [code]
Medical Visual Question Answering 
- [arXiv 2021] MuVAM: A Multi-View Attention-based Model for Medical Visual Question Answering [pdf]
- [MICCAI 2022] Consistency-preserving Visual Question Answering in Medical Imaging [pdf] [code]
- [TMI 2020] A Question-Centric Model for Visual Question Answering in Medical Imaging [pdf] [code]
- [arXiv 2021] Medical Visual Question Answering: A Survey [pdf]
- [arXiv 2022] Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformer [pdf] [code]
- [CLEF 2020 Working Notes] HCP-MIC at VQA-Med 2020: Effective visual representation for medical visual question answering [pdf] [code]
- [CLEF 2020 Working Notes] TeamS at VQA-Med 2021: BBN-Orchestra for long-tailed medical visual question answering [pdf] [code]
- [Nature Scientific Reports 2021] MedFuseNet: An attention-based multimodal deep learning model for visual question answering in the medical domain [pdf]
- [ECCV 2022] Distilled Dual-Encoder Model for Vision-Language Understanding [pdf] [code]
- [arXiv 2022] A Dual-Attention Learning Network with Word and Sentence Embedding for Medical Visual Question Answering [pdf] [code]
- [arXiv 2022] MF2-MVQA: A Multi-stage Feature Fusion method for Medical Visual Question Answering [pdf]
- [arXiv 2022] Self-supervised vision-language pretraining for Medical visual question answering [pdf] [code]
- [arXiv 2022] UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering [pdf]
- [arXiv 2023] Interpretable Medical Image Visual Question Answering via Multi-Modal Relationship Graph Learning [pdf]
- [arXiv 2023] Medical visual question answering using joint self-supervised learning [pdf]
Vision-Language Prompt 
- [arXiv 2022] Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video [pdf]
- [CVPR 2022] LAVT: Language-Aware Vision Transformer for Referring Image Segmentation [pdf] [code]
- [CVPR 2022] Image Segmentation Using Text and Image Prompts [pdf] [code]
- [IJCV 2022] Learning to Prompt for Vision-Language Models [pdf] [code]
- [CVPR 2022] Conditional Prompt Learning for Vision-Language Models [pdf] [code]
- [arXiv 2022] Neural Prompt Search [pdf] [code]
- [CVPR 2022] Prompt-RSVQA: Prompting Visual Context to a Language Model for Remote Sensing Visual Question Answering [pdf]
- [arXiv 2022] Prompt-to-Prompt Image Editing with Cross Attention Control [pdf]
- [arXiv 2022] Prompt Tuning for Generative Multimodal Pretrained Models [pdf] [code]
- [arXiv 2022] P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting [pdf] [code]
- [arXiv 2022] MILAN: Masked Image Pretraining on Language Assisted Representation [pdf] [code]
- [arXiv 2022] Class-Aware Visual Prompt Tuning for Vision-Language Pre-Trained Model [pdf]
- [arXiv 2022] Prompt Vision Transformer for Domain Generalization [pdf]
- [arXiv 2022] Prompt-Matched Semantic Segmentation [pdf]
- [ECCV 2022] Learning from Unlabeled 3D Environments for Vision-and-Language Navigation [pdf] [code]
- [arXiv 2022] Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task [pdf] [code]
- [arXiv 2022] Prompt Tuning with Soft Context Sharing for Vision-Language Models [pdf]
- [ICLR 2023] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models [pdf] [code]
- [NeurIPS 2022] Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models [pdf] [code]
- [NeurIPS 2022] M^4I: Multi-modal Models Membership Inference [pdf] [code]
- [IJCAI 2022] Declaration-based Prompt Tuning for Visual Question Answering [pdf] [code]
- [MICCAI 2022] Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training [pdf] [code]
- [NeurIPS 2022] GLIPv2: Unifying Localization and VL Understanding [pdf] [code]
- [arXiv 2022] Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study [pdf]
- [arXiv 2022] Language-Aware Soft Prompting for Vision & Language Foundation Models [pdf]
- [ICLR 2023] LPT: Long-tailed Prompt Tuning for Image Classification [pdf]
- [arXiv 2022] Prompt Learning with Optimal Transport for Vision-Language Models [pdf]
- [arXiv 2022] Variational prompt tuning improves generalization of vision-language models [pdf]
- [arXiv 2022] MaPLe: Multi-modal Prompt Learning [pdf] [code]
- [arXiv 2022] Learning to Decompose Visual Features with Latent Textual Prompts [pdf]
- [arXiv 2022] Visual Prompting for Adversarial Robustness [pdf]
- [arXiv 2022] Unified Vision and Language Prompt Learning [pdf] [code]
- [arXiv 2022] CPL: Counterfactual Prompt Learning for Vision and Language Models [pdf]
- [arXiv 2022] Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Models [pdf]
- [arXiv 2022] Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models [pdf] [code]
- [arXiv 2022] Multitask Vision-Language Prompt Tuning [pdf] [code]
- [arXiv 2022] ProSFDA: Prompt Learning based Source-free Domain Adaptation for Medical Image Segmentation [pdf] [code]
- [arXiv 2022] PromptCap: Prompt-Guided Task-Aware Image Captioning [pdf]
- [arXiv 2022] Prompt Tuning for Parameter-efficient Medical Image Segmentation [pdf] [code]
- [arXiv 2022] Task Residual for Tuning Vision-Language Models [pdf] [code]
- [arXiv 2022] Texts as Images in Prompt Tuning for Multi-Label Image Recognition [pdf] [code]
- [AAAI 2023] Controllable Image Captioning via Prompting [pdf]
- [arXiv 2022] See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning [pdf]
- [arXiv 2022] From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models [pdf]
- [arXiv 2022] Position-guided Text Prompt for Vision-Language Pre-training [pdf] [code]
- [arXiv 2022] Doubly Right Object Recognition: A Why Prompt for Visual Rationales [pdf]
- [arXiv 2022] PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discovery [pdf]
- [arXiv 2022] Unified vision and language prompt learning [pdf] [code]
- [arXiv 2022] Prompt Vision Transformer for Domain Generalization [pdf]
- [AAAI 2023] Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation [pdf]
- [NeurIPS 2022] DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations [pdf] [code]
- [ICLR 2023] Visual Classification via Description from Large Language Models [pdf]
- [CVPR 2022] Prompt Distribution Learning [pdf]
- [arXiv 2023] Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts [pdf] [code]
- [arXiv 2023] StyLIP: Multi-Scale Style-Conditioned Prompt Learning for CLIP-based Domain Generalization [pdf]
- [ICLR 2023] Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection [pdf] [code]