Awesome Prompting Papers in Computer Vision

Introduction

A curated list of prompt-based papers in computer vision and vision-language learning.

Keywords:

Task tag, e.g.,
Abbreviation tag, e.g.,
Characteristic tag: Some characteristic makes this paper unique, e.g.,
Bold font: We highlight some pilot work that may contribute to the prevalence of visual prompting.

Prompt Learning

This section contains papers designing prompt (containing adapter) modules for parameter-efficient adaptation of foundation models.

Vision Prompt

Learning to Prompt for Continual Learning [pdf] [code]

CVPR 2022
Visual Prompt Tuning [pdf] [code]

ECCV 2022
Exploring Visual Prompts for Adapting Large-Scale Models [pdf] [code]

Arxiv 2022/03
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning [pdf] [code]

ECCV 2022
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition [pdf] [code]

Arxiv 2022/05
Vision Transformer Adapter for Dense Predictions [pdf] [code]

Arxiv 2022/05
Neural Prompt Search [pdf] [code]

Arxiv 2022/06
Convolutional Bypasses Are Better Vision Transformer Adapters [pdf] [code]

Arxiv 2022/07
Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets [pdf]

Arxiv 2022/08
Prompt Vision Transformer for Domain Generalization [pdf]

Arxiv 2022/08
Visual Prompting via Image Inpainting [pdf] [code]

Arxiv 2022/09

Vision-Language Prompt

Learning Transferable Visual Models From Natural Language Supervision [pdf] [code]

ICML 2021
Learning to Prompt for Vision-Language Models [pdf] [code]

IJCV 2022
Prompt Distribution Learning [pdf]

CVPR 2022
Conditional Prompt Learning for Vision-Language Models [pdf] [code]

CVPR 2022
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting [pdf] [code]

CVPR 2022
Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos [pdf] [code]

CVPR 2022
PointCLIP: Point Cloud Understanding by CLIP [pdf] [code]

CVPR 2022
VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks [pdf] [code]

CVPR 2022
Can Language Understand Depth? [pdf] [code]

ACM MM 2022
Expanding Language-Image Pretrained Models for General Video Recognition [pdf] [code]

ECCV 2022
Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification [pdf] [code]

ECCV 2022

Colorful Prompt Tuning for Pre-trained Vision-Language Models [pdf]

Arxiv 2021/08
ActionCLIP: A New Paradigm for Video Action Recognition [pdf] [code]

Arxiv 2021/09
CLIP-Adapter: Better Vision-Language Models with Feature Adapters [pdf] [code]

Arxiv 2021/10
Amortized Prompt: Lightweight Fine-Tuning for CLIP in Domain Generalization [pdf]

Arxiv 2021/11
Prompting Visual-Language Models for Efficient Video Understanding [pdf] [code]

Arxiv 2021/12
Unsupervised Prompt Learning for Vision-Language Models [pdf] [code]

Arxiv 2022/04
Prompt-aligned Gradient for Prompt Tuning [pdf] [code]

Arxiv 2022/05
Parameter-Efficient Image-to-Video Transfer Learning [pdf]

Arxiv 2022/06
DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations [pdf]

Arxiv 2022/06
Rethinking the Openness of CLIP [pdf]

Arxiv 2022/06
OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression [pdf]

Arxiv 2022/06
Prompt Tuning for Generative Multimodal Pretrained Models [pdf] [code]

Arxiv 2022/06
Prompt Tuning with Soft Context Sharing for Vision-Language Models [pdf]

Arxiv 2022/08
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models [pdf] [code]

Arxiv 2022/09

Language-Interactable Prompt

Language-interactable prompter develops few/zero-shot capabilities by prompting one/several independent foundational models (VLMs, LMs, VMs, etc.) with the language interface.

Multimodal Few-Shot Learning with Frozen Language Models [pdf]

NIPS 2021
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA [pdf] [code]

AAAI 2022
A Good Prompt Is Worth Millions of Parameters? Low-resource Prompt-based Learning for Vision-Language Models [pdf]

ACL 2022
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning [pdf] [code]

CVPR 2022

ClipCap: CLIP Prefix for Image Captioning [pdf] [code]

Arxiv 2021/11
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language [pdf] [code]

Arxiv 2022/04
Flamingo: a Visual Language Model for Few-Shot Learning [pdf]

Arxiv 2022/04
Language Models Can See: Plugging Visual Controls in Text Generation [pdf] [code]

Arxiv 2022/05
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models [pdf]

Arxiv 2022/06
Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning [pdf]

Arxiv 2022/06

Application of Prompt

This section contains awesome papers using the prompt module as tools, like papers using prompts for pretraining or specific applications.

Unifying Vision-and-Language Tasks via Text Generation [pdf] [code]

ICML 2021
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery [pdf] [code]

ICCV 2021
Grounded Language-Image Pre-training [pdf] [code]

CVPR 2022
Align and Prompt: Video-and-Language Pre-training with Entity Prompts [pdf] [code]

CVPR 2022
GroupViT: Semantic Segmentation Emerges from Text Supervision [pdf] [code]

CVPR 2022
Unified Multimodal Pretraining and Prompt-based Tuning for Vision-Language Understanding and Generation [pdf]

Arxiv 2021/12
Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning [pdf]

Arxiv 2022/08

Other Resources

PromptPapers: A comprehensive curated list for prompting papers (mainly in natural language processing)
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing [pdf] Arxiv 2021/07

Alxemade/Awesome_Prompting_Papers_in_Computer_Vision