shanyu-sys/Awesome-multi-modal-embeddings

Awesome-multi-modal-embeddings

This list includes the most state-of-the-art projects on multi-modal-embeddings, which are

• with high impact and code stability
  • high number of GitHub stars
  • model used in other papers' evaluations
• paper accepted in top conferences (CVPR, ICML, .etc)

Note that some of projects with more than 3 modalities may not be widely used.

More than 3 modalities

• ImageBind: One Embedding Space To Bind Them All
• High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning

3 Modalities: vision-language-audio

2 Modalities: vision-language

• LAVIS - A Library for Language-Vision Intelligence

• OpenCLIP - An open source implementation of OpenAI's CLIP

• (BEiT-3) Image as a Foreign Language: BEiT Pretraining for Vision and Vision-Language Tasks

• I-JEPA

Other Awesome multimodal Repos:

• https://github.com/pliang279/awesome-multimodal-ml
• https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models
• https://github.com/DirtyHarryLYL/Transformer-in-Vision

Survey paper:

• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey