/Awesome-ComposableAI

A curated list of Composable AI methods: Building AI system by composing modules.

Paris

LEGOcool Awesome-ComposableAI LEGOcool

Awesome Awesome ComposableAI

A curated list of Composable AI methods: Building AI system by composing modules.

It includes Composable-Model, Composable-Task, Composable-Gen, Composable-Agent, and Composable-X, etc.

Contributions are welcome!

lego_brick Let's build AI as Lego! lego_brick

Table of Content

Definition

A paramount issue in AI is the combinational challenge: It is infeasible to enumerate all possibilities for an intelligent system.

Composable AI offers a solution to this challenge by emphasizing the creation of modular, flexible, and reusable AI components. These components can be assembled and reconfigured in various ways, enabling the construction of customized AI systems that are specifically tailored to individual tasks, domains, or applications.

Composable-Task

Projects & Papers

Title & Authors Intro Useful Links
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Yueting Zhuang
Preprint'23

[Jarvis (Project)]
[Github]
[Demo]
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, Lijuan Wang
Preprint'23

[MM-REACT (Project)]
intro [Github]
[Page]
[Demo]
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs
Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, Yun Wang, Linjun Shou, Ming Gong, Nan Duan Preprint'23
intro [Github]
OpenAGI: When LLM Meets Domain Experts
Yingqiang Ge, Wenyue Hua, Jianchao Ji, Juntao Tan, Shuyuan Xu, Yongfeng Zhang

[OpenAGI (Project)]
intro Github
ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, Mohamed Elhoseiny
Preprint'23
[ChatCaptioner (Project)]
intro Github
Visual Programming: Compositional visual reasoning without training
Tanmay Gupta, Aniruddha Kembhavi
CVPR'23
Visprog (Project)
intro [Github]
[Page]
ViperGPT: Visual Inference via Python Execution for Reasoning
Dídac Surís, Sachit Menon, Carl Vondrick
CVPR'23
Viper (Project)
intro [Github]
[Page]
Composing Ensembles of Pre-trained Models via Iterative Consensus
Shuang Li, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Igor Mordatch
ICLR'23
intro [Page]
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, Pete Florence
ICLR'23
socraticmodels (Project)
intro [Github]
[Page]

Project Only

Title & Authors Intro Useful Links
Grounded-SAM (Project)
[Grounding DINO + Segment-Anything + X]
Shilong Liu and Zhaoyang Zeng and Tianhe Ren and Feng Li and Hao Zhang and Jie Yang and Chunyuan Li and Jianwei Yang and Hang Su and Jun Zhu and Lei Zhang
intro [Github]
[Demo]
Semantic-Segment-Anything (Project)
[Close-set Segmenters + Open-vocabulary Models]
Jiaqi Chen, Zeyu Yang, Li Zhang
intro [Github]
Star
Inpaint-Anything: Segment Anything Meets Image Inpainting (Project)
[SAM + LaMa + Stable Diffusion]
Tao Yu
intro [Github]
Segment Anything and Name It (Project)
[Visual ChatGPT + GLIP + Segment-Anything]
Peize Sun and Shoufa Chen
intro [Github]
Star
EditAnything (Project)
[Segment Anything + ControlNet + BLIP2 + Stable Diffusion]
Shanghua Gao, Pan Zhou
intro [Github]

Composable-Model

Projects & Papers

Title & Authors Intro Useful Links
AdapterHub: A Framework for Adapting Transformers
Álvaro Barbero Jiménez
EMNLP'20

[adapter-transformers (Project)]
Girl in a jacket [Github]
[Page]
Deep Model Reassembly
Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, Xinchao Wang
NeurIPS'22

[DeRy (Project)]
intro [Github]
[Page]
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean
ICLR'17

[mixture-of-experts (Project)]
intro [Github]

Composable-Gen

Projects & Papers

Title & Authors Intro Useful Links
Mixture of Diffusers for scene composition and high resolution image generation
Álvaro Barbero Jiménez
Preprint'23

[Mixture-of-Diffusers (Project)]
[Github]
[Demo]
Compositional Visual Generation with Composable Diffusion Models
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, Joshua B. Tenenbaum
ECCV'22

[Composable Diffusion (Project)]
intro [Github]
[Demo]
[Page]
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, William Yang Wang
ICLR'23

[Structured-Diffusion-Guidance (Project)]
[Github]
[Page]
Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC
Yilun Du, Conor Durkan, Robin Strudel, Joshua B. Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, Will Grathwohl
Preprint'23

[reduce_reuse_recycle (Project)]
intro [Github]
[Page]
Learning to Compose Visual Relations
Nan Liu, Shuang Li, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba
NeurIPS'21

[compose-visual-relations (Project)]
intro [Github]
[Page]

Composable-Agent

Paper Only

  • MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, Sergey Levine NeurIPS'19 [Paper]
  • Composable Planning with Attributes Amy Zhang, Sainbayar Sukhbaatar, Adam Lerer, Arthur Szlam, Rob Fergus ICML'18 [Paper]

Composable-Product

Title & Authors Intro Useful Links
langchain (Project)
[LLM + X]
intro Github

Composable-X

TO BE UPDATE