Xiangyan Liu*, Rongxue Li*, Wei Ji, Tao Lin
This is an official PyTorch implementation of the paper Towards Robust Multi-modal Reasoning via Model Selection (Preprint). In this work, we:
- formulate the model selection problem in multi-modal reasoning contexts as an initial endeavor.
- introduce
$M^3$ , a model selection framework for multi-modal models with multi-step reasoning, jointly modeling the relationship between samples, selected models, and subtask dependencies. - create a comprehensive dataset called MS-GQA to facilitate the research for the community.
- provide an effective yet efficient model selection solution, with trivial test-time overhead.
The reasoning capabilities of LLM (Large Language Model) are widely acknowledged in recent research, inspiring studies on tool learning and autonomous agents. LLM serves as the "brain" of agent, orchestrating multiple tools for collaborative multi-step task solving. Unlike methods invoking tools like calculators or weather APIs for straightforward tasks, multi-modal agents excel by integrating diverse AI models for complex challenges. However, current multi-modal agents neglect the significance of model selection: they primarily focus on the planning and execution phases, and will only invoke predefined task-specific models for each subtask, making the execution fragile. Meanwhile, other traditional model selection methods are either incompatible with or suboptimal for the multi-modal agent scenarios, due to ignorance of dependencies among subtasks arising by multi-step reasoning.
To this end, we identify the key challenges therein and propose the
The
- networkx==2.8.4
- numpy==1.23.1
- torch==1.13.1+cu117
- torch_geometric==2.3.1
- tqdm==4.64.0
If you want to run MetaGL or MetaGL++, you can use metagl_enviroment.sh
First, you need to download the corresponding data file from the Google Drive link
provided and place it in the specified data folder
. Then, you will need to run code/preprocess.py
to preprocess the computation graph. After that, the commonly used command line for
For simple one-time running with default hyperparameters:
python code/run_m3.py --backbone gat --loss cce --encoder blip --seed 1
For baseline NCF and NCF++:
## NCF
python code/run_ncf.py --loss cce --encoder blip --seed 1
## NCF++
python code/run_ncf++.py --loss cce --encoder blip --program_encoder bert --seed 1
For baseline MetaGL and MetaGL++:
## MetaGL
python code/run_metagl.py --loss cce --encoder blip --seed 1
## MetaGL++
python code/run_metagl.py --loss cce --encoder blip --seed 1 --if_add_program 1
For multiple complex hyper-parameter tuning experiments in
sh code/scripts/run_m3.sh
Note, you can freely change the range of hyperparameter values according to your needs in run_m3.sh
.
We recently found that inherent randomness in certain PyG sub-libraries causes variability in experimental results (see link). This variation does not impact the conclusions in our paper but results in minor fluctuations. If you have better solutions, feel free to reach out anytime 👏.
If you find this repository helpful for your project, please consider citing our work:
@article{liu2023towards,
title={Towards Robust Multi-Modal Reasoning via Model Selection},
author={Liu, Xiangyan and Li, Rongxue and Ji, Wei and Lin, Tao},
journal={arXiv preprint arXiv:2310.08446},
year={2023}
}