marslanm/Multimodality-Representation-Learning

This repository provides a comprehensive collection of research papers focused on multimodal representation learning, all of which have been cited and discussed in the survey just accepted https://dl.acm.org/doi/abs/10.1145/3617833 .

Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications

Multimodal Deep Learnig based Research

Survey Papers
Task-specific Methods
Pretrainig Approaches
Multimodal Applications (Understanidng, Classification, Generation, Retrieval, Translation)
Multimodal Datasets

Survey

Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications.
Muhammad Arslan Manzoor, Sarah Albarri, Ziting Xian, Zaiqiao Meng, Preslav Nakov, and Shangsong Liang.
[PDF]

Vision-Language Pre-training:Basics, Recent Advances, and Future Trends.[17th Oct, 2022]
Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao.
[PDF]

VLP: A survey on vision-language pre-training.[18th Feb, 2022]
Feilong Chen, Duzhen Zhang, Minglun Han, Xiuyi Chen, Jing Shi, Shuang Xu, and Bo Xu.
[PDF]

A Survey of Vision-Language Pre-Trained Models.[18th Feb, 2022]
Yifan Du, Zikang Liu, Junyi Li, Wayne Xin Zhao.
[PDF]

Vision-and-Language Pretrained Models: A Survey.[15th Apr, 2022]
Siqu Long, Feiqi Cao, Soyeon Caren Han, Haiqin Yang.
[PDF]

Comprehensive reading list for Multimodal Literature
[Github]

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.[28th Jul, 2021]
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, Graham Neubig
[PDF]

Recent Advances and Trends in Multimodal Deep Learning: A Review.[24th May, 2021]
Jabeen Summaira, Xi Li, Amin Muhammad Shoib, Songyuan Li, Jabbar Abdul.
[PDF]

Task-specific-Methods

Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network.[9th Feb, 2021]

Jiayi Ji, Yunpeng Luo, Xiaoshuai Sun, Fuhai Chen, Gen Luo, Yongjian Wu, Yue Gao, Rongrong Ji

Cascaded Recurrent Neural Networks for Hyperspectral Image Classification.[Aug, 2019]

Renlong Hang, Qingshan Liu, Danfeng Hong, Pedram Ghamisi

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.[2015 NIPS]

Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun

Microsoft coco: Common objects in context.[2014 ECCV]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, C. Lawrence Zitnick

Multimodal Deep Learning.[2011 ICML]

Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, Andrew Y. Ng

Extracting and composing robust features with denoising autoencoders.[5th July, 2008]

Pascak Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol

Multi-Gate Attention Network for Image Captioning.[13th Mar, 2021]

WEITAO JIANG, XIYING LI, HAIFENG HU, QIANG LU, AND BOHONG LIU

AMC: Attention guided Multi-modal Correlation Learning for Image Search.[2017 CVPR]

Kan Chen, Trung Bui, Chen Fang, Zhaowen Wang, Ram Nevatia

Video Captioning via Hierarchical Reinforcement Learning.[2018 CVPR]

Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, William Yang Wang

Gaussian Process with Graph Convolutional Kernel for Relational Learning.[14th Aug, 2021]

Jinyuan Fang, Shangsong Liang, Zaiqiao Meng, Qiang Zhang

Multi-Relational Graph Representation Learning with Bayesian Gaussian Process Network.[28th June, 2022]

Guanzheng Chen, Jinyuan Fang, Zaiqiao Meng, Qiang Zhang, Shangsong Liang

Pretraining-Approaches

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction.[5th Jan, 2022]

Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman Mohamed

A Survey of Vision-Language Pre-Trained Models.[18th Feb, 2022]

Yifan Du, Zikang Liu, Junyi Li, Wayne Xin Zhao

Attention is All you Need.[2017 NIPS]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin

VinVL: Revisiting Visual Representations in Vision-Language Models.[2021 CVPR]

Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, Jianfeng Gao

M6: Multi-Modality-to-Multi-Modality Multitask Mega-transformer for Unified Pretraining.[Aug, 2021]

Junyang Lin, Rui Men, An Yang, Chang Zhou, Yichang Zhang, Peng Wang, Jingren Zhou, Jie Tang, Hongxia Yang

AMMU: A survey of transformer-based biomedical pretrained language models.[23th Mar, 2020]

Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, Sivanesan Sangeetha

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning

RoBERTa: A Robustly Optimized BERT Pretraining Approach.[26th Jul, 2019]

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.[11th Oct, 2018]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

BioBERT: a pre-trained biomedical language representation model for biomedical text mining.[10th Sep, 2019]

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, onghyeon Kim, Sunkyu Kim, Chan Ho So, Jaewoo Kang

HateBERT: Retraining BERT for Abusive Language Detection in English.[23th Oct, 2020]

Tommaso Caselli, Valerio Basile, Jelena Mitrovic, Michael Granitzer

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training.[15th Jul, 2020]

Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, XIan-Ling Mao, Heyan Huang, Ming Zhou

Pre-training technique to localize medical BERT and enhance biomedical BERT.[14th May, 2020]

Shoya Wada, Toshihiro Takeda, Shiro Manabe, Shozo Konishi, Jun Kamohara, Yasushi, Matsumura

Don't Stop Pretraining: Adapt Language Models to Domains and Tasks.[23th Apr, 2020]

Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A. Smith

Knowledge Inheritance for Pre-trained Language Models.[28th May, 2021]

Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

Improving Language Understanding by Generative Pre-Training.[2018]

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever

Shuffled-token Detection for Refining Pre-trained RoBERTa

Subhadarshi Panda, Anjali Agrawal, Jeewon Ha, Benjamin Bloch

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.[26th Sep, 2019]

Zhenzhong Lan, Minga Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut

Exploring the limits of transfer learning with a unified text-to-text transformer.[1st Jan, 2020]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu

End-to-End Object Detection with Transformers.[3rd Nov, 2020]

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko

Deformable DETR: Deformable Transformers for End-to-End Object Detection.[8th Oct, 2018]

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai

Unified Vision-Language Pre-Training for Image Captioning and VQA.[2020 AAAI]

Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, Jianfeng Gao

VirTex: Learning Visual Representations From Textual Annotations.[2021 CVPR]

Karan Desai, Justin Johnson

Ernie-vil: Knowledge enhanced vision-language representations through scene graphs.[2021 AAAI]

Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

OSCAR: Object-Semantics Aligned Pre-training for Vision-Language Tasks.[24th Sep, 2020]

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao

Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision.[14th Oct, 2020]

Hao Tan, Mohit Bansal

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models.[2015 ICCV]

Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, Svetlana Lazebnik

Distributed representations of words and phrases and their compositionality.[2013 NIPS]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, Jeff Dean

AllenNLP: A Deep Semantic Natural Language Processing Platform.[20 Mar, 2018]

Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, Luke Zettlemoyer

Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data.[Jul, 2020]

Emily M. Bender, Alexander Koller

Experience Grounds Language.[21th Apr, 2020]

Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, Joseph Turian

Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training?

Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, Abdelrahman Mohamed

Unifying Achitectures

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.[11th Oct, 2018]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

Improving Language Understanding by Generative Pre-Training.[2018]

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever

End-to-End Object Detection with Transformers.[3rd Nov, 2020]

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko

UNITER: UNiversal Image-TExt Representation Learning.[24th Sep, 2020]

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, Jingjing Liu

UNITER: UNiversal Image-TExt Representation Learning.[2021 ICCV]

Ronghang Hu, Amanpreet Singh

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text.[2021 NIPS]

Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, Boqing Gong

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework.[2022 ICML]

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yang

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.[29th Oct, 2019]

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer

Multimodal-Applications

Understanding

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction.[5th Jan, 2022]

Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman Mohamed

Self-Supervised Multimodal Opinion Summarization.[27th May, 2021]

Jinbae lm, Moonki Kim, Hoyeop Lee, Hyunsouk Cho, Sehee Chung

Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training?

Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, Abdelrahman Mohamed

LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding.[29th Dec, 2020]

Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou

Structext: Structured text understanding with multi-modal transformers.[17th Oct, 2021]

Yulin Li, Yuxi Qian, Yuechen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, Errui Ding

ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction.[20th Sep, 2019]

Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, C. V. Jawahar

FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents.[20th Sep, 2019]

Guillaume Jaume, Hazim Kemal Ekenel, Jean-Philippe Thiran

XYLayoutLM: Towards Layout-Aware Multimodal Networks for Visually-Rich Document Understanding.[2022 CVPR]

Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, Liqing Zhang

Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos.[2022 EMNLP]

Nayu Liu, Xian SUn, Hongfeng Yu, Wenkai Zhang, Guangluan Xu

Multimodal Abstractive Summarization for How2 Videos.[19th Jun, 2019]

Shruti Palaskar, Jindrich Libovicky, Spandana Gella, Florian Metze

Vision guided generative pre-trained language models for multimodal abstractive summarization.[6th Sep, 2021]

Tiezheng Yu, Wenliang Dai, Zihan Liu, Pascale Fung

How2: A Large-scale Dataset for Multimodal Language Understanding.[1st Nov, 2018]

Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loic Barrault, Lucia Specia, Florian Metze

wav2vec 2.0: A framework for self-supervised learning of speech representations.[2020 NIPS]

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael Auli

DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization.[11th Dec, 2020]

Shaoshi Ling, Yuzong Liu

LRS3-TED: a large-scale dataset for visual speech recognition.[3rd Sep, 2018]

Triantafyllos Afouras, Joon Son Chung, Andrew Zisserman

Recurrent Neural Network Transducer for Audio-Visual Speech Recognition.[Dec 2019]

Takaki Makino, Hank Liao, Yannis Assael, Brendan Shillingford, Basilio Garcia, Otavio Braga, Olivier Siohan

Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis.[2020 CVPR]

K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, C.V. Jawahar

On the importance of super-Gaussian speech priors for machine-learning based speech enhancement.[28th Nov, 2017]

Robert Rehr, Timo Gerkmann

Active appearance models.[1998 ECCV]

T. F. Cootes, G. J. Edwards, C. J. Taylor

Leveraging category information for single-frame visual sound source separation.[20th Jul, 2021]

Lingyu Zhu, Esa Rahtu

The Sound of Pixels.[2018 ECCV]

Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, Antonio Torralba

Classification

Vqa: Visual question answering.[2015 ICCV]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh

Topic-based content and sentiment analysis of Ebola virus on Twitter and in the news.[1th Jul, 2016]

Erin Hea-Jin Kim, Yoo Kyung Jeong, Yuyong Kim, Keun Young kang, Min Song

On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis.[6th Jul, 2017]

Jose Camacho-Collados, Mohammad Taher Pilehvar

Market strategies used by processed food manufacturers to increase and consolidate their power: a systematic review and document analysis.[26th Jan, 2021]

Benjamin Wood, Owain Williams, Vijaya Nagarajan, Gary Sacks

Swafn: Sentimental words aware fusion network for multimodal sentiment analysis.[2020 COLING]

Minping Chen, Xia Li

Adaptive online event detection in news streams.[15th Dec, 2017]

Linmei Hu, Bin Zhang, Lei Hou, Juanzi Li

Multi-source multimodal data and deep learning for disaster response: A systematic review.[27th Nov, 2021]

Nilani Algiriyage, Raj Prasanna, Kristin Stock, Emma E. H. Doyle, David Johnston

A Survey of Data Representation for Multi-Modality Event Detection and Evolution.[2nd Nov, 2021]

Kejing Xiao, Zhaopeng Qian, Biao Qin.

Crisismmd: Multimodal twitter datasets from natural disasters.[15th Jun, 2018]

Firoj Alam, Ferda Ofli, Muhammad Imran

Multi-modal generative adversarial networks for traffic event detection in smart cities.[1st Sep, 2021]

Qi Chen, WeiWang, Kaizhu Huang, Suparna De, Frans Coenen

Proppy: Organizing the news based on their propagandistic content.[5th Sep, 2019]

Alberto Barron-Cedeno, Israa Jaradat, Giovanni Da San Martino, Preslav Nakov

Fine-grained analysis of propaganda in news article.[Nov 2019]

Giovanni Da San Martino, Seunghak Yu, Alberto Barron-Cedeno, Rostislav Petrov, Preslav Nakov

Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs.[Oct, 2017]

Zhiwei Jin, Juan Cao, Han Guo, Yongdong Zhang, Jiebo Luo

𝖲𝖠𝖥𝖤: Similarity-Aware Multi-modal Fake News Detection.[6th May, 2020]

Xinyi Zhou, Jindi Wu, Reza Zafarani

From Recognition to Cognition: Visual Commonsense Reasoning.[2019 CVPR]

Rowan Zellers, Yonatan Bisk, Ali Farhadi, Yejin Choi

KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for visual commonsense reasoning.[27th Oct, 2021]

Dandan Song, Siyi Ma, Zhanchen Sun, Sicheng Yang, Lejian Liao

LXMERT: Learning Cross-Modality Encoder Representations from Transformers.[20th Aug, 2019]

Hao Tan, Mohit Bansal

Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers.[2 Apr, 2020]

Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, Jianlong Fu

Vision-Language Navigation With Self-Supervised Auxiliary Reasoning Tasks.[2020 CVPR]

Fengda Zhu, Yi Zhu, Xiaojun Chang, Xiaodan Liang

Generation

Recent advances and trends in multimodal deep learning: A review.[24th May, 2021]

Jabeen Summaira, Xi Li, Amin Muhammad Shoib, Songyuan Li, Jabbar Abdul

Vqa: Visual question answering.[2015 ICCV]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh

Microsoft coco: Common objects in context.[2014 ECCV]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, C. Lawrence Zitnick

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.[11th Oct, 2018]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

Distributed Representations of Words and Phrases and their Compositionality.[2013 NIPS]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, Jeff Dean

LRS3-TED: a large-scale dataset for visual speech recognition.[3rd Sep, 2018]

Triantafyllos Afouras, Joon Son Chung, Andrew Zisserman

A lip sync expert is all you need for speech to lip generation in the wild.[12th Oct, 2019]

K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, C.V. Jawahar

Unified Vision-Language Pre-Training for Image Captioning and VQA.[2020 AAAI]

Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, Jianfeng Gao

Show and Tell: A Neural Image Caption Generator.[2015 CVPR]

Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan

SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning.[2017 CVPR]

Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, Tat-Seng Chua

Self-Critical Sequence Training for Image Captioning.[2017 CVPR]

Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, Vaibhava Goel

Visual question answering: A survey of methods and datasets.[Oct, 2017]

Qi WU, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, Anton van den Hengel

How to find a good image-text embedding for remote sensing visual question answering?.[24th Sep, 2021]

Christel Chappuis, Sylvain Lobry, Benjamin Kellenberger, Bertrand Le, Saux, Devis Tuia

An Improved Attention for Visual Question Answering.[2021 CVPR]

Tanzila Rahman, Shih-Han Chou, Leonid Sigal, Giuseppe Carenini

Analyzing Compositionality of Visual Question Answering.[2019 NIPS]

Sanjay Subramanian, Sameer Singh, Matt Gardner

OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge.[2019 CVPR]

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, Roozbeh Mottaghi

MultiBench: Multiscale Benchmarks for Multimodal Representation Learning.[15th Jul, 2021]

Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A. Lee, Yuke Zhu5, Ruslan Salakhutdinov1, Louis-Philippe Morency

Benchmarking Multimodal AutoML for Tabular Data with Text Fields.[4th Nov, 20201]

Xingjian Shi, Jonas Mueller, Nick Erickson, Mu Li, Alexander J. Smola

Multimodal Explanations: Justifying Decisions and Pointing to the Evidence.[2018 CVPR]

Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, Marcus Rohrbach

Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering.[2018 CVPR]

Aishwarya Agrawal, Dhruv Batra, Devi Parikh, Aniruddha Kembhavi

Generative Adversarial Text to Image Synthesis.[2016 ICML]

Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee

The Caltech-UCSD Birds-200-2011 Dataset.

AttnGAN: Fine-Grained Text to Image Generation With Attentional Generative Adversarial Networks.[2018 CVPR]

Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, Xiaodong He

LipSound: Neural Mel-spectrogram Reconstruction for Lip Reading.[15 Sep, 2019]

Leyuan Qu, Cornelius Weber, Stefan Wermter

The Conversation: Deep Audio-Visual Speech Enhancement.[11th Apr, 2018]

Triantafyllos Afouras, Joon Son Chung, Andrew Zisserman

TCD-TIMIT: An Audio-Visual Corpus of Continuous Speech.[5th May, 2015]

Naomi Harte, Eoin Gillen

Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning.[20 Oct, 2017]

WeiPing, KainanPeng, AndrewGibiansk, SercanO. Arık, Ajay Kannan, Sharan Narang

Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions.[15th Apr, 2018]

Jonathan Shen, Ruiming Pang, Ron J. Weiss, Mile Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, Rif A. Saurous, Yannis Agiomvrgiannakis, Yonghui Wu

Vid2speech: Speech reconstruction from silent video.[5th Mar, 2017]

Ariel Ephrat, Shmuel Peleg

Lip2Audspec: Speech Reconstruction from Silent Lip Movements Video.[15th Apr, 2018]

Hassan Akbari, Himani Arora, Liangliang Cao, Nima Mesgarani

Video-Driven Speech Reconstruction using Generative Adversarial Networks.[14th Jun, 2019]

Konstantinos Vougioukas, Pingchuan Ma, Stavros Petridi, Maja Pantic

Retrieval

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks.[2019 NIPS]

Jiasen Lu, Dhruv Batra, Devi Parikh, Stefan Lee

Learning Robust Patient Representations from Multi-modal Electronic Health Records: A Supervised Deep Learning Approach.[2021]

Leman Akoglu, Evimaria Terzi, Xianli Zhang, Buyue Qian, Yang Liu, Xi Chen, Chong Guan, Chen Li

Referring Expression Comprehension: A Survey of Methods and Datasets.[7th Dec, 2020]

Yanyuan QIao, Chaorui Deng, Qi Wu

VL-BERT: Pre-training of Generic Visual-Linguistic Representations.[22th Aug, 2019]

Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai

Clinically Accurate Chest X-Ray Report Generation.[2019 MLHC]

Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, Marzyeh Ghassemi

Translation

Deep Residual Learning for Image Recognition.[2016 CVPR]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

Probing the Need for Visual Context in Multimodal Machine Translation.[20th Mar, 2019]

Ozan Caglayan, Pranava Madhyastha, Lucia Specia, Loic Barrault

Neural Machine Translation by Jointly Learning to Align and Translate.[1st Sep, 2014]

Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio

Multi-modal neural machine translation with deep semantic interactions.[Apr, 2021]

Jinsong Su, Jinchang Chen, Hui Jiang, Chulun Zhou, Huan Lin, Yubin Ge, Qingqiang Wu, Yongxuan Lai

Multimodal-Datasets

Vqa: Visual question answering.**[2015 ICCV]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh

Microsoft coco: Common objects in context.[2014 ECCV]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, C. Lawrence Zitnick

Pre-training technique to localize medical BERT and enhance biomedical BERT.[14th May, 2020]

Shoya Wada, Toshihiro Takeda, Shiro Manabe, Shozo Konishi, Jun Kamohara, Yasushi, Matsumura

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models.[2015 ICCV]

Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, Svetlana Lazebnik

ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction.[20th Sep, 2019]

Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, C. V. Jawahar

FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents.[20th Sep, 2019]

Guillaume Jaume, Hazim Kemal Ekenel, Jean-Philippe Thiran

How2: A Large-scale Dataset for Multimodal Language Understanding.[1st Nov, 2018]

Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loic Barrault, Lucia Specia, Florian Metze

Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis.[2020 CVPR]

K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, C.V. Jawahar

The Sound of Pixels.[2018 ECCV]

Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, Antonio Torralba

Crisismmd: Multimodal twitter datasets from natural disasters.[15th Jun, 2018]

Firoj Alam, Ferda Ofli, Muhammad Imran

From Recognition to Cognition: Visual Commonsense Reasoning.[2019 CVPR]

Rowan Zellers, Yonatan Bisk, Ali Farhadi, Yejin Choi

The Caltech-UCSD Birds-200-2011 Dataset.

Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics.[30th Aug, 2013]

M. Hodosh, P. Young, J. Hockenmaier

Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph.[Jul, 2018]

AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, Louis-Philippe Morency

MIMIC-III, a freely accessible critical care database.[24th May, 2016]

Alistair E.W. Johnson, Tom J. Pollard, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Leo Anthony Celi & Roger G. Mark

Fashion 200K Benchmark

Indoor scene segmentation using a structured light sensor.[Nov 2011]

Nathan Silberman, Rob Fergus

Indoor Segmentation and Support Inference from RGBD Images.[2012 ECCV]

Nathan Silberman, Derek Hoiem, Pushmeet Kohli, Rob Fergus

Good News, Everyone! Context Driven Entity-Aware Captioning for News Images.[2019 CVPR]

Ali Furkan Biten, Lluis Gomez, Marcal Rusinol, Dimosthenis Karatzas

MSR-VTT: A Large Video Description Dataset for Bridging Video and Language.[2016 CVPR]

Jun Xu, Tao Mei, Ting Yao, Yong Rui

Video Question Answering via Gradually Refined Attention over Appearance and Motion.[Oct, 2017]

Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, Yueting Zhuang

TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering.[2017 CVPR]

Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, Gunhee Kim

Multi-Target Embodied Question Answering.[2019 CVPR]

Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L. Berg, Dhruv Batra

VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering.[14th Aug, 2019]

Catalina Cangea, Eugene Belilovsky, Pietro Lio, Aaron Courville

An Analysis of Visual Question Answering Algorithms.[2017 ICCV]

Kushal Kafle, Christopher Kanan

nuScenes: A Multimodal Dataset for Autonomous Driving.[2020 CVPR]

Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, Oscar Beijbom

Automated Flower Classification over a Large Number of Classes.[20th Jan, 2009]

Maria-Elena Nilsback, Andrew Zisserman

MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos.[20th Jun, 2016]

Amir Zadeh, Rowan Zellers, Eli Pincus, Louis-Philippe Morency

Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition.[18th Jan, 2020]

Yuanhang Zhang, Shuang Yang, Jingyun Xiao, Shiguang Shan, Xilin Chen

The MIT Stata Center dataset. [2013]

Maurice Fallon, Hordur Johannsson, Michael Kaess and John J Leonard

data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language.[2022 ICML]

Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, Michael Auli

FLAVA: A Foundational Language and Vision Alignment Model.[2022 CVPR]

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, Douwe Kiela

UC2: Universal Cross-Lingual Cross-Modal Vision-and-Language Pre-Training.[2021 CVPR]

Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, Jingjing Liu

Citation

If you find the listing and survey useful for your work, please cite the paper:

@article{manzoor2023multimodality,
  title={Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications},
  author={Manzoor, Muhammad Arslan and Albarri, Sarah and Xian, Ziting and Meng, Zaiqiao and Nakov, Preslav and Liang, Shangsong},
  journal={arXiv preprint arXiv:2302.00389},
  year={2023}
}