/MABSA-Survey

MABSA-Survey

1. MABSA

1.1 Multimodal Aspect Term Extraction(MATE)/Multimodal Named Entity Recognition(MNER)

1.1.1 Methods Based on Cross-modal Attention

  1. D. Lu, L. Neves, V. Carvalho, N. Zhang, H. Ji, Visual attention model for name tagging in multimodal social media, in: ACL 2018, 2018, pp. 1990– 1999. [PDF]
  2. Q. Zhang, J. Fu, X. Liu, X. Huang, Adaptive co-attention network for named entity recognition in tweets, in: AAAI 2018, volume 32, 2018. [PDF]
  3. O. Arshad, I. Gallo, S. Nawaz, A. Calefati, et al., Aiding intra-text representations with visual context for multimodal named entity recognition, in: ICDAR 2019, IEEE Computer Society, 2019, pp. 337–342. [PDF]
  4. C. Zheng, Z. Wu, T. Wang, Y. Cai, Q. Li, Object-aware multimodal named entity recognition in social media posts with adversarial learning, IEEE Transactions on Multimedia 23 (2020) 2520–2532. [PDF]

1.1.2 Transformer-based methods

  1. J. Yu, J. Jiang, L. Yang, R. Xia, Improving multimodal named entity recognition via entity span detection with unified multimodal transformer, ACL, 2020. [PDF]
  2. L. Sun, J. Wang, K. Zhang, Y. Su, F. Weng, Rpbert: a text-image relation propagation-based bert model for multimodal ner, in: AAAI 2021, volume 35, 2021, pp. 13860–13868. [PDF]
  3. L. Liu, M. Wang, M. Zhang, L. Qing, X. He, Uamner: uncertainty-aware multimodal named entity recognition in social media posts, Applied Intelligence 52 (2022) 4109–4125. [PDF]
  4. X. Wang, J. Ye, Z. Li, J. Tian, Y. Jiang, M. Yan, J. Zhang, Y. Xiao, Cat-mner: multimodal named entity recognition with knowledge-refined cross-modal attention, in: ICME 2022, IEEE, 2022, pp. 1–6. [PDF]
  5. B. Xu, S. Huang, C. Sha, H. Wang, Maf: a general matching and alignment framework for multimodal named entity recognition, in: ACM WSDM 2022, 2022, pp. 1215–1223. [PDF]
  6. B. Zhou, Y. Zhang, K. Song, W. Guo, G. Zhao, H. Wang, X. Yuan, A spanbased multimodal variational autoencoder for semi-supervised multimodal named entity recognition, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 6293–6302. [PDF]
  7. J. Lu, D. Zhang, J. Zhang, P. Zhang, Flat multi-modal interaction transformer for named entity recognition, in: Proceedings of the 29th International Conference on Computational Linguistics, 2022, pp. 2055–2064. [PDF]
  8. X. Chen, N. Zhang, L. Li, S. Deng, C. Tan, C. Xu, F. Huang, L. Si, H. Chen, Hybrid transformer with multi-level fusion for multimodal knowledge graph completion, in: ACM SIGIR 2022, 2022, pp. 904–915. [PDF]
  9. M. Jia, L. Shen, X. Shen, L. Liao, M. Chen, X. He, Z. Chen, J. Li, Mnerqg: An end-to-end mrc framework for multimodal named entity recognition with query grounding, in: AAAI 2023, volume 37, 2023, pp. 8032–8040. [PDF]
  10. A. Guo, X. Zhao, Z. Tan, W. Xiao, Mgicl: multi-grained interaction contrastive learning for multimodal named entity recognition, in: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, 2023, pp. 639–648. [PDF]
  11. S. Cui, J. Cao, X. Cong, J. Sheng, Q. Li, T. Liu, J. Shi, Enhancing multimodal entity and relation extraction with variational information bottleneck, IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024). [PDF]

1.1.3 Methods Involving Graph Neural Networks

  1. D. Zhang, S. Wei, S. Li, H. Wu, Q. Zhu, G. Zhou, Multi-modal graph fusion for named entity recognition with targeted visual guidance, in: AAAI 2021, volume 35, 2021, pp. 14347–14355. [PDF]
  2. F. Zhao, C. Li, Z. Wu, S. Xing, X. Dai, Learning from different text-image pairs: A relation-enhanced graph convolutional network for multimodal ner, in: ACM Multimedia 2022, 2022, pp. 3983–3992. [PDF]
  3. W. Mai, Z. Zhang, K. Li, Y. Xue, F. Li, Dynamic graph construction framework for multimodal named entity recognition in social media, IEEE Transactions on Computational Social Systems (2023). [PDF]

1.1.4 Methods Based on Modality Translation

  1. S. Chen, G. Aguilar, L. Neves, T. Solorio, Can images help recognize entities? a study of the role of images for multimodal ner, in: W-NUT 2021, 2021, pp. 87–96. [PDF]
  2. X. Wang, M. Gui, Y. Jiang, Z. Jia, N. Bach, T. Wang, Z. Huang, K. Tu, Ita: Image-text alignments for multi-modal named entity recognition, in: NAACL-HLT 2022, 2022, pp. 3176–3189. [PDF]
  3. C. Zheng, J. Feng, Y. Cai, X. Wei, Q. Li, Rethinking multimodal entity and relation extraction from a translation point of view, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 6810–6824. [PDF]

1.1.5 Prompt-based methods

  1. X. Wang, J. Tian, M. Gui, Z. Li, J. Ye, M. Yan, Y. Xiao, Promptmner: prompt-based entity-related visual clue extraction and integration for multimodal named entity recognition, in: International Conference on Database Systems for Advanced Applications, Springer, 2022, pp. 297–305. [PDF]
  2. X. Hu, J. Chen, A. Liu, S. Meng, L. Wen, P. S. Yu, Prompt me up: Unleashing the power of alignments for multimodal entity and relation extraction, in: Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 5185–5194. [PDF]
  3. J. Li, H. Li, Z. Pan, D. Sun, J. Wang, W. Zhang, G. Pan, Prompting chatgpt in mner: enhanced multimodal named entity recognition with auxiliary refined knowledge, in: The 2023 Conference on Empirical Methods in Natural Language Processing, 2023 [PDF]

1.2 Multimodal Aspect-based Sentiment Classification(MABSC)

  1. J. Yu, J. Jiang, Adapting bert for target-oriented multimodal sentiment classification, IJCAI, 2019. [PDF]
  2. J. Yu, J. Jiang, R. Xia, Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification, IEEE/ACM Transactions on Audio, Speech, and Language Processing 28 (2019) 429–439. [PDF]
  3. Z. Khan, Y. Fu, Exploiting bert for multimodal target sentiment classification through input space translation, in: ACM Multimedia 2021, 2021, pp. 3034– 3042. [PDF]
  4. Z. Zhang, Z. Wang, X. Li, N. Liu, B. Guo, Z. Yu, Modalnet: an aspect-level sentiment classification model by exploring multimodal data with fusion discriminant attentional network, World Wide Web 24 (2021) 1957–1974. [[PDF]](Modalnet: an aspect-level sentiment classification model by exploring multimodal data with fusion discriminant attentional network)
  5. F. Zhao, Z. Wu, S. Long, X. Dai, S. Huang, J. Chen, Learning from adjectivenoun pairs: A knowledge-enhanced framework for target-oriented multimodal sentiment classification, in: Proceedings of the 29th International Conference on Computational Linguistics, 2022, pp. 6784–6794. [PDF]
  6. H. Yang, Y. Zhao, B. Qin, Face-sensitive image-to-emotional-text crossmodal translation for multimodal aspect-based sentiment analysis, in: EMNLP 2022, 2022, pp. 3324–3335. [PDF]
  7. J. Yu, J. Wang, R. Xia, J. Li, Targeted multimodal sentiment classification based on coarse-to-fine grained image-target matching, in: IJCAI 2022, 2022, pp. 4482–4488. [PDF]
  8. J. Ye, J. Zhou, J. Tian, R. Wang, J. Zhou, T. Gui, Q. Zhang, X. Huang, Sentiment-aware multimodal pre-training for multimodal sentiment analysis, Knowledge-Based Systems 258 (2022) 110021. [PDF]
  9. Y. Yu, D. Zhang, S. Li, Unified multi-modal pre-training for few-shot sentiment analysis with prompt-based learning, in: ACM Multimedia 2022, 2022, pp. 189–198. [PDF]
  10. L. Jia, T. Ma, H. Rong, N. Al-Nabhan, Affective region recognition and fusion network for target-level multimodal sentiment classification, IEEE Transactions on Emerging Topics in Computing (2023). [PDF]
  11. L. Xiao, X. Wu, S. Yang, J. Xu, J. Zhou, L. He, Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis, Information Processing & Management 60 (2023) 103508. [PDF]
  12. Y. Huang, Z. Chen, J. Chen, J. Z. Pan, Z. Yao, W. Zhang, Target-oriented sentiment classification with sequential cross-modal semantic graph, in: International Conference on Artificial Neural Networks, Springer, 2023, pp. 587–599. [PDF]
  13. Q. Wang, H. Xu, Z. Wen, B. Liang, M. Yang, B. Qin, R. Xu, Image-totext conversion and aspect-oriented filtration for multimodal aspect-based sentiment analysis, IEEE Transactions on Affective Computing (2023). [PDF]
  14. J. Yang, M. Xu, Y. Xiao, X. Du, Amifn: Aspect-guided multi-view interactions and fusion network for multimodal aspect-based sentiment analysis, Neurocomputing 573 (2024) 127222. [PDF]
  15. J. Yu, K. Chen, R. Xia, Hierarchical interactive multimodal transformer for aspect-based multimodal sentiment analysis, IEEE Transactions on Affective Computing (2022). [PDF]

1.3 Multimodal Category-based Sentiment Classification(MCBSC)

  1. N. Xu, W. Mao, G. Chen, Multi-interactive memory network for aspect based multimodal sentiment analysis, in: AAAI 2019, volume 33, 2019, pp. 371–378. [PDF]
  2. H. Yang, Y. Zhao, J. Liu, Y. Wu, B. Qin, Macsa: A multimodal aspectcategory sentiment analysis dataset with multimodal fine-grained aligned annotations, arXiv preprint arXiv:2206.13969 (2022). [PDF]
  3. J. Zhou, J. Zhao, J. X. Huang, Q. V. Hu, L. He, Masad: A large-scale dataset for multimodal aspect-based sentiment analysis, Neurocomputing 455 (2021) 47–58. [PDF]

1.4 Multimodal Aspect-Sentiment Pair Extraction (MASPE)

  1. X. Ju, D. Zhang, R. Xiao, J. Li, S. Li, M. Zhang, G. Zhou, Joint multi-modal aspect-sentiment analysis with auxiliary cross-modal relation detection, in: EMNLP 2021, 2021, pp. 4395–4405. [PDF]
  2. Y. Ling, J. Yu, R. Xia, Vision-language pre-training for multimodal aspectbased sentiment analysis, in: ACL 2022, 2022, pp. 2149–2159. [PDF]
  3. L. Yang, J.-C. Na, J. Yu, Cross-modal multitask transformer for end-toend multimodal aspect-based sentiment analysis, Information Processing & Management 59 (2022) 103038. [PDF]
  4. Z. Yu, J. Wang, L.-C. Yu, X. Zhang, Dual-encoder transformers with cross-modal alignment for multimodal aspect-based sentiment analysis, in: AACL-IJNLP 2022, 2022, pp. 414–423. [PDF]
  5. X. Yang, S. Feng, D. Wang, Q. Sun, W. Wu, Y. Zhang, P. Hong, S. Poria, Few-shot joint multimodal aspect-sentiment analysis based on generative multimodal prompt, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), ACL 2023 Findings, 2023, pp. 11575–11589. [PDF]
  6. R. Zhou, W. Guo, X. Liu, S. Yu, Y. Zhang, X. Yuan, AoM: Detecting aspectoriented information for multimodal aspect-based sentiment analysis, in: ACL 2023 Findings, 2023, pp. 8184–8196. [PDF]
  7. F. Zhao, C. Li, Z. Wu, Y. Ouyang, J. Zhang, X. Dai, M2df: Multi-grained multi-curriculum denoising framework for multimodal aspect-based sentiment analysis, in: EMNLP 2023, 2023, pp. 9057–9070. [PDF]
  8. J. Mu, F. Nie, W. Wang, J. Xu, J. Zhang, H. Liu, Mocolnet: A momentum contrastive learning network for multimodal aspect-level sentiment analysis, IEEE Transactions on Knowledge and Data Engineering (2023). [PDF]
  9. X. Yang, S. Feng, D. Wang, Y. Zhang, S. Poria, Few-shot multimodal sentiment analysis based on multimodal probabilistic fusion prompts, in: Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 6045–6053. [PDF]
  10. T. Peng, Z. Li, P. Wang, L. Zhang, H. Zhao, A novel energy based model mechanism for multi-modal aspect-based sentiment analysis, arXiv preprint arXiv:2312.08084 (2023). [PDF]
  11. L. Xiao, X. Wu, J. Xu, W. Li, C. Jin, L. He, Atlantis: Aesthetic-oriented multiple granularities fusion network for joint multimodal aspect-based sentiment analysis, Information Fusion (2024) 102304. [PDF]

1.5 Multimodal Aspect-Category-Sentiment Triple Extraction (MACSTE)

  1. L. Yang, J. Wang, J.-C. Na, J. Yu, Generating paraphrase sentences for multimodal entity-category-sentiment triple extraction, Knowledge-Based Systems 278 (2023) 110823. [PDF]
  2. L. Yang, Z. Wang, Z. Li, J.-C. Na, J. Yu, An empirical study of multimodal entity-based sentiment analysis with chatgpt: Improving in-context learning via entity-aware contrastive learning, Information Processing & Management 61 (2024) 103724. [PDF]

2.MABSA Datasets

2.1 MATE Datasets

Twitter-2015/2017 [Paper]

WikiDiverse [Paper] [Source]

Twitter-GMNER [Paper] [Source]

Twitter-FGMNER [Paper] [Source]

2.2 MFSC Datasets

2.2.1 MABSC Datasets

Twitter-15/17 [Paper] [Source]

2.2.2 MCBSC Datasets

Multi-ZOL [Paper] [Source]

MASAD [Paper] [Source]