/multimodal-knowledge-graph

A collection of resources on multimodal knowledge graph, including datasets, papers and contests.

mmkg
Multi-modal Knowledge Graph

This repository is a collection of resources on multimodal knowledge graph, including datasets, research papers and contests.

Content

Multimodal Knowledge Graph Completion

Title Venue Year Code
Image-embodied Knowledge Representation Learning IJICA 2016 Github
KG-BERT: BERT for Knowledge Graph Completion arXiv 2019 Github
Multimodal Data Enhanced Representation Learning IJCNN 2019 Github
Multi-modal Multi-relational Feature Aggregation Network for Medical Knowledge Representation Learning ACM Multimedia 2020
Is Visual Context Really Helpful for Knowledge Graph? A Representation Learning Perspective ACM Multimedia 2021 Github
Endowing Language Models with Multimodal Knowledge Graph Representations arXiv 2022 Github
Hyper-node Relational Graph Attention Network for Multi-modal Knowledge Graph Completion TOMM 2022 Github
IMKGA-SM: Interpretable Multimodal Knowledge Graph Answer Prediction via Sequence Modeling arXiv 2022
MMKRL: A robust embedding approach for multi-modal knowledge graph representation learning Appl. Intell. 2022
MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion EMNLP 2022 Github
OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport NIPS 2022 Github
Relation-enhanced Negative Sampling for Multimodal Knowledge Graph Completion ACM Multimedia 2022
MPKGAC: Multimodal Product Attribute Completion in E-commerce WWW (Companion Volume) 2023 Github
LAFA: Multimodal Knowledge Graph Completion with Link Aware Fusion and Aggregation AAAI 2024

Multimodal Entity Aligment

Title Venue Year Code
MMEA: Entity Alignment for Multi-modal Knowledge Graph KSEM 2020 Github
Multi-modal entity alignment in hyperbolic space Neurocomputing 2021
Visual Pivoting for (Unsupervised) Entity Alignment AAAI 2021 Github
Multi-modal Siamese Network for Entity Alignment KDD 2022 Github
MultiJAF: Multi-modal joint entity alignment framework for multi-modal knowledge graph Neurocomputing 2022
Multi-modal Contrastive Representation Learning for Entity Alignment COLING 2022 Github
Attribute-Consistent Knowledge Graph Representation Learning for Multi-Modal Entity Alignment WWW 2023 Github
Leveraging multimodal features for knowledge graph entity alignment based on dynamic self-attention networks Expert Systems with Applications 2023 Github
Probing the Impacts of Visual Context in Multimodal Entity Alignment Data Sci. Eng. 2023
MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid ACM Multimedia 2023 Github
Revisit and Outstrip Entity Alignment: A Perspective of Generative Models arXiv 2023
Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment ISWC 2023 Github
Multi-Modal Knowledge Graph Transformer Framework for Multi-Modal Entity Alignment arXiv 2023 Github
Universal Multi-modal Entity Alignment via Iteratively Fusing Modality Similarity Paths arXiv 2023 Github

Multimodal Entity Linking

Title Venue Year Code
Multimodal Named Entity Disambiguation for Noisy Social Media Posts ACL 2018
Building a Multimodal Entity Linking Dataset From Tweets LREC 2020 Github
Multimodal Entity Linking for Tweets ECIR 2020 Github
Attention-Based Multimodal Entity Linking with High-Quality Images DASFAA 2021
Weibo-MEL, Wikidata-MEL and Richpedia-MEL: Multimodal Entity Linking Benchmark Datasets CCKS 2021 Github
Multimodal Entity Linking: A New Dataset and A Baseline ACM Multimedia 2021 Github
Visual Entity Linking via Multi-modal Learning Data Intell. 2022
WikiDiverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Types ACL 2022 Github
Multimodal Entity Linking with Gated Hierarchical Fusion and Contrastive Training SIGIR 2022 Github
Multimodal Entity Linking with Mixed Fusion Mechanism DASFAA 2022 Github
Multimodal Knowledge Learning for Named Entity Disambiguation EMNLP-findings 2022
AMELI: Enhancing Multimodal Entity Linking with Fine-Grained Attributes arXiv 2023
Multi-Grained Multimodal Interaction Network for Entity Linking KDD 2023 Github
Benchmarking Diverse-Modal Entity Linking with Generative Models ACL-findings 2023
Generative Multimodal Entity Linking arXiv 2023 Github
Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities ICCV 2023 Github
DRIN: Dynamic Relation Interactive Network for Multimodal Entity Linking ACM Multimedia 2023 Github
MMEL: A Joint Learning Framework for Multi-Mention Entity Linking UAI 2023 Github
S2abEL: A Dataset for Entity Linking from Scientific Tables EMNLP 2023 Github
A Dual-way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking AAAI 2024 Github

Multimodal Named Entity Recognition

Title Venue Year Code
Multimodal Named Entity Recognition for Short Social Media Posts NAACL-HLT 2018
Adaptive Co-attention Network for Named Entity Recognition in Tweets AAAI 2018
Visual Attention Model for Name Tagging in Multimodal Social Media ACL 2018
Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition ICDAR 2019
Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer ACL 2020 Github
Object-Aware Multimodal Named Entity Recognition in Social Media Posts With Adversarial Learning IEEE Trans. Multim. 2021
RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER AAAI 2021 Github
Multi-modal Graph Fusion for Named Entity Recognition with Targeted Visual Guidance AAAI 2021 Github
MNRE: A Challenge Multimodal Dataset for Neural Relation Extraction with Visual Evidence in Social Media Posts ICME 2021 Github
PromptMNER: Prompt-Based Entity-Related Visual Clue Extraction and Integration for Multimodal Named Entity Recognition DASFAA 2022
CWI: A multimodal deep learning approach for named entity recognition from social media using character, word and image features Neural Comput. Appl. 2022
MAF: A General Matching and Alignment Framework for Multimodal Named Entity Recognition WSDM 2022 Github
Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction arXiv 2022 Github
ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition NAACL-HLT 2022 Github
CAT-MNER: Multimodal Named Entity Recognition with Knowledge-Refined Cross-Modal Attention ICME 2022
Query Prior Matters: A MRC Framework for Multimodal Named Entity Recognition ACM Multimedia 2022
Learning from Different text-image Pairs: A Relation-enhanced Graph Convolutional Network for Multimodal NER ACM Multimedia 2022 Github
Named Entity and Relation Extraction with Multi-Modal Retrieval EMNLP 2022 Github
Joint Multimodal Entity-Relation Extraction Based on Edge-enhanced Graph Alignment Network and Word-pair Relation Tagging AAAI 2023 Github
A Novel Framework for Multimodal Named Entity Recognition with Multi-level Alignments arXiv 2023
Grounded Multimodal Named Entity Recognition on Social Media ACL 2023 Github
MPMRC-MNER: A Unified MRC framework for Multimodal Named Entity Recognition based Multimodal Prompt CIKM 2023
MGICL: Multi-Grained Interaction Contrastive Learning for Multimodal Named Entity Recognition CIKM 2023
Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction ACM Multimedia 2023
Fine-Grained Multimodal Named Entity Recognition and Grounding with a Generative Framework ACM Multimedia 2023 Github
MCG-MNER: A Multi-Granularity Cross-Modality Generative Framework for Multimodal NER with Instruction ACM Multimedia 2023 Github
Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledge EMNLP-findings 2023 Github
In-context Learning for Few-shot Multimodal Named Entity Recognition EMNLP-findings 2023
ESPVR: Entity Spans Position Visual Regions for Multimodal Named Entity Recognition EMNLP-findings 2023

Multimodal Event Extraction

Title Venue Year Code
Image Enhanced Event Detection in News Articles AAAI 2020 Github
Cross-media Structured Common Space for Multimedia Event Extraction ACL 2020
MERL: Multimodal Event Representation Learning in Heterogeneous Embedding Spaces AAAI 2021
A Chinese Financial Event Extraction Dataset Constructed by Distant Supervision IJCKG 2021 Github
CLIP-Event: Connecting Text and Images with Event Structures CVPR 2022 Github
Multimedia Event Extraction From News With a Unified Contrastive Learning Framework ACM Multimedia 2022 Github