/RpBERT

RpBERT: A Text-image Relationship Propagation Based BERT Model for Multimodal NER

Primary LanguagePython

RpBERT

RpBERT: A Text-image Relationship Propagation Based BERT Model for Multimodal NER

Environment

Python packages

  • python==3.7
  • torch==1.2.0
  • gensim==3.8.0
  • numpy==1.18.3
  • torchcrf==1.0.4
  • pytorch-pretrained-bert==0.6.2 (pip install -r requirements.txt)

Configuration

All configuration are listed in /cfgs/. Please verify parameters before running the codes.

Data

datasets/
├── fudan/
│   ├──ner_img/
│   ├── train
│   ├── dev
│   └── test
├── snap/
│   ├──ner_img/
│   ├── train
│   ├── dev
│   └── test
└── bloomberg/
    ├──rel_img/
    └──textimage-data-image

####The MNER datasets format is as follows:

IMGID:94770
RT	O
@ShervinSinatra	O
:	O
Amsterdam	B-ORG
Savage	I-ORG
AF	I-ORG
.	O
http://t.co/btkv2ddE58	O

IMGID:1330809
RT	O
@noisywoman	O
:	O
@davrosz	O
@SamAntixMusic	O
This	O
Peter	B-PER
Dutton	I-PER
?	O
http://t.co/yjPOz2aLEW	O

Pre-trained Models

pretrained/
├── embeddings/
│   ├──en-fasttext-crawl-300d-1M
│   ├──common_characters_large
├── bert-base-uncased/
│   ├── vocab.txt
│   ├── bert_config.json
│   └── pytorch_model.bin
├── bert-large-uncased/
│   ├── vocab.txt
│   ├── bert_config.json
│   └── pytorch_model.bin
└── resnet/
    └──resnet152-b121ed2d.pth

Usage

Training

  • python main.py

Testing

  • python test.py