Uncertain Knowledge Graphs Embedding is implemented by means of BERT pretrained natural language model. Exploring approximate inference on the embedding space. Considering the commonsense reasoning based on these works.
- python 3.7
- pytorch 1.6.0+cpu
- transformers 4.0.0
- pretrained BERT models, such as 'bert-base-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz", etc.
In data directory, train.tsv is the training data, test.tsv the testing data and val.tsv the validation data.
| dataset | entities | relations | facts | average of confidence | standard deviation | ratio of facts/relations |
|---|---|---|---|---|---|---|
| CN15k | 15,000 | 36 | 234,675 | 0.627 | 0.234 | 15.6 |
| PPI5k | 4,999 | 7 | 271,666 | 0.415 | 0.213 | 54.3 |
The unit of all values on MSE and MAE is 10$^{-2}$. "-" means the corresponding data did not report in literatures.
| dataset | CN15k | PPI5k | ||||
|---|---|---|---|---|---|---|
| metrics | MSE | MAE | epochs | MSE | MAE | epochs |
| URGE | 10.32 | 22.72 | - | 1.44 | 6.00 | - |
| UKGE$_{rect}$ | 8.61 | 19.90 | >100 | 0.95 | 3.79 | >100 |
| UKGE$_{logi}$ | 9.86 | 20.74 | >100 | 0.96 | 4.07 | >100 |
| UKG$_s$E | 7.71 | 21.34 | 19 | 0.98 | 5.98 | 80 |
| UKGE$_{bert}$ | 6.61 | 19.3 | 2 | 0.85 | 5.38 | 75 |
python.exe ./prediction.py
Using Theano backend.
Model: "bert_fine_tuning_UKGE"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
lstm_1 (LSTM) (None, 768) 4721664
_________________________________________________________________
dense_1 (Dense) (None, 1) 769
=================================================================
Total params: 4,722,433
Trainable params: 4,722,433
Non-trainable params: 0
_________________________________________________________________
...
Prediction for (HEAD, ?relation?, TAIL) RANKING:
Head = ' car '
Tail = ' people '
relation candidate 1 = ('hasa', 0.8294004)
relation candidate 2 = ('usedfor', 0.760266) *
relation candidate 3 = ('createdby', 0.65154946)
relation candidate 4 = ('definedas', 0.63723207)
relation candidate 5 = ('atlocation', 0.6002464)
The project is under working. We are very sorry for any trouble to you. You are welcome to put forward your valuable opinions by comments.