Source code used for the experiments in:
Pasquale Minervini, Thomas Demeester, Tim Rocktäschel, Sebastian Riedel: Adversarial Sets for Regularising Neural Link Predictors, UAI 2017
@inproceedings{DBLP:conf/uai/MinerviniDRR17,
author = {Pasquale Minervini and
Thomas Demeester and
Tim Rockt{\"{a}}schel and
Sebastian Riedel},
title = {Adversarial Sets for Regularising Neural Link Predictors},
booktitle = {Proceedings of the Thirty-Third Conference on Uncertainty in Artificial
Intelligence, {UAI} 2017},
year = {2017},
crossref = {DBLP:conf/uai/2017}
}
@proceedings{DBLP:conf/uai/2017,
editor = {Gal Elidan and
Kristian Kersting and
Alexander T. Ihler},
title = {Proceedings of the Thirty-Third Conference on Uncertainty in Artificial
Intelligence, {UAI} 2017},
publisher = {{AUAI} Press},
year = {2017}
}
Injecting Background Knowledge in Neural Models via Adversarial Set Regularisation.
Usage:
$ python3 setup.py install --user
running install
running bdist_egg
running egg_info
[...]
$ ./bin/kbp-cli.py -h
INFO:kbp-cli.py:Command line: -h
usage: Rule Injection via Adversarial Training [-h] --train TRAIN [--valid VALID] [--test TEST] [--debug] [--debug-scores DEBUG_SCORES [DEBUG_SCORES ...]] [--debug-embeddings DEBUG_EMBEDDINGS]
[--debug-results] [--lr LR] [--initial-accumulator-value INITIAL_ACCUMULATOR_VALUE] [--nb-batches NB_BATCHES] [--nb-epochs NB_EPOCHS] [--model MODEL]
[--similarity SIMILARITY] [--loss LOSS] [--pairwise-loss PAIRWISE_LOSS] [--corrupt-relations] [--margin MARGIN] [--embedding-size EMBEDDING_SIZE]
[--predicate-embedding-size PREDICATE_EMBEDDING_SIZE] [--hidden-size HIDDEN_SIZE] [--unit-cube]
[--all-one-entities ALL_ONE_ENTITIES [ALL_ONE_ENTITIES ...]] [--predicate-l2 PREDICATE_L2] [--predicate-norm PREDICATE_NORM] [--auc] [--seed SEED]
[--clauses CLAUSES] [--sar-weight SAR_WEIGHT] [--sar-similarity SAR_SIMILARITY] [--adv-lr ADV_LR] [--adversary-epochs ADVERSARY_EPOCHS]
[--discriminator-epochs DISCRIMINATOR_EPOCHS] [--adv-weight ADV_WEIGHT] [--adv-margin ADV_MARGIN] [--adv-batch-size ADV_BATCH_SIZE] [--adv-init-ground]
[--adv-ground-samples ADV_GROUND_SAMPLES] [--adv-ground-tol ADV_GROUND_TOL] [--adv-pooling ADV_POOLING] [--subsample-size SUBSAMPLE_SIZE]
[--head-subsample-size HEAD_SUBSAMPLE_SIZE] [--materialize] [--save SAVE]
optional arguments:
-h, --help show this help message and exit
--train TRAIN, -t TRAIN
--valid VALID, -v VALID
--test TEST, -T TEST
--debug, -D Debug flag
--debug-scores DEBUG_SCORES [DEBUG_SCORES ...] List of files containing triples we want to compute the score of
--debug-embeddings DEBUG_EMBEDDINGS
--debug-results Report fine-grained ranking results
--lr LR, -l LR
--initial-accumulator-value INITIAL_ACCUMULATOR_VALUE
--nb-batches NB_BATCHES, -b NB_BATCHES
--nb-epochs NB_EPOCHS, -e NB_EPOCHS
--model MODEL, -m MODEL Model
--similarity SIMILARITY, -s SIMILARITY Similarity function
--loss LOSS Loss function
--pairwise-loss PAIRWISE_LOSS Pairwise loss function
--corrupt-relations Also corrupt the relation of each training triple for generating negative examples
--margin MARGIN, -M MARGIN Margin
--embedding-size EMBEDDING_SIZE, --entity-embedding-size EMBEDDING_SIZE, -k EMBEDDING_SIZE Entity embedding size
--predicate-embedding-size PREDICATE_EMBEDDING_SIZE, -p PREDICATE_EMBEDDING_SIZE Predicate embedding size
--hidden-size HIDDEN_SIZE, -H HIDDEN_SIZE Size of the hidden layer (if necessary, e.g. ER-MLP)
--unit-cube Project all entity embeddings on the unit cube (rather than the unit ball)
--all-one-entities ALL_ONE_ENTITIES [ALL_ONE_ENTITIES ...] Entities with all-one entity embeddings
--predicate-l2 PREDICATE_L2 Weight of the L2 regularization term on the predicate embeddings
--predicate-norm PREDICATE_NORM Norm of the predicate embeddings
--auc, -a Measure the predictive accuracy using AUC-PR and AUC-ROC
--seed SEED, -S SEED Seed for the PRNG
--clauses CLAUSES, -c CLAUSES File containing background knowledge expressed as Horn clauses
--sar-weight SAR_WEIGHT Schema-Aware Regularization, regularizer weight
--sar-similarity SAR_SIMILARITY Schema-Aware Regularization, similarity measure
--adv-lr ADV_LR, -L ADV_LR Adversary learning rate
--adversary-epochs ADVERSARY_EPOCHS Adversary - number of training epochs
--discriminator-epochs DISCRIMINATOR_EPOCHS Discriminator - number of training epochs
--adv-weight ADV_WEIGHT, -W ADV_WEIGHT Adversary weight
--adv-margin ADV_MARGIN Adversary margin
--adv-batch-size ADV_BATCH_SIZE Size of the batch of adversarial examples to use
--adv-init-ground Initialize adversarial embeddings using real entity embeddings
--adv-ground-samples ADV_GROUND_SAMPLES Number of ground samples on which to compute the ground loss
--adv-ground-tol ADV_GROUND_TOL, --adv-ground-tolerance ADV_GROUND_TOL Epsilon-tolerance when calculating the ground loss
--adv-pooling ADV_POOLING Pooling method used for aggregating adversarial losses (sum, mean, max, logsumexp)
--subsample-size SUBSAMPLE_SIZE Fraction of training facts to use during training (e.g. 0.1)
--head-subsample-size HEAD_SUBSAMPLE_SIZE Fraction of training facts to use during training (e.g. 0.1)
--materialize Materialize all facts using clauses and logical inference
--save SAVE Path for saving the serialized model
If the parameter --adv-lr is not specified, the method does not perform any adversarial training -- i.e. it simply trains the Knowledge Graph Embedding models by minimizing a standard pairwise loss in, such as the margin-based ranking loss in [1].
Example - Embedding the WN18 Knowledge Graph using Complex Embeddings:
$ ./bin/kbp-cli.py --train data/wn18/wordnet-mlj12-train.txt --valid data/wn18/wordnet-mlj12-valid.txt --test data/wn18/wordnet-mlj12-test.txt --lr 0.1 --model ComplEx --similarity dot --margin 5 --embedding-size 100 --nb-epochs 100
INFO:kbp-cli.py:Command line: --train data/wn18/wordnet-mlj12-train.txt --valid data/wn18/wordnet-mlj12-valid.txt --test data/wn18/wordnet-mlj12-test.txt --lr 0.1 --model ComplEx --similarity dot --margin 5 --embedding-size 100 --nb-epochs 100
INFO:kbp-cli.py:#Training Triples: 141442, #Validation Triples: 5000, #Test Triples: 5000
INFO:kbp-cli.py:#Entities: 40943 #Predicates: 18
INFO:kbp-cli.py:Samples: 141442, no. batches: 10 -> batch size: 14145
INFO:kbp-cli.py:Epoch: 1/1 Loss: 9.9205 ± 0.0819
INFO:kbp-cli.py:Epoch: 1/1 Fact Loss: 1403175.1875
INFO:kbp-cli.py:Epoch: 2/1 Loss: 9.1740 ± 0.1622
INFO:kbp-cli.py:Epoch: 2/1 Fact Loss: 1297591.9766
INFO:kbp-cli.py:Epoch: 3/1 Loss: 8.3536 ± 0.0911
INFO:kbp-cli.py:Epoch: 3/1 Fact Loss: 1181545.5312
INFO:kbp-cli.py:Epoch: 4/1 Loss: 7.7184 ± 0.0576
INFO:kbp-cli.py:Epoch: 4/1 Fact Loss: 1091707.6953
INFO:kbp-cli.py:Epoch: 5/1 Loss: 7.1793 ± 0.0403
INFO:kbp-cli.py:Epoch: 5/1 Fact Loss: 1015459.9766
INFO:kbp-cli.py:Epoch: 6/1 Loss: 6.7136 ± 0.0240
INFO:kbp-cli.py:Epoch: 6/1 Fact Loss: 949580.1484
INFO:kbp-cli.py:Epoch: 7/1 Loss: 6.3037 ± 0.0203
INFO:kbp-cli.py:Epoch: 7/1 Fact Loss: 891614.3750
INFO:kbp-cli.py:Epoch: 8/1 Loss: 5.9400 ± 0.0131
INFO:kbp-cli.py:Epoch: 8/1 Fact Loss: 840160.1172
INFO:kbp-cli.py:Epoch: 9/1 Loss: 5.6087 ± 0.0192
INFO:kbp-cli.py:Epoch: 9/1 Fact Loss: 793311.0703
INFO:kbp-cli.py:Epoch: 10/1 Loss: 5.3024 ± 0.0215
INFO:kbp-cli.py:Epoch: 10/1 Fact Loss: 749982.7891
[..]
INFO:kbp-cli.py:Epoch: 91/1 Loss: 0.2401 ± 0.0085
INFO:kbp-cli.py:Epoch: 91/1 Fact Loss: 33953.7292
INFO:kbp-cli.py:Epoch: 92/1 Loss: 0.2347 ± 0.0053
INFO:kbp-cli.py:Epoch: 92/1 Fact Loss: 33194.0295
INFO:kbp-cli.py:Epoch: 93/1 Loss: 0.2315 ± 0.0077
INFO:kbp-cli.py:Epoch: 93/1 Fact Loss: 32742.8557
INFO:kbp-cli.py:Epoch: 94/1 Loss: 0.2280 ± 0.0066
INFO:kbp-cli.py:Epoch: 94/1 Fact Loss: 32254.6658
INFO:kbp-cli.py:Epoch: 95/1 Loss: 0.2221 ± 0.0078
INFO:kbp-cli.py:Epoch: 95/1 Fact Loss: 31411.6094
INFO:kbp-cli.py:Epoch: 96/1 Loss: 0.2181 ± 0.0055
INFO:kbp-cli.py:Epoch: 96/1 Fact Loss: 30844.9131
INFO:kbp-cli.py:Epoch: 97/1 Loss: 0.2119 ± 0.0073
INFO:kbp-cli.py:Epoch: 97/1 Fact Loss: 29971.1079
INFO:kbp-cli.py:Epoch: 98/1 Loss: 0.2107 ± 0.0061
INFO:kbp-cli.py:Epoch: 98/1 Fact Loss: 29807.2014
INFO:kbp-cli.py:Epoch: 99/1 Loss: 0.2081 ± 0.0054
INFO:kbp-cli.py:Epoch: 99/1 Fact Loss: 29432.5969
INFO:kbp-cli.py:Epoch: 100/1 Loss: 0.2042 ± 0.0067
INFO:kbp-cli.py:Epoch: 100/1 Fact Loss: 28878.2566
[..]
INFO:inferbeddings.evaluation.base:### MICRO (test filtered):
INFO:inferbeddings.evaluation.base: -- left >> mean: 438.0892, median: 1.0, mrr: 0.857, hits@10: 92.32%
INFO:inferbeddings.evaluation.base: -- right >> mean: 441.4096, median: 1.0, mrr: 0.868, hits@10: 92.38%
INFO:inferbeddings.evaluation.base: -- global >> mean: 439.7494, median: 1.0, mrr: 0.862, hits@10: 92.35%
Example - Embedding the WN18 Knowledge Graph using Translating Embeddings:
$ ./bin/kbp-cli.py --train data/wn18/wordnet-mlj12-train.txt --valid data/wn18/wordnet-mlj12-valid.txt --test data/wn18/wordnet-mlj12-test.txt --lr 0.1 --model TransE --similarity l1 --margin 2 --embedding-size 50 --nb-epochs 1000
INFO:kbp-cli.py:#Training Triples: 141442, #Validation Triples: 5000, #Test Triples: 5000
INFO:kbp-cli.py:#Entities: 40943 #Predicates: 18
INFO:kbp-cli.py:Samples: 141442, no. batches: 10 -> batch size: 14145
INFO:kbp-cli.py:Epoch: 1/1 Loss: 3.3778 ± 0.5889
INFO:kbp-cli.py:Epoch: 1/1 Fact Loss: 477762.2461
INFO:kbp-cli.py:Epoch: 2/1 Loss: 1.3837 ± 0.1561
INFO:kbp-cli.py:Epoch: 2/1 Fact Loss: 195715.7637
INFO:kbp-cli.py:Epoch: 3/1 Loss: 0.5752 ± 0.0353
INFO:kbp-cli.py:Epoch: 3/1 Fact Loss: 81351.6055
INFO:kbp-cli.py:Epoch: 4/1 Loss: 0.2984 ± 0.0071
INFO:kbp-cli.py:Epoch: 4/1 Fact Loss: 42206.5698
INFO:kbp-cli.py:Epoch: 5/1 Loss: 0.1842 ± 0.0028
INFO:kbp-cli.py:Epoch: 5/1 Fact Loss: 26058.0952
INFO:kbp-cli.py:Epoch: 6/1 Loss: 0.1287 ± 0.0017
INFO:kbp-cli.py:Epoch: 6/1 Fact Loss: 18210.4518
INFO:kbp-cli.py:Epoch: 7/1 Loss: 0.0980 ± 0.0023
INFO:kbp-cli.py:Epoch: 7/1 Fact Loss: 13866.0588
INFO:kbp-cli.py:Epoch: 8/1 Loss: 0.0795 ± 0.0034
INFO:kbp-cli.py:Epoch: 8/1 Fact Loss: 11243.2173
INFO:kbp-cli.py:Epoch: 9/1 Loss: 0.0653 ± 0.0019
INFO:kbp-cli.py:Epoch: 9/1 Fact Loss: 9239.1135
INFO:kbp-cli.py:Epoch: 10/1 Loss: 0.0562 ± 0.0026
INFO:kbp-cli.py:Epoch: 10/1 Fact Loss: 7942.8276
[..]
INFO:kbp-cli.py:Epoch: 990/1 Loss: 0.0026 ± 0.0006
INFO:kbp-cli.py:Epoch: 990/1 Fact Loss: 373.8182
INFO:kbp-cli.py:Epoch: 991/1 Loss: 0.0030 ± 0.0007
INFO:kbp-cli.py:Epoch: 991/1 Fact Loss: 419.6469
INFO:kbp-cli.py:Epoch: 992/1 Loss: 0.0025 ± 0.0006
INFO:kbp-cli.py:Epoch: 992/1 Fact Loss: 354.5707
INFO:kbp-cli.py:Epoch: 993/1 Loss: 0.0028 ± 0.0004
INFO:kbp-cli.py:Epoch: 993/1 Fact Loss: 398.7795
INFO:kbp-cli.py:Epoch: 994/1 Loss: 0.0032 ± 0.0006
INFO:kbp-cli.py:Epoch: 994/1 Fact Loss: 450.6929
INFO:kbp-cli.py:Epoch: 995/1 Loss: 0.0028 ± 0.0005
INFO:kbp-cli.py:Epoch: 995/1 Fact Loss: 390.7763
INFO:kbp-cli.py:Epoch: 996/1 Loss: 0.0028 ± 0.0009
INFO:kbp-cli.py:Epoch: 996/1 Fact Loss: 392.9878
INFO:kbp-cli.py:Epoch: 997/1 Loss: 0.0028 ± 0.0005
INFO:kbp-cli.py:Epoch: 997/1 Fact Loss: 391.4912
INFO:kbp-cli.py:Epoch: 998/1 Loss: 0.0026 ± 0.0003
INFO:kbp-cli.py:Epoch: 998/1 Fact Loss: 362.8399
INFO:kbp-cli.py:Epoch: 999/1 Loss: 0.0026 ± 0.0006
INFO:kbp-cli.py:Epoch: 999/1 Fact Loss: 365.5944
INFO:kbp-cli.py:Epoch: 1000/1 Loss: 0.0025 ± 0.0007
INFO:kbp-cli.py:Epoch: 1000/1 Fact Loss: 354.8261
[..]
INFO:inferbeddings.evaluation.base:### MICRO (valid filtered):
INFO:inferbeddings.evaluation.base: -- left >> mean: 472.5754, median: 2.0, mrr: 0.491, hits@10: 94.22%
INFO:inferbeddings.evaluation.base: -- right >> mean: 480.887, median: 2.0, mrr: 0.495, hits@10: 94.26%
INFO:inferbeddings.evaluation.base: -- global >> mean: 476.7312, median: 2.0, mrr: 0.493, hits@10: 94.24%
[..]
INFO:inferbeddings.evaluation.base:### MICRO (test filtered):
INFO:inferbeddings.evaluation.base: -- left >> mean: 399.6008, median: 2.0, mrr: 0.493, hits@10: 94.24%
INFO:inferbeddings.evaluation.base: -- right >> mean: 424.7734, median: 2.0, mrr: 0.493, hits@10: 94.28%
INFO:inferbeddings.evaluation.base: -- global >> mean: 412.1871, median: 2.0, mrr: 0.493, hits@10: 94.26%[1] Bordes, A. et al. - Translating Embeddings for Modeling Multi-relational Data - NIPS 2013