problems while reproducing results on FB15k
Closed this issue · 1 comments
thjashin commented
The procedure I follow is
First git clone the repo.
Under the FB15k directory
- python FB15k_TransE.py
- python FB15k_evaluation.py FB15k_TransE/best_valid_model.pkl
The result I get is
Using gpu device 0: Tesla K40m
/home/cc/jxshi/env/local/lib/python2.7/site-packages/theano/tensor/subtensor.py:114: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
stop in [None, length, maxsize] or
### MICRO:
-- left >> mean: 2872.72939, median: 682.0, hits@10: 11.131%
-- right >> mean: 2362.83263, median: 399.0, hits@10: 15.427%
-- global >> mean: 2617.78101, median: 533.0, hits@10: 13.279%
### MACRO:
-- left >> mean: 3811.39395, median: 3553.36108, hits@10: 6.763%
-- right >> mean: 3343.12054, median: 3118.40843, hits@10: 10.424%
-- global >> mean: 3577.25725, median: 2691.94641, hits@10: 8.594%
The training log is
Using gpu device 0: Tesla K40m
DD{'ndim': 50, 'test_all': 10, 'loadmodelBi': False, 'loadmodelTri': False, 'nhid': 50, 'lremb': 0.01, 'savepath': 'FB15k_TransE', 'seed': 123, 'marge': 1.0, 'simfn': 'L1', 'neval': 1000, 'dataset': 'FB15k', 'nbatches': 100, 'lrparam': 1.0, 'loademb': False, 'datapath': '../data/', 'Nrel': 1345, 'totepochs': 500, 'rhoL': 5, 'Nent': 16296, 'Nsyn': 14951, 'loadmodel': False, 'rhoE': 1, 'op': 'TransE'}
/home/cc/jxshi/env/local/lib/python2.7/site-packages/theano/tensor/subtensor.py:114: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
stop in [None, length, maxsize] or
batchsize: 4831
nbatches: 100
BEGIN TRAINING
-- EPOCH 10 (14.0093 seconds per epoch):
COST >> 0.7421 +/- 0.2195, % updates: 32.669%
MEAN RANK >> valid: 2618.799, train: 1588.12
##### NEW BEST VALID >> test: 2506.168
(the evaluation took 62.386 seconds)
-- EPOCH 20 (14.357 seconds per epoch):
COST >> 0.5985 +/- 0.0286, % updates: 28.821%
MEAN RANK >> valid: 2573.779, train: 1479.3145
##### NEW BEST VALID >> test: 2419.6415
(the evaluation took 69.338 seconds)
-- EPOCH 30 (14.5237 seconds per epoch):
COST >> 0.59 +/- 0.0283, % updates: 28.66%
MEAN RANK >> valid: 2596.0705, train: 1507.8925
(the evaluation took 41.674 seconds)
-- EPOCH 40 (13.9154 seconds per epoch):
COST >> 0.5868 +/- 0.028, % updates: 28.612%
MEAN RANK >> valid: 2525.7585, train: 1435.894
##### NEW BEST VALID >> test: 2480.78
(the evaluation took 60.28 seconds)
-- EPOCH 50 (14.035 seconds per epoch):
COST >> 0.5861 +/- 0.0288, % updates: 28.612%
MEAN RANK >> valid: 2544.1805, train: 1451.6185
(the evaluation took 39.961 seconds)
-- EPOCH 60 (13.8575 seconds per epoch):
COST >> 0.5857 +/- 0.0278, % updates: 28.615%
MEAN RANK >> valid: 2580.475, train: 1557.4625
(the evaluation took 41.311 seconds)
-- EPOCH 70 (13.9798 seconds per epoch):
COST >> 0.586 +/- 0.0273, % updates: 28.638%
MEAN RANK >> valid: 2547.1255, train: 1464.591
(the evaluation took 41.477 seconds)
-- EPOCH 80 (13.6954 seconds per epoch):
COST >> 0.585 +/- 0.0284, % updates: 28.604%
MEAN RANK >> valid: 2553.614, train: 1504.0965
(the evaluation took 40.815 seconds)
-- EPOCH 90 (13.5098 seconds per epoch):
COST >> 0.5851 +/- 0.0283, % updates: 28.626%
MEAN RANK >> valid: 2519.132, train: 1569.168
##### NEW BEST VALID >> test: 2399.125
(the evaluation took 61.435 seconds)
-- EPOCH 100 (13.6294 seconds per epoch):
COST >> 0.5851 +/- 0.0271, % updates: 28.613%
MEAN RANK >> valid: 2510.854, train: 1510.5975
##### NEW BEST VALID >> test: 2461.665
(the evaluation took 65.216 seconds)
-- EPOCH 110 (13.6886 seconds per epoch):
COST >> 0.5851 +/- 0.0279, % updates: 28.605%
MEAN RANK >> valid: 2578.334, train: 1477.527
(the evaluation took 44.238 seconds)
-- EPOCH 120 (13.6551 seconds per epoch):
COST >> 0.5854 +/- 0.0277, % updates: 28.615%
MEAN RANK >> valid: 2563.495, train: 1482.032
(the evaluation took 42.621 seconds)
-- EPOCH 130 (13.7041 seconds per epoch):
COST >> 0.5843 +/- 0.0275, % updates: 28.606%
MEAN RANK >> valid: 2549.267, train: 1493.6845
(the evaluation took 42.67 seconds)
-- EPOCH 140 (13.9192 seconds per epoch):
COST >> 0.585 +/- 0.0281, % updates: 28.606%
MEAN RANK >> valid: 2534.716, train: 1474.618
(the evaluation took 40.11 seconds)
-- EPOCH 150 (14.1649 seconds per epoch):
COST >> 0.5853 +/- 0.0277, % updates: 28.616%
MEAN RANK >> valid: 2538.495, train: 1502.6005
(the evaluation took 43.591 seconds)
-- EPOCH 160 (13.7459 seconds per epoch):
COST >> 0.5852 +/- 0.0288, % updates: 28.628%
MEAN RANK >> valid: 2563.7135, train: 1457.409
(the evaluation took 47.238 seconds)
-- EPOCH 170 (13.9505 seconds per epoch):
COST >> 0.585 +/- 0.0284, % updates: 28.627%
MEAN RANK >> valid: 2598.625, train: 1508.668
(the evaluation took 41.62 seconds)
-- EPOCH 180 (13.7503 seconds per epoch):
COST >> 0.5846 +/- 0.028, % updates: 28.617%
MEAN RANK >> valid: 2562.2285, train: 1514.315
(the evaluation took 41.185 seconds)
-- EPOCH 190 (13.7498 seconds per epoch):
COST >> 0.5853 +/- 0.0276, % updates: 28.628%
MEAN RANK >> valid: 2545.985, train: 1508.7025
(the evaluation took 40.715 seconds)
-- EPOCH 200 (13.911 seconds per epoch):
COST >> 0.5846 +/- 0.0275, % updates: 28.616%
MEAN RANK >> valid: 2545.276, train: 1489.0895
(the evaluation took 41.367 seconds)
-- EPOCH 210 (13.6507 seconds per epoch):
COST >> 0.5852 +/- 0.0283, % updates: 28.63%
MEAN RANK >> valid: 2521.616, train: 1501.777
(the evaluation took 46.349 seconds)
-- EPOCH 220 (13.8924 seconds per epoch):
COST >> 0.5853 +/- 0.0279, % updates: 28.636%
MEAN RANK >> valid: 2576.8105, train: 1464.5985
(the evaluation took 40.534 seconds)
-- EPOCH 230 (14.0016 seconds per epoch):
COST >> 0.5851 +/- 0.0277, % updates: 28.633%
MEAN RANK >> valid: 2581.774, train: 1488.3685
(the evaluation took 40.768 seconds)
-- EPOCH 240 (13.6372 seconds per epoch):
COST >> 0.5847 +/- 0.0277, % updates: 28.605%
MEAN RANK >> valid: 2561.3445, train: 1490.669
(the evaluation took 41.671 seconds)
-- EPOCH 250 (13.588 seconds per epoch):
COST >> 0.5846 +/- 0.0275, % updates: 28.603%
MEAN RANK >> valid: 2533.8615, train: 1491.4595
(the evaluation took 40.431 seconds)
-- EPOCH 260 (14.1092 seconds per epoch):
COST >> 0.5852 +/- 0.0281, % updates: 28.629%
MEAN RANK >> valid: 2524.907, train: 1529.538
(the evaluation took 43.785 seconds)
-- EPOCH 270 (13.615 seconds per epoch):
COST >> 0.5856 +/- 0.0279, % updates: 28.649%
MEAN RANK >> valid: 2594.1715, train: 1500.197
(the evaluation took 41.121 seconds)
-- EPOCH 280 (13.6351 seconds per epoch):
COST >> 0.5848 +/- 0.0281, % updates: 28.625%
MEAN RANK >> valid: 2478.206, train: 1473.8335
##### NEW BEST VALID >> test: 2431.339
(the evaluation took 62.283 seconds)
-- EPOCH 290 (14.1048 seconds per epoch):
COST >> 0.5844 +/- 0.028, % updates: 28.604%
MEAN RANK >> valid: 2594.1305, train: 1587.6305
(the evaluation took 42.206 seconds)
-- EPOCH 300 (13.5902 seconds per epoch):
COST >> 0.5852 +/- 0.0278, % updates: 28.629%
MEAN RANK >> valid: 2535.6355, train: 1474.9225
(the evaluation took 42.145 seconds)
-- EPOCH 310 (13.7048 seconds per epoch):
COST >> 0.5853 +/- 0.0281, % updates: 28.633%
MEAN RANK >> valid: 2518.5735, train: 1520.5495
(the evaluation took 40.396 seconds)
-- EPOCH 320 (13.5596 seconds per epoch):
COST >> 0.5847 +/- 0.0285, % updates: 28.611%
MEAN RANK >> valid: 2547.382, train: 1435.3145
(the evaluation took 41.601 seconds)
-- EPOCH 330 (13.8951 seconds per epoch):
COST >> 0.5853 +/- 0.0277, % updates: 28.627%
MEAN RANK >> valid: 2558.76, train: 1495.843
(the evaluation took 40.745 seconds)
-- EPOCH 340 (13.7287 seconds per epoch):
COST >> 0.5851 +/- 0.028, % updates: 28.615%
MEAN RANK >> valid: 2578.2635, train: 1464.18
(the evaluation took 41.701 seconds)
-- EPOCH 350 (13.9722 seconds per epoch):
COST >> 0.5852 +/- 0.028, % updates: 28.644%
MEAN RANK >> valid: 2570.4365, train: 1506.5885
(the evaluation took 40.943 seconds)
-- EPOCH 360 (13.6693 seconds per epoch):
COST >> 0.585 +/- 0.0283, % updates: 28.621%
MEAN RANK >> valid: 2592.5005, train: 1488.669
(the evaluation took 41.489 seconds)
-- EPOCH 370 (14.0562 seconds per epoch):
COST >> 0.5853 +/- 0.0275, % updates: 28.613%
MEAN RANK >> valid: 2527.0815, train: 1510.029
(the evaluation took 41.186 seconds)
-- EPOCH 380 (13.5753 seconds per epoch):
COST >> 0.5851 +/- 0.0278, % updates: 28.608%
MEAN RANK >> valid: 2566.2895, train: 1480.227
(the evaluation took 40.583 seconds)
-- EPOCH 390 (13.662 seconds per epoch):
COST >> 0.5849 +/- 0.0286, % updates: 28.609%
MEAN RANK >> valid: 2506.2965, train: 1561.4965
(the evaluation took 40.525 seconds)
-- EPOCH 400 (13.5609 seconds per epoch):
COST >> 0.5852 +/- 0.0281, % updates: 28.635%
MEAN RANK >> valid: 2606.5325, train: 1474.29
(the evaluation took 41.654 seconds)
-- EPOCH 410 (14.1833 seconds per epoch):
COST >> 0.5856 +/- 0.0287, % updates: 28.63%
MEAN RANK >> valid: 2551.1865, train: 1511.1175
(the evaluation took 41.022 seconds)
-- EPOCH 420 (13.9187 seconds per epoch):
COST >> 0.5854 +/- 0.028, % updates: 28.636%
MEAN RANK >> valid: 2579.4155, train: 1496.488
(the evaluation took 41.552 seconds)
-- EPOCH 430 (13.7581 seconds per epoch):
COST >> 0.5853 +/- 0.0282, % updates: 28.629%
MEAN RANK >> valid: 2539.8345, train: 1487.854
(the evaluation took 40.044 seconds)
-- EPOCH 440 (13.6784 seconds per epoch):
COST >> 0.5851 +/- 0.0274, % updates: 28.629%
MEAN RANK >> valid: 2585.2645, train: 1449.604
(the evaluation took 40.061 seconds)
-- EPOCH 450 (13.9173 seconds per epoch):
COST >> 0.5853 +/- 0.0277, % updates: 28.624%
MEAN RANK >> valid: 2522.25, train: 1470.082
(the evaluation took 40.182 seconds)
-- EPOCH 460 (13.8374 seconds per epoch):
COST >> 0.5849 +/- 0.0279, % updates: 28.637%
MEAN RANK >> valid: 2555.768, train: 1478.828
(the evaluation took 42.134 seconds)
-- EPOCH 470 (13.5488 seconds per epoch):
COST >> 0.5852 +/- 0.0282, % updates: 28.617%
MEAN RANK >> valid: 2569.101, train: 1505.802
(the evaluation took 40.231 seconds)
-- EPOCH 480 (13.9631 seconds per epoch):
COST >> 0.5851 +/- 0.0273, % updates: 28.632%
MEAN RANK >> valid: 2580.9845, train: 1510.754
(the evaluation took 41.682 seconds)
-- EPOCH 490 (13.75 seconds per epoch):
COST >> 0.5849 +/- 0.0277, % updates: 28.608%
MEAN RANK >> valid: 2596.538, train: 1509.0875
(the evaluation took 39.438 seconds)
-- EPOCH 500 (13.7304 seconds per epoch):
COST >> 0.5853 +/- 0.0282, % updates: 28.623%
MEAN RANK >> valid: 2559.398, train: 1533.861
(the evaluation took 43.392 seconds)
thjashin commented
Solved by set lrparams to 0.01 indead of 1