Performance gap on nell dataset
Closed this issue · 2 comments
huang0926huang commented
Hello. Thank you for your work. I have a question about the performance. The hardware environment is RTX3090. On nell dataset, the inductive performance of v4 is very poor. I can't figure it out.
nell V2 BEST:[TEST] MRR:0.3906 H@1:0.2989 H@10:0.5690
nell V3 BEST:[TEST] MRR:0.4039 H@1:0.3065 H@10:0.5827
nell V4 BEST:[TEST] MRR:0.1768 H@1:0.1213 H@10:0.2761
The output of nell v4 inductive experiment is as follows:
n_train: 1752 n_valid: 1063 n_test: 1882
0.0005, 1.0000, 0.000398, 16, 5, 5, 20, 0.1472,tanh
0 [VALID] MRR:0.0529 H@1:0.0236 H@10:0.1038 [TEST] MRR:0.0546 H@1:0.0187 H@10:0.1154 [TIME] train:12.9177 inference:9.9888
1 [VALID] MRR:0.1067 H@1:0.0536 H@10:0.2001 [TEST] MRR:0.1085 H@1:0.0439 H@10:0.2668 [TIME] train:25.6865 inference:10.7422
2 [VALID] MRR:0.1485 H@1:0.0877 H@10:0.2543 [TEST] MRR:0.2202 H@1:0.1299 H@10:0.4171 [TIME] train:39.1507 inference:10.6362
3 [VALID] MRR:0.1674 H@1:0.0963 H@10:0.3097 [TEST] MRR:0.3057 H@1:0.2122 H@10:0.4872 [TIME] train:51.4868 inference:9.4114
4 [VALID] MRR:0.1862 H@1:0.1061 H@10:0.3552 [TEST] MRR:0.3245 H@1:0.2298 H@10:0.5028 [TIME] train:63.3683 inference:9.3812
5 [VALID] MRR:0.1999 H@1:0.1107 H@10:0.3968 [TEST] MRR:0.3343 H@1:0.2426 H@10:0.5069 [TIME] train:75.2433 inference:9.4468
6 [VALID] MRR:0.2162 H@1:0.1217 H@10:0.4204 [TEST] MRR:0.3363 H@1:0.2481 H@10:0.5062 [TIME] train:87.1026 inference:9.5270
7 [VALID] MRR:0.2279 H@1:0.1240 H@10:0.4400 [TEST] MRR:0.3371 H@1:0.2488 H@10:0.5062 [TIME] train:98.9746 inference:9.5352
8 [VALID] MRR:0.2396 H@1:0.1378 H@10:0.4458 [TEST] MRR:0.3364 H@1:0.2478 H@10:0.5062 [TIME] train:110.8165 inference:9.4535
9 [VALID] MRR:0.2524 H@1:0.1476 H@10:0.4608 [TEST] MRR:0.3302 H@1:0.2391 H@10:0.5014 [TIME] train:122.7399 inference:9.6338
10 [VALID] MRR:0.2572 H@1:0.1471 H@10:0.4746 [TEST] MRR:0.3192 H@1:0.2294 H@10:0.4855 [TIME] train:134.6518 inference:9.5855
11 [VALID] MRR:0.2810 H@1:0.1845 H@10:0.4781 [TEST] MRR:0.3100 H@1:0.2129 H@10:0.4820 [TIME] train:146.5643 inference:9.6225
12 [VALID] MRR:0.2904 H@1:0.1926 H@10:0.4792 [TEST] MRR:0.2917 H@1:0.1966 H@10:0.4630 [TIME] train:158.4607 inference:9.6154
13 [VALID] MRR:0.2982 H@1:0.2018 H@10:0.4879 [TEST] MRR:0.2807 H@1:0.1938 H@10:0.4406 [TIME] train:170.4131 inference:9.6089
14 [VALID] MRR:0.3035 H@1:0.2070 H@10:0.4879 [TEST] MRR:0.2672 H@1:0.1869 H@10:0.4160 [TIME] train:182.3094 inference:9.3736
15 [VALID] MRR:0.3106 H@1:0.2157 H@10:0.4948 [TEST] MRR:0.2453 H@1:0.1735 H@10:0.3735 [TIME] train:194.2673 inference:9.3673
16 [VALID] MRR:0.3173 H@1:0.2243 H@10:0.5006 [TEST] MRR:0.2345 H@1:0.1704 H@10:0.3490 [TIME] train:206.1473 inference:9.4422
17 [VALID] MRR:0.3207 H@1:0.2272 H@10:0.4983 [TEST] MRR:0.2299 H@1:0.1666 H@10:0.3431 [TIME] train:217.9668 inference:9.3241
18 [VALID] MRR:0.3251 H@1:0.2336 H@10:0.5029 [TEST] MRR:0.2308 H@1:0.1641 H@10:0.3504 [TIME] train:229.8867 inference:9.3705
19 [VALID] MRR:0.3308 H@1:0.2399 H@10:0.4994 [TEST] MRR:0.2269 H@1:0.1652 H@10:0.3366 [TIME] train:241.8563 inference:9.3745
20 [VALID] MRR:0.3314 H@1:0.2388 H@10:0.5104 [TEST] MRR:0.2253 H@1:0.1624 H@10:0.3383 [TIME] train:253.7770 inference:9.4537
21 [VALID] MRR:0.3356 H@1:0.2434 H@10:0.5092 [TEST] MRR:0.2157 H@1:0.1520 H@10:0.3321 [TIME] train:265.5326 inference:9.4283
22 [VALID] MRR:0.3387 H@1:0.2491 H@10:0.5110 [TEST] MRR:0.2175 H@1:0.1517 H@10:0.3338 [TIME] train:277.5382 inference:9.3782
23 [VALID] MRR:0.3444 H@1:0.2566 H@10:0.5156 [TEST] MRR:0.2137 H@1:0.1472 H@10:0.3310 [TIME] train:289.4781 inference:9.4062
25 [VALID] MRR:0.3463 H@1:0.2561 H@10:0.5185 [TEST] MRR:0.2082 H@1:0.1413 H@10:0.3328 [TIME] train:313.3938 inference:10.3590
26 [VALID] MRR:0.3487 H@1:0.2578 H@10:0.5265 [TEST] MRR:0.2012 H@1:0.1330 H@10:0.3262 [TIME] train:326.0800 inference:9.3745
27 [VALID] MRR:0.3528 H@1:0.2670 H@10:0.5248 [TEST] MRR:0.2028 H@1:0.1368 H@10:0.3238 [TIME] train:338.0613 inference:9.4322
30 [VALID] MRR:0.3538 H@1:0.2659 H@10:0.5311 [TEST] MRR:0.1960 H@1:0.1320 H@10:0.3169 [TIME] train:374.4753 inference:9.8614
33 [VALID] MRR:0.3569 H@1:0.2659 H@10:0.5317 [TEST] MRR:0.1993 H@1:0.1348 H@10:0.3269 [TIME] train:410.9307 inference:9.3776
36 [VALID] MRR:0.3585 H@1:0.2670 H@10:0.5358 [TEST] MRR:0.1953 H@1:0.1317 H@10:0.3169 [TIME] train:446.7413 inference:9.3202
37 [VALID] MRR:0.3591 H@1:0.2641 H@10:0.5363 [TEST] MRR:0.1929 H@1:0.1285 H@10:0.3120 [TIME] train:458.6029 inference:9.2554
38 [VALID] MRR:0.3597 H@1:0.2641 H@10:0.5404 [TEST] MRR:0.1909 H@1:0.1282 H@10:0.3075 [TIME] train:470.4032 inference:9.3962
40 [VALID] MRR:0.3629 H@1:0.2693 H@10:0.5369 [TEST] MRR:0.1839 H@1:0.1202 H@10:0.3058 [TIME] train:494.1581 inference:9.3678
42 [VALID] MRR:0.3652 H@1:0.2705 H@10:0.5409 [TEST] MRR:0.1867 H@1:0.1240 H@10:0.3013 [TIME] train:517.8739 inference:9.4638
43 [VALID] MRR:0.3655 H@1:0.2705 H@10:0.5502 [TEST] MRR:0.1829 H@1:0.1192 H@10:0.2982 [TIME] train:529.7134 inference:9.4103
44 [VALID] MRR:0.3674 H@1:0.2722 H@10:0.5479 [TEST] MRR:0.1775 H@1:0.1151 H@10:0.2892 [TIME] train:541.5437 inference:9.3675
46 [VALID] MRR:0.3709 H@1:0.2785 H@10:0.5490 [TEST] MRR:0.1780 H@1:0.1196 H@10:0.2813 [TIME] train:565.4286 inference:9.5164
47 [VALID] MRR:0.3715 H@1:0.2797 H@10:0.5427 [TEST] MRR:0.1797 H@1:0.1216 H@10:0.2813 [TIME] train:577.3102 inference:9.4576
49 [VALID] MRR:0.3756 H@1:0.2814 H@10:0.5519 [TEST] MRR:0.1768 H@1:0.1213 H@10:0.2761 [TIME] train:601.0566 inference:9.3577
BEST:[VALID] MRR:0.3756 H@1:0.2814 H@10:0.5519 [TEST] MRR:0.1768 H@1:0.1213 H@10:0.2761 [TIME] train:601.0566 inference:9.3577
And I would like to ask why most of the results of my experiments have a difference of about 0.03. Such as:
FB15K-237 V2[TEST] MRR:0.4435 (-0.025) H@1:0.3474 (-0.034) H@10:0.6156(-0.013)
nell V2[TEST] MRR:0.3906(0.028) H@1:0.2989 (-0.020) H@10:0.5690(-0.032)
I would appreciate it if you could give me some ideas. Sorry for taking up your time.
yzhangee commented
Hello, thank you for your interest. The NELL dataset is not as stable as the other two. Since we have adjusted the code, the original hyperparameters may not be very suitable. You can try to adjust the hyperparameters by tools like hyperopt.
huang0926huang commented
Thank you for your advice. I will try.