Report of empirical evaluation, observations and investigations on LSTM based Language Modeling
| Model |
Val. PPL |
Test PPL |
Last Epoch |
Last LR |
Seq_len |
Tokenizer |
| HLSTM |
1.826 |
1.779 |
64 |
0.007813 |
25 |
moses |
| HLSTM |
1.952 |
1.907 |
143 |
2.98E-08 |
25 |
Split |
| HLSTM |
2.38 |
2.296 |
155 |
3.81E-06 |
35 |
moses |
| HLSTM |
2.847 |
2.782 |
113 |
0.00048828125 |
35 |
Basic_English |
| HLSTM |
21.98 |
20.39 |
139 |
2.98E-08 |
50 |
moses |
| HLSTM |
47.19 |
45.49 |
152 |
2.38E-07 |
50 |
Basic_English |
| HLSTM |
51.75 |
49.58 |
151 |
2.33E-10 |
50 |
Split |
| HLSTM |
68.75 |
63.34 |
122 |
3.81E-06 |
70 |
moses |
| Model |
Val. PPL |
Test PPL |
Last Epoch |
Last LR |
Seq_len |
Tokenizer |
| H2HLSTM |
3.069 |
2.987 |
145 |
4.66E-10 |
25 |
Split |
| H2HLSTM |
4.085 |
3.855 |
117 |
1.91E-06 |
35 |
moses |
| H2HLSTM-NTASGD |
32.32 |
31.01 |
108 |
9.53E-07 |
50 |
Basic_English |
| H2HLSTM |
34.8 |
33.59 |
97 |
6.10E-05 |
50 |
Basic_English |
| H2HLSTM |
38.15 |
35.18 |
88 |
2.98E-08 |
50 |
moses |
| H2HLSTM |
67.48 |
64.29 |
99 |
2.29E-05 |
50 |
Split |
| Model |
Val. PPL |
Test PPL |
Last Epoch |
Last LR |
Seq_len |
Tokenizer |
| HLSTM |
1.885 |
1.942 |
12(24h) |
0.5 |
25 |
Split |
| Model |
Val. PPL |
Test PPL |
Last Epoch |
Last LR |
Seq_len |
Tokenizer |
| HLSTM |
1.964 |
1.782 |
149 |
2.91E-11 |
25 |
Split |
| HLSTM |
2.651 |
2.651 |
2.355 |
1.49E-08 |
21 |
Basic_English |
| H2HLSTM |
3.94 |
3.44 |
123 |
1.19E-07 |
21 |
Moses |
| HLSTM |
87.17 |
79.64 |
78 |
3.81E-06 |
50 |
Basic_English |
| LSTM |
87.75 |
74.87 |
66 |
1.90E-06 |
21 |
Moses |