Can't reproduce the results from the paper

[jthomy]

The code is very easy to get running (thank you for that), however, on active_logical_ttb i get 0% validation accuracy in one run and 0.2 the other time, on passive_logical_ttb I have zero and on car_cdr_seq 0.77 (those I only tried one run).

Therefore, I guess the training is either extremely unstable (but it seems to me that there's a lot missing to 100% on all tasks), or there is some unknown issue with the code? Were you able to reproduce the results with this repository?

I noticed a very low loss (between 0.008 and 0.002), in case that's not intended.

[psoulos]

Hi @jthomy, thanks for checking out the code. I verified that there is a bug in the implementation which is fixed as of #2!