Difference between user and sequence representation
JoaoLages opened this issue · 6 comments
Can anyone explain me why the user and sequence representation are calculated in this way?? seems like the last state of the LSTM is the sequence representation and the rest is the user representation. I'm not following how this works.
Ok, I'm starting to understand. I missed this pad that you do to your input. You mask the input so that the task is kind of like 'predicting the last input token'.
That makes more sense.
However, I am not following why you still mask the input at the end and use the last input representation
While predicting, for me it would make sense if you would use the whole input, non-padded, and the whole output afterwards, not only a portion of it related with the last input token. By that I mean, using the full user_representations variable and not adding any pad.
Ah ok, I think I finally understood. Only user_representations will contain vectors that try to become as clone as the input vector, after passing through the embedding layer. This means that only the last embedding vector of user_representations will actually contain the predictions for the next input token, makes sense.
I will leave this issue open to see if somebody can confirm this.
This sounds about right: the representation at step i encodes all user behaviours up to i.
It'd be cool if we could have a custom hidden layer with that behavior to add more non-linearities and transformations to the model
What's the big advantage over training only one time step at a time? By that means, each i would have a single bakprop