dheerajrajagopal/SelfExplain

XLNet has <cls> in last

sk-s-hub opened this issue · 1 comments

sk-s-hub commented

SelfExplain/model/SE_XLNet.py

Line 118 in ec914ca

phrase_level_activations = phrase_level_activations - self.activation(hidden_state[:,0,:].unsqueeze(1))

I am a bit confused on this line. XLNet has CLS token in last. hidden state first token would capture the first token.

dheerajrajagopal commented

good catch ! We were porting the code from our previous repo and we had accidentally used the roberta-version for this line. The latest commit should resolve this.