BM-K/KoDiffCSE

There may be an error in the models.py file

yurkoff-mv opened this issue · 6 comments

yurkoff-mv commented

Hello.
In the models.py file, lines 75-83 take only the 'source' part of the input. I assume that in the first case (line 75) the 'z1_masked_input' part should be taken, in the second case (line 80) the 'z2_masked_input' part should be taken. Is it so?

BM-K commented

The key ingredient to get this to work with identical positive pairs is through the use of independently sampled dropout masks for $x_i$ and $x^+_i$. Therefor this code works fine.
The generator receives z1_masked_input and z2_masked_input as an input.

yurkoff-mv commented

It turns out that the encoder is in the train state?
How are negative sentence samples obtained?
And yet, have you tried to freeze all layers of the BERT/RoBERTa model except for the last one and train only it?

BM-K commented
  1. It turns out that the encoder is in the train state?
    -> I didn't understand your question.
  2. How are negative sentence samples obtained?
    -> Negative samples are obtained by in-batch negative sampling method.
  3. And yet, have you tried to freeze all layers of the BERT/RoBERTa model except for the last one and train only it?
    -> Didn't try.
yurkoff-mv commented
  1. A torch model can be in eval and train states. In the train state, Dropout is enabled.
  2. Now I understand.
    Thanks for your replies.
BM-K commented

In the /model/diffcse/processor.py file, lines 129 and 161 set model to training or evaluation states.

yurkoff-mv commented

Yes, now I have seen. Thank you.

  • 👍1