h_masked for embedding?

Hi @dapowan,

One more clarification question: When we only need to generate the embeddings, we set output_embed=True, and therefore the h_masked is essentially a h that is the embedding output without any masking?

LIMU-BERT-Public/models.py

Line 171 in a355baa

h_masked = self.transformer(input_seqs)

h is for masked sequences (which is h_masked) only when output_embed=False and masked_pos is provided.

LIMU-BERT-Public/models.py

Line 175 in a355baa

masked_pos = masked_pos[:, :, None].expand(-1, -1, h_masked.size(-1))

Thanks!

Yes.