evolutionaryscale/esm

The dimension of sequence logist

wwzll123 opened this issue · 1 comments

wwzll123 commented

First of all thank you for your great work in integrating protein sequence, structure and function into a single model!

I found that the shape of output of sequence regression_head is L×64, which confused me. The size of the sequence vocab defined in constants\esm3.py is 33. Intuitively, the output dimensions of the sequence logistic should be the same as the sequence vocab. Just like in ESM2. According to your description in the pre-print, this output of logistic regression should be used directly to calculate the cross-entropy with the primary sequence. This is obviously strange.
2de5ce1b05e3e86a041bb4e06b65824

In addition, I found the "Xseq" is missing in the section "A.1.4. ESM3 Inputs and Forward Pass".
1724074100243

ebetica commented

We use 64 to improve model throughput. Only 29 tokens are used (section A.1.3 for description). Thanks for pointing out the error in the paper!