evolutionaryscale/esm

Is it possible to map esm3 embedding back to sequence?

johnnytam100 opened this issue · 3 comments

johnnytam100 commented

I want to explore the esm3 space but wondering how to map the modified embedding back to sequence.

prwoolley commented

Not a developer, but it would be easier to map the modified embedding through the language modeling head to get the sequence logits (P(sequence|embeddings)). Mapping backwards (creating the inverse function for ESM3) might not be reliable or deterministic since multiple inputs can embed similarly.

  • 👍1
ebetica commented

@johnnytam100 you would probably have to train your own model to do this, ESM3 does not natively learn any nice manifold in embedding space.

johnnytam100 commented

Just curious to know: how a nice manifold vs not nice manifold would affect the outcome of this embed -> seq task?