Why not directly use Emb(W) as X_0?

Question

Why not directly use Emb(W) as X_0?

leekum2018 opened this issue 2 years ago · 2 comments

leekum2018 commented 2 years ago

Thanks for your nice work. I have a question and have difficulty understanding it, that is, why not directly use $Emb(W)$ as $X_0$, instead, $X_0 = Emb(W)+ N(0, \sigma_0 I)$ in the paper. Looking forward to your reply, thanks!

Answer 1 · 2023-02-20T13:29:01.000Z

+1, I also have this question

Answer 2 · 2023-03-16T21:28:01.000Z

FYI, this was discussed in openreivew. $\sigma_0$ is set to 0.0001 and it becomes spiky Gaussian, and it was empirical choice according to the authors.