hkproj/mamba-notes

Confusion in discretization process

TokisakiKurumi2001 opened this issue · 1 comments

Hi there. This note is really helpful for me to understand about the Mamba. Thank you for such an amazing job you've done.

However, when looking at the notes, I stumble into a problem trying to the discrete formula.

From you notes, using the Euler Method, we will arrive the final formula as below.

$h_t = \mathbf{\bar{A}}h_{t-1} + \mathbf{\bar{B}}x_{t-1}$.

From that equation, I interpret that the new hidden state depends on the previous hidden state and the previous input x.

However, in the Mamba paper, I notice that the equation is slightly different.

$h_t = \mathbf{\bar{A}}h_{t-1} + \mathbf{\bar{B}}x_t$.

The term that changed here is $x_{t-1}$ become $x_t$.

Does it imply that using different algorithm (ZOH vs Euler) will result in different final equation? In addition, I found this Wikipedia link that states the equation different from the paper.

I'm a newbie here so if there is any mistake, please tell me know. Thank you.

From this blog, it seems that under small $\Delta$, $x_t \approx x_{t+1}$. I think the authors make the substitution here.