davidmrau/mixture-of-experts

Why is weighted sum calculated in the logarithmic space?

XieWeikai opened this issue · 1 comments

In the implementation of combine in class SparseDispatcher, the code first apply exp(), then calculate weighted sum and finally go back to log space. Why to do that? I think the result is not as same as the original paper.
In the paper, we have
y = sum(G(x) * E(x))
but in your code, I think you calculate
y = log(sum(G(x) * exp(E(X)) ) )
It seems not the same

fixed