astramind-ai/Mixture-of-depths

Can we combine this method with moe?

kostum123 opened this issue · 1 comments

kostum123 commented

I wonder if we can combine a mixture of depths and mixture of experts methods in one model?"

GiacomoLeoneMaria commented

In general, there should be no problem