[Retraining] Use Liger Kernel to avoid multi-head logits materialization and scale the context length by N times
ByronHsu opened this issue · 1 comments
ByronHsu commented
https://github.com/linkedin/Liger-Kernel/tree/main/examples/medusa
With the implementation of FusedLinearCrossEntropy and other kernels in Liger-Kernel, we are able to effectively reduce the memory while increase the throughput. We are happy to collaborate and integrate with our kernels!
ByronHsu commented