microsoft/MInference
To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
PythonMIT
Stargazers
- abacajsoftware eng building things
- AlvL1225
- atveitMicrosoft
- bys0318Tsinghua University
- caoshiyiUC Berkeley
- Clement25Singapore
- DinghowLepton AI
- fangtaosongUniversity of Chinese Academy of Sciences, NLP LAB
- gklabHangZhou
- HansPolo113
- happierpigUC Berkeley
- hemingkxPolyU
- iofu728Microsoft
- JerryYin777University of Minnesota Twin Cities
- johnnynunezBarcelona
- justusschock@Lightning-AI
- Light-of-Hers
- liyucheng09
- luca-medeirosMedility
- ninjasaid2k
- Pent
- renllMicrosoft
- saadazghourSomewhere in Morocco
- SakitsMIT, EECS
- SCUTlihaoyuTencent
- seshurajup@dolcera
- showgood163
- SiriusNEOTsinghua University
- tchatonLightning.ai | Pytorch Lightning
- Vbansal21None
- xzyaoiETH Zurich / @eth-easl
- yangzhibo-creator
- yuweihaoNational University of Singapore
- zhangkefeng01
- ZiangWu-77
- ZiruiOu