/minSLM

Minimal Small Language Model implementation

Primary LanguagePython

Minimal Small Language Model (SLM) implementation

This repo is meant to implement a SLM and incorporate interesting research papers that that can improve the language model in a minimal way.

TODO

  • Speculative Decoding
  • KV-Cache
  • RoPE
  • Infini-Attention