rmsnorm

There are 6 repositories under rmsnorm topic.

  • bzhangGo/rmsnorm

    Root Mean Square Layer Normalization

    Language:Python2194112

  • knotgrass/Griffin

    Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

    Language:Python9211

  • dtunai/Tri-RMSNorm

    Efficient kernel for RMS normalization with fused operations, includes both forward and backward passes, compatibility with PyTorch.

    Language:Python7102

  • s-chh/PyTorch-Scratch-LLM

    Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.

    Language:Python2200

  • rmgogogo/nano-aigc

    Generative models nano version for fun. No STOA here, nano first.

    Language:Jupyter Notebook0200

  • sushantkumar23/nano-gpt

    Simple character level Transformer

    Language:Jupyter Notebook0100