kv-cache

Zefan-Cai/KVCache-Factory

Unified KV Cache Compression Methods for Auto-Regressive Models

harleyszhang/llm_note

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.

NVIDIA/kvpress

LLM KV cache compression made easy

therealoliver/Deepdive-llama3-from-scratch

Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.

FMInference/H2O

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

Zefan-Cai/Awesome-LLM-KV-Cache

Awesome-LLM-KV-Cache: A curated list of 📙Awesome LLM KV Cache Papers with Codes.

NVIDIA-Merlin/HierarchicalKV

HierarchicalKV is a part of NVIDIA Merlin and provides hierarchical key-value storage to meet RecSys requirements. The key capability of HierarchicalKV is to store key-value feature-embeddings on high-bandwidth memory (HBM) of GPUs and in host memory. It also can be used as a generic key-value storage.

itsnamgyu/block-transformer

Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)

kddubey/cappr

Completion After Prompt Probability. Make your LLM make a choice

aju22/LLaMA2

This repository contains an implementation of the LLaMA 2 (Large Language Model Meta AI) model, a Generative Pretrained Transformer (GPT) variant. The implementation focuses on the model architecture and the inference process. The code is restructured and heavily commented to facilitate easy understanding of the key parts of the architecture.

hkproj/pytorch-llama-notes

Notes about LLaMA 2 model

DRSY/EasyKV

Easy control for Key-Value Constrained Generative LLM Inference(https://arxiv.org/abs/2402.06262)

phkhanhtrinh23/milliGPT

This a minimal implementation of a GPT model but it has some advanced features such as temperature/ top-k/ top-p sampling, and KV Cache.

DongmingShenDS/Mistral_From_Scratch

Mistral and Mixtral (MoE) from scratch

mehdihosseinimoghadam/AVA-Mistral-7B

Fine-Tuned Mistral 7B Persian Large Language Model LLM / Persian Mistral 7B

reshalfahsi/image-captioning-mobilenet-llama3

Image Captioning With MobileNet-LLaMA 3

s-chh/PyTorch-Scratch-LLM

Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.

glisses/Efficient-Effective-KV-Cache-Replacement-Policy-for-LLMs

SCAC strategy for efficient and effective KV cache eviction in LLMs

jaameypr/keyvalue-caching

Java-based caching solution designed to temporarily store key-value pairs with a specified time-to-live (TTL) duration.

andrewhsugithub/min-llama

my llama3 implementation

burcgokden/PLDR-LLM-with-KVG-cache

Implementation of PLDR-LLM with KV-cache and G-cache in Pytorch for the paper titled "PLDR-LLMs Learn A Generalizable Tensor Operator That Can Replace Its Own Deep Neural Net At Inference"

lamaparbat/EXPRESS_REDIS_CACHING_RATE_LIMIT

EXPRESS REST API CACHING + RATE LIMITING + KV-STORE

prajeshshrestha/Llama-2.0-architecture-and-inference-from-scratch-with-PyTorch