Amazing Resources

List of references and online resources related to data science, machine learning and deep learning.

👍 Courses / Tutorials

👍 Cheat Sheets

👍 AWS / SageMaker

👍 Videos

👍 Books

👍 Papers

2015 Cyclical Learning Rates for Training Neural Networks (https://arxiv.org/abs/1506.01186)
2017 Decoupled Weight Decay Regularization (https://arxiv.org/abs/1711.05101)
2018 Mixed Precision Training (https://arxiv.org/pdf/1710.03740.pdf)
2020 ReadNet: A Hierarchical Transformer Framework for Web Article Readability Analysis (https://link.springer.com/chapter/10.1007/978-3-030-45439-5_3)
2021 A Survey of Transformers (https://arxiv.org/pdf/2106.04554.pdf)
2022 Formal Algorithms for Transformers (https://arxiv.org/pdf/2207.09238.pdf)
2023 Transformer models: an introduction and catalog (https://arxiv.org/pdf/2302.07730.pdf) (http://bit.ly/3YFqRn9)

👍 Articles

👍 Interesting and Fun

👍 GitHub Repositories

👍 Kaggle

👍 DeepNote

👍 Blogs

👍 Company Tech Blogs

👍 Maths

👍 Datasets

👍 Synthetic Data

👍 Utilities / Tools

👍 Job / Interview / DS Portfolio

👍 Salary Negotiation

👍 System Design

👍 Algorithms / Technical Coding

👍 Videos for Algorithms / Technical Coding / Interview Prep

👍 Bit Hacks

👍 Google foobar

🔦 PyTorch-Related

🔦 PyTorch-Related Discussions

⏩ fast.ai

NVIDIA eBooks

Genomic Data Science

🤖 Transformer Architecture / Anatomy / Guide

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration [Paper] (https://github.com/mit-han-lab/llm-awq)
AutoAWQ (https://github.com/casper-hansen/AutoAWQ)
Microsoft DeepSpeed - Deep learning optimization software suite for both training and inference (https://github.com/microsoft/DeepSpeed) (https://www.deepspeed.ai/)
DeepSpeed Chat - Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales (https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/DeepSpeed-Chat) (https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat)
DeepSpeed's Bag of Tricks for Speed & Scale (https://www.kolaayonrinde.com/blog/2023/07/14/deepspeed-train.html)
🤗 PEFT: Parameter-Efficient Fine-Tuning of Billion-Scale Models on Low-Resource Hardware (https://huggingface.co/blog/peft) (https://github.com/huggingface/peft)
🤗 TRL - Transformer Reinforcement Learning (https://github.com/huggingface/trl) (https://huggingface.co/docs/trl/index)
bitsandbytes - 8-bit optimizers, matrix multiplication (LLM.int8()), and quantization functions (https://github.com/TimDettmers/bitsandbytes)
GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers (https://github.com/IST-DASLab/gptq)
AutoGPTQ: LLMs quantization package with user-friendly apis, based on GPTQ algorithm (https://github.com/PanQiWei/AutoGPTQ)
Quantize 🤗 Transformers models (https://huggingface.co/docs/transformers/main_classes/quantization)
Which Quantization Method is Right for You? (GPTQ vs. GGUF vs. AWQ) (https://maartengrootendorst.substack.com/p/which-quantization-method-is-right)
Optimum-Benchmark 🏋️ (https://github.com/huggingface/optimum-benchmark)
NEFTune - add random noise to the embedding vectors of the training data during the forward pass of fine-tuning (https://github.com/neelsjain/NEFTune)
LoRA+: Efficient Low Rank Adaptation of Large Models (https://github.com/nikhil-ghosh-berkeley/loraplus)
DoRA: Weight-Decomposed Low-Rank Adaptation (https://github.com/catid/dora/tree/main)
tensor_parallel - much faster than huggingface's device_map and lightweight than vLLM? (https://github.com/BlackSamorez/tensor_parallel)
Nanotron - Minimalistic large language model 3D-parallelism training (https://github.com/huggingface/nanotron/)
FastChat - platform for training, serving, and evaluating large language model based chatbots (https://github.com/lm-sys/FastChat)
Half-Quadratic Quantization (HQQ) (https://mobiusml.github.io/hqq_blog/) (https://github.com/mobiusml/hqq)
AQLM - Extreme Compression of Large Language Models via Additive Quantization (https://github.com/Vahe1994/AQLM) (https://towardsdatascience.com/the-aqlm-quantization-algorithm-explained-8cf33e4a783e)
torchtune - A Native-PyTorch Library for LLM Fine-tuning (https://github.com/pytorch/torchtune)
Prefect - Modern workflow orchestration for data and ML engineers (https://www.prefect.io/)
Modal - serverless platform to run generative AI models, large-scale batch jobs, job queues, etc (https://modal.com/)
Beating Proprietary Models with a Quick Fine-Tune - Finetuning Quora Embeddings (https://modal.com/blog/fine-tuning-embeddings) (https://github.com/567-labs/fastllm/blob/main/applications/finetune-quora-embeddings/Readme.md)
Fine-tune an LLM in minutes (ft. Llama 2, CodeLlama, Mistral, etc.) (https://modal.com/docs/examples/llm-finetuning) (https://github.com/modal-labs/llm-finetuning)

FastChat - platform for training, serving, and evaluating large language model based chatbots (https://github.com/lm-sys/FastChat)
LLM Evaluation Metrics: Everything You Need for LLM Evaluation (https://www.confident-ai.com/blog/llm-evaluation-metrics-everything-you-need-for-llm-evaluation)
DeepEval - open-source LLM evaluation framework specialized for unit testing LLM outputs (https://github.com/confident-ai/deepeval)
LLM Task-Specific Evals that Do & Don't Work (https://eugeneyan.com/writing/evals/)

7 Ways To Speed Up Inference of Your Hosted LLMs (https://betterprogramming.pub/speed-up-llm-inference-83653aa24c47)
vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (https://github.com/vllm-project/vllm) (https://vllm.ai/)
🤗 TGI: Text Generation Inference - Fast optimized inference for LLMs (https://github.com/huggingface/text-generation-inference)
LMDeploy: a toolkit for compressing, deploying, and serving LLM (https://github.com/InternLM/lmdeploy)
OpenVINO: an open-source toolkit for optimizing and deploying AI inference (https://github.com/openvinotoolkit) (https://docs.openvino.ai/2023.0/home.html)
How continuous batching enables 23x throughput in LLM inference while reducing p50 latency (https://www.anyscale.com/blog/continuous-batching-llm-inference)
Squeeze more out of your GPU for LLM inference—a tutorial on Accelerate & DeepSpeed (https://preemo.medium.com/squeeze-more-out-of-your-gpu-for-llm-inference-a-tutorial-on-accelerate-deepspeed-610fce3025fd)
Performance bottlenecks in deploying LLMs—a primer for ML researchers (https://preemo.medium.com/performance-bottlenecks-in-deploying-llms-a-primer-for-ml-researchers-c2b51c2084a8)
Inference using the pre-trained Alpaca-LoRA (https://www.mlexpert.io/machine-learning/tutorials/alpaca-and-llama-inference) (https://colab.research.google.com/drive/15VstUxU48CT3mRudFrj3FIv6Z4cIXnon?usp=sharing)
Optimizing your LLM in production (https://huggingface.co/blog/optimize-llm)
StreamingLLM: Efficient Streaming Language Models with Attention Sinks (https://github.com/mit-han-lab/streaming-llm) (https://arxiv.org/pdf/2309.17453.pdf)
S-LoRA: Serving Thousands of Concurrent LoRA Adapters (https://github.com/s-lora/s-lora)
Recipe for Serving Thousands of Concurrent LoRA Adapters (https://lmsys.org/blog/2023-11-15-slora/)
DeepSparse by Neural Magic - Sparsity-aware deep learning inference runtime for CPUs (https://github.com/neuralmagic/deepsparse/tree/main)
SparseML by Neural Magic - an open-source model optimization toolkit that enables you to create inference-optimized sparse models using pruning, quantization, and distillation algorithms (https://github.com/neuralmagic/sparseml)
Marlin - Mixed Auto-Regressive Linear kernel, an extremely optimized FP16xINT4 matmul kernel aimed at LLM inference (https://github.com/IST-DASLab/marlin)
LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning (https://github.com/datamllab/LongLM) (https://www.reddit.com/r/LocalLLaMA/comments/18x8g6c/llm_maybe_longlm_selfextend_llm_context_window/)
Flash-Decoding for long-context inference (https://pytorch.org/blog/flash-decoding/)
10 Ways To Run LLMs Locally And Which One Works Best For You (https://matilabs.ai/2024/02/07/run-llms-locally/)
Towards 100x Speedup: Full Stack Transformer Inference Optimization (https://yaofu.notion.site/Towards-100x-Speedup-Full-Stack-Transformer-Inference-Optimization-43124c3688e14cffaf2f1d6cbdf26c6c)
Deploy Deep Learning Models at Scale using NVIDIA Triton Inference Server (https://github.com/decodingml/articles-code/tree/main/articles/computer_vision/deploy_deep_learning_at_scale_nvidia_triton_server)
LMDeploy - a toolkit for compressing, deploying, and serving LLM (https://github.com/InternLM/lmdeploy)
LLM Inference Series: 5. Dissecting model performance (https://medium.com/@plienhar/llm-inference-series-5-dissecting-model-performance-6144aa93168f)
How to compute LLM embeddings 3X faster with model quantization - with ONNX model quantization / ONNX transformer optimization (https://medium.com/nixiesearch/how-to-compute-llm-embeddings-3x-faster-with-model-quantization-25523d9b4ce5)

Curating Trillion-Token Datasets: Introducing NVIDIA NeMo Data Curator (https://developer.nvidia.com/blog/curating-trillion-token-datasets-introducing-nemo-data-curator/)
ftfy: fixes text for you (https://github.com/rspeer/python-ftfy)
Cosmopedia: how to create large-scale synthetic data for pre-training (https://huggingface.co/blog/cosmopedia) (https://github.com/huggingface/cosmopedia)
DataTrove - a library to process, filter and deduplicate text data at a very large scale (https://github.com/huggingface/datatrove/)

Retentive Networks (RetNet) Explained: The much-awaited Transformers-killer is here (https://medium.com/ai-fusion-labs/retentive-networks-retnet-explained-the-much-awaited-transformers-killer-is-here-6c17e3e8add8)
Mamba Explained - The State Space Model taking on Transformers (https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html)
Introducing Jamba: AI21's Groundbreaking SSM-Transformer Model (https://www.ai21.com/blog/announcing-jamba)
ModuleFormer: a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts (https://github.com/IBM/ModuleFormer)
JetMoE: Reaching LLaMA2 Performance with 0.1M Dollars (https://research.myshell.ai/jetmoe)

gsanou/Amazing-Resources

Amazing Resources

👍 Courses / Tutorials

👍 Cheat Sheets

👍 AWS / SageMaker

👍 Videos

👍 Books

👍 Papers

👍 Articles

CNN

RNN

NLP

NVIDIA Recommender Systems

Collaborative Filtering / Recommender Systems

Similarity Search / ANNS / Vector Indexing

Search / Code Search / Information Retrieval

Search Ranking

Lucene / Solr / Elasticsearch

General ML/DL Articles

Time Series

EDA / Data Visualization

Colab

Python

Database / Storage

Reinforcement Learning

👍 Interesting and Fun

👍 GitHub Repositories

👍 Kaggle

👍 DeepNote

👍 Blogs

👍 Company Tech Blogs

👍 Maths

👍 Datasets

👍 Synthetic Data

👍 Utilities / Tools

👍 Job / Interview / DS Portfolio

👍 Salary Negotiation

👍 System Design

👍 Algorithms / Technical Coding

👍 Videos for Algorithms / Technical Coding / Interview Prep

👍 Bit Hacks

👍 Google foobar

🔦 PyTorch-Related

🔦 PyTorch-Related Discussions

⏩ fast.ai

NVIDIA eBooks

Genomic Data Science

🤖 Transformer Architecture / Anatomy / Guide

🤖 Transformer / Attention / LLM Visualization

🤖 Transformer Libraries

🤖 Transformer Toolkit / Techniques / Methods

🤖 RAG

🤖 Lightning AI ⚡⚡⚡

🤖 LLM Leaderboard

🤖 LLM Evaluation

🤖 Transformer Models / Timeline

🤖 Transformer / LLM Inference / Deployment

🤖 Transformer / LLM Data Curator

🤖 Transformer / LLM Sample Applications

🤖 Q&A

Discussion on LLM Padding / Formatting Function

Merging weights with quantized model

Merge / Fusion / MoE

Prompt Engineering / Instructions

Agent

LLM - Misc

Unsloth

Transformer Alternatives