/Awesome-LLM

Awesome-LLM: a curated list of Large Language Model

Creative Commons Zero v1.0 UniversalCC0-1.0

Awesome-LLM Awesome

🔥 Large Language Models(LLM) have taken the NLP community the Whole World by storm. Here is a curated list of papers about large language models, especially relating to ChatGPT. It also contains frameworks for LLM training, tools to deploy LLM, courses and tutorials about LLM and all publicly available LLM checkpoints and APIs:

Milestone Papers

Date keywords Institute Paper Publication
2017-06 Transformers Google Attention Is All You Need NeurIPS
2018-06 GPT 1.0 OpenAI Improving Language Understanding by Generative Pre-Training
2018-10 BERT Google BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding NAACL
2019-02 GPT 2.0 OpenAI Language Models are Unsupervised Multitask Learners
2019-09 Megatron-LM NVIDIA Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
2019-10 T5 Google Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer JMLR
2019-10 ZeRO Microsoft ZeRO: Memory Optimizations Toward Training Trillion Parameter Models SC
2020-01 Scaling Law OpenAI Scaling Laws for Neural Language Models
2020-05 GPT 3.0 OpenAI Language models are few-shot learners NeurIPS
2021-01 Switch Transformers Google Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity JMLR
2021-08 Codex OpenAI Evaluating Large Language Models Trained on Code
2021-08 Foundation Models Stanford On the Opportunities and Risks of Foundation Models
2021-09 FLAN Google Finetuned Language Models are Zero-Shot Learners ICLR
2021-10 T0 HuggingFace et al. Multitask Prompted Training Enables Zero-Shot Task Generalization ICLR
2021-12 GLaM Google GLaM: Efficient Scaling of Language Models with Mixture-of-Experts ICML
2021-12 WebGPT OpenAI WebGPT: Improving the Factual Accuracy of Language Models through Web Browsing
2021-12 Retro DeepMind Improving language models by retrieving from trillions of tokens ICML
2021-12 Gopher DeepMind Scaling Language Models: Methods, Analysis & Insights from Training Gopher
2022-01 COT Google Chain-of-Thought Prompting Elicits Reasoning in Large Language Models NeurIPS
2022-01 LaMDA Google LaMDA: Language Models for Dialog Applications
2022-01 Minerva Google Solving Quantitative Reasoning Problems with Language Models NeurIPS
2022-01 Megatron-Turing NLG Microsoft&NVIDIA Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
2022-03 InstructGPT OpenAI Training language models to follow instructions with human feedback
2022-04 PaLM Google PaLM: Scaling Language Modeling with Pathways
2022-04 Chinchilla DeepMind An empirical analysis of compute-optimal large language model training NeurIPS
2022-05 OPT Meta OPT: Open Pre-trained Transformer Language Models
2022-06 Emergent Abilities Google Emergent Abilities of Large Language Models TMLR
2022-06 BIG-bench Google Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
2022-06 METALM Microsoft Language Models are General-Purpose Interfaces
2022-09 Sparrow DeepMind Improving alignment of dialogue agents via targeted human judgements
2022-10 Flan-T5/PaLM Google Scaling Instruction-Finetuned Language Models
2022-10 GLM-130B Tsinghua GLM-130B: An Open Bilingual Pre-trained Model ICLR
2022-11 HELM Stanford Holistic Evaluation of Language Models
2022-11 BLOOM BigScience BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
2022-11 Galactica Meta Galactica: A Large Language Model for Science
2022-12 OPT-IML Meta OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
2023-01 Flan 2022 Collection Google The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
2023-02 LLaMA Meta LLaMA: Open and Efficient Foundation Language Models
2023-02 Kosmos-1 Microsoft Language Is Not All You Need: Aligning Perception with Language Models
2023-03 PaLM-E Google PaLM-E: An Embodied Multimodal Language Model
2023-03 GPT 4 OpenAI GPT-4 Technical Report

Other Papers

If you're interested in the field of LLM, you may find the above list of milestone papers helpful to explore its history and state-of-the-art. However, each direction of LLM offers a unique set of insights and contributions, which are essential to understanding the field as a whole. For a detailed list of papers in various subfields, please refer to the following link (it is possible that there are overlaps between different subfields):

(:exclamation: We would greatly appreciate and welcome your contribution to the following list. ❗)

  • LLM-Evaluation

    Evaluate different LLMs including ChatGPT in different fields

  • LLM-Acceleration

    Hardware and software acceleration for LLM training and inference

  • LLM-Application

    Use LLM to do some really cool stuff

  • LLM-Augmentation

    Augment LLM in different aspects including faithfulness, expressiveness, domain-specific knowledge etc.

  • LLM-Detection

    Detect LLM-generated text from texts written by humans

  • Chain-of-Thought

    Chain of thought—a series of intermediate reasoning steps—significantly improves the ability of large language models to perform complex reasoning.

  • In-Context-Learning

    Large language models (LLMs) demonstrate an in-context learning (ICL) ability, that is, learning from a few examples in the context.

  • RLHF

    Reinforcement Learning from Human Preference

  • Prompt-Learning

    A Good Prompt is Worth 1,000 Words

  • Instruction-Tuning

    Finetune a language model on a collection of tasks described via instructions

LLM Leaderboard

There are three important steps for a ChatGPT-like LLM:

  1. Pre-training
  2. Instruction Tuning
  3. Alignment

The following list makes sure that all LLMs are compared apples to apples.

Pre-trained LLM

Model Size Architecture Access Date Origin
Switch Transformer 1.6T Decoder(MOE) - 2021-01 Paper
GLaM 1.2T Decoder(MOE) - 2021-12 Paper
PaLM 540B Decoder - 2022-04 Paper
MT-NLG 530B Decoder - 2022-01 Paper
J1-Jumbo 178B Decoder api 2021-08 Paper
OPT 175B Decoder api | ckpt 2022-05 Paper
BLOOM 176B Decoder api | ckpt 2022-11 Paper
GPT 3.0 175B Decoder api 2020-05 Paper
LaMDA 137B Decoder - 2022-01 Paper
GLM 130B Decoder ckpt 2022-10 Paper
YaLM 100B Decoder ckpt 2022-06 Blog
LLaMA 65B Decoder ckpt 2022-09 Paper
GPT-NeoX 20B Decoder ckpt 2022-04 Paper
UL2 20B agnostic ckpt 2022-05 Paper
鹏程.盘古α 13B Decoder ckpt 2021-04 Paper
T5 11B Encoder-Decoder ckpt 2019-10 Paper
CPM-Bee 10B Decoder api 2022-10 Paper
rwkv-4 7B RWKV ckpt 2022-09 Github
GPT-J 6B Decoder ckpt 2022-09 Github
GPT-Neo 2.7B Decoder ckpt 2021-03 Github
GPT-Neo 1.3B Decoder ckpt 2021-03 Github

Instruction finetuned LLM

Model Size Architecture Access Date Origin
Flan-PaLM 540B Decoder - 2022-10 Paper
BLOOMZ 176B Decoder ckpt 2022-11 Paper
InstructGPT 175B Decoder api 2022-03 Paper
Galactica 120B Decoder ckpt 2022-11 Paper
OpenChatKit 20B - ckpt 2023-3 -
Flan-UL2 20B Decoder ckpt 2023-03 Blog
Gopher - - - - -
Chinchilla - - - - -
Flan-T5 11B Encoder-Decoder ckpt 2022-10 Paper
T0 11B Encoder-Decoder ckpt 2021-10 Paper
Alpaca 7B Decoder demo 2023-03 Github

Aligned LLM

Model Size Architecture Access Date Origin
GPT 4 - - - 2023-03 Blog
ChatGPT - Decoder demo|api 2022-11 Blog
Sparrow 70B - - 2022-09 Paper
Claude - - demo|api 2023-03 Blog

LLM Training Frameworks

Serving OPT-175B, BLOOM-176B and CodeGen-16B using Alpa

Alpa is a system for training and serving large-scale neural networks. Scaling neural networks to hundreds of billions of parameters has enabled dramatic breakthroughs such as GPT-3, but training and serving these large-scale neural networks require complicated distributed system techniques. Alpa aims to automate large-scale distributed training and serving with just a few lines of code.

Megatron-LM GPT2 tutorial

DeepSpeed is an easy-to-use deep learning optimization software suite that enables unprecedented scale and speed for DL Training and Inference. Visit us at deepspeed.ai or our Github repo.

pretrain_gpt3_175B.sh

Megatron-LM could be visited here. Megatron (1, 2, and 3) is a large, powerful transformer developed by the Applied Deep Learning Research team at NVIDIA. This repository is for ongoing research on training large transformer language models at scale. We developed efficient, model-parallel (tensor, sequence, and pipeline), and multi-node pre-training of transformer based models such as GPT, BERT, and T5 using mixed precision.

Open source solution replicates ChatGPT training process! Ready to go with only 1.6GB GPU memory and gives you 7.73 times faster training!

Colossal-AI provides a collection of parallel components for you. We aim to support you to write your distributed deep learning models just like how you write your model on your laptop. We provide user-friendly tools to kickstart distributed training and inference in a few lines.

BMTrain is an efficient large model training toolkit that can be used to train large models with tens of billions of parameters. It can train models in a distributed manner while keeping the code as simple as stand-alone training.

Mesh TensorFlow (mtf) is a language for distributed deep learning, capable of specifying a broad class of distributed tensor computations. The purpose of Mesh TensorFlow is to formalize and implement distribution strategies for your computation graph over your hardware/processors. For example: "Split the batch over rows of processors and split the units in the hidden layer across columns of processors." Mesh TensorFlow is implemented as a layer over TensorFlow.

This tutorial discusses parallelism via jax.Array.

Tools for deploying LLM

💙 Haystack

Haystack is an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data. It supports 🔍 Semantic Search, 🤖 Agents, ❓ Question Answering, 📝 Summarization and a range of other applications.

🦜️🔗 LangChain

Large language models (LLMs) are emerging as a transformative technology, enabling developers to build applications that they previously could not. But using these LLMs in isolation is often not enough to create a truly powerful app - the real power comes when you can combine them with other sources of computation or knowledge. This library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include ❓ Question Answering over specific documents, 💬 Chatbots and 🤖 Agents.

👋 wechat-chatgpt

Use ChatGPT On Wechat via wechaty

Tutorials about LLM

  • [Susan Zhang] Open Pretrained Transformers Youtube
  • [Ameet Deshpande] How Does ChatGPT Work? Slides
  • [Yao Fu] 预训练，指令微调，对齐，专业化：论大语言模型能力的来源 Bilibili
  • [Hung-yi Lee] ChatGPT 原理剖析 Youtube
  • [Jay Mody] GPT in 60 Lines of NumPy Link
  • [ICML 2022] Welcome to the "Big Model" Era: Techniques and Systems to Train and Serve Bigger Models Link
  • [NeurIPS 2022] Foundational Robustness of Foundation Models Link
  • [Andrej Karpathy] Let's build GPT: from scratch, in code, spelled out. Video|Code
  • [DAIR.AI] Prompt Engineering Guide Link
  • [邱锡鹏] 大型语言模型的能力分析与应用 Slides | Video
  • [Philipp Schmid] Fine-tune FLAN-T5 XL/XXL using DeepSpeed & Hugging Face Transformers Link
  • [HuggingFace] Illustrating Reinforcement Learning from Human Feedback (RLHF) Link
  • [HuggingFace] What Makes a Dialog Agent Useful? Link
  • [张俊林]通向AGI之路：大型语言模型(LLM)技术精要 Link
  • [大师兄]ChatGPT/InstructGPT详解 Link
  • [HeptaAI]ChatGPT内核：InstructGPT，基于反馈指令的PPO强化学习 Link
  • [Yao Fu] How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources Link
  • [Stephen Wolfram] What Is ChatGPT Doing … and Why Does It Work? Link
  • [Jingfeng Yang] Why did all of the public reproduction of GPT-3 fail? Link
  • [Hung-yi Lee] ChatGPT (可能)是怎麼煉成的 - GPT 社會化的過程 Video

Courses about LLM

  • [Princeton] Understanding Large Language Models Homepage
  • [OpenBMB] 大模型公开课 主页
  • [Stanford] CS224N-Lecture 11: Prompting, Instruction Finetuning, and RLHF Slides
  • [Stanford] CS324-Large Language Models Homepage
  • [Stanford] CS25-Transformers United V2 Homepage
  • [Stanford Webinar] GPT-3 & Beyond Video
  • [李沐] InstructGPT论文精读 Bilibili Youtube
  • [陳縕儂] OpenAI InstructGPT 從人類回饋中學習 ChatGPT 的前身 Youtube
  • [李沐] HELM全面语言模型评测 Bilibili
  • [李沐] GPT，GPT-2，GPT-3 论文精读 Bilibili Youtube
  • [Aston Zhang] Chain of Thought论文 Bilibili Youtube
  • [MIT] Introduction to Data-Centric AI Homepage

Opinions about LLM

Other Awesome Lists

Other Useful Resources

Contributing

This is an active repository and your contributions are always welcome!

I will keep some pull requests open if I'm not sure if they are awesome for LLM, you could vote for them by adding 👍 to them.

If you have any question about this opinionated list, do not hesitate to contact me chengxin1998@stu.pku.edu.cn.