YandongJi/LLM4Academic

Theory and Practice about LLMs

LLM4Academic

LLM4Academic is a repository for everything I want to know about large language models (LLMs). There are two parts in this repository: (1) Theory: reading list, survey, curated sources; (2) Practice: insightful experiment (demo, framework, etc.) implemented by myself.

"In theory, theory and practice are the same. In practice, they are not."

Table of Contents

• Theory

  • Reading List
  • Courses
  • Dataset Collections
  • Open Source LLM
  • Evaluation Benchmarks

• Practice

  • API
  • Instruction Tuning
  • Constrained Generation
  • Reference

Theory

Reading List

Reading list and related notes for LLM, see Reading List for details.

• Key Findings

• Architecture
• Efficiency
• Instruction Tuning
• In Context Learning
• Mixture of Experts
• Reasoning
  • Abstract Reasoning
  • Chain of Thought
• Reference (survey, lists, and etc.)

Courses

• Johns Hopkins University: CS 601.x71 NLP: Self-supervised Models Spring 2023/Fall 2022
• Princeton University: COS 597G: Understanding Large Language Models Fall 2022
• Stanford University: CS25: Transformers United V2 Fall 2021/Winter 2023
• Stanford University: CS 324 - Advances in Foundation Models Winter 2022/Winter 2023

Dataset Collections

Datasets for Pretrain/Finetune/Instruction-tune LLMs, see Datasets for details.

• Pretraining Corpora
• Instruction

Open Source LLMs

Collection of various open-source LLMs, see Open Souce LLMs for details.

• Pretrained Model
• Multitask Supervised Finetuned Model
• Instruction Finetuned Model
  • English
  • Chinese
  • Multilingual
• Human Feedback Finetuned Model
• Domain Finetuned Model
• Open Source Projects
  • reproduce/framework
  • accelerate
  • evaluation
  • deployment/demo
• Reference

Evaluation Benchmarks

Collection of automatic evaluation benchmarks, see Evaluation Benchmarks for details.

• English
  • Comprehensive
  • Knowledge
  • Reason
    • Hard Mathematical, Theorem
  • Code
  • Personalization
• Chinese
  • Comprehensive
  • Safety
• Multilingual

Practice

API

LLM API demos, see API for details.

• Claude
• ChatGPT

Instruction Tuning

1. Instruction Construct: Construct Instruction by mixture or self-instruct
2. Fine Tuning: Instruction Tuning on 4 LLM with multilingual instructions

see Instruction Tuning for details.

• Experiments
  • Datasets
    • Collection
    • Bootstrap
  • Model Cards
  • Usage
• Results

Constrained Generation

constrain LLM to generate specific answer (e.g., some open ended QA, limited vocabulary tasks), see Constrained Generate for details.

• Common method (constrain vocabulary + sample algorithm)
• Trie + Beam search (has issues currently)

Reference

• llm-action, 本项目旨在分享大模型相关技术原理以及实战经验