Maintenance PR's Welcome Awesome

A Comprehensive Overview of Large Language Models

This repo is for our paper: https://arxiv.org/abs/2307.06435

Please cite the paper, if our work is useful to your research:

@article{naveed2023comprehensive,
  title={A Comprehensive Overview of Large Language Models},
  author={Naveed, Humza and Khan, Asad Ullah and Qiu, Shi and Saqib, Muhammad and Anwar, Saeed and Usman, Muhammad and Barnes, Nick and Mian, Ajmal},
  journal={arXiv preprint arXiv:2307.06435},
  year={2023}
}

Contents

Surveys

  • Towards Reasoning in Large Language Models: A Survey, arXiv, 2022. [Paper]
  • Emergent Abilities of Large Language Models, arXiv, 2022. [Paper]
  • Several categories of Large Language Models (LLMs): A Short Survey arXiv, 2023. [Paper]
  • Retrieving Multimodal Information for Augmented Generation: A Survey, arXiv, 2023. [Paper]
  • Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions, JMIR, 2023. [Paper]
  • Language Model Behavior: A Comprehensive Survey, arXiv, 2023. [Paper]
  • Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond, arXiv, 2023. [Paper]
  • Beyond One-Model-Fits-All: A Survey of Domain Specialization for Large Language Models, arXiv, 2023. [Paper]
  • A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage, TechRxiv, 2023. [Paper]
  • Recent advances in natural language processing via large pre-trained language models: A survey, ACM Surveys, 2021. [Paper]
  • Complex QA and language models hybrid architectures, Survey, arXiv, 2023. [Paper]
  • Challenges and Applications of Large Language Models, arXiv, 2023. [Paper]
  • Augmented Language Models: a Survey, arXiv, 2023. [Paper]
  • A Survey on Multimodal Large Language Models, arXiv, 2023. [Paper]
  • A Survey on Evaluation of Large Language Models, arXiv, 2023. [Paper]
  • A Survey of Large Language Models, arXiv, 2023. [Paper]
  • ChatGPT for good? On opportunities and challenges of large language models for education, LID, 2023. [Paper]
  • A Short Survey of Viewing Large Language Models in Legal Aspect, arXiv, 2023. [Paper]
  • Aligning Large Language Models with Human: A Survey, arXiv, 2023. [Paper]
  • A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT, arXiv, 2023. [Paper]
  • Instruction Tuning for Large Language Models: A Survey, aeXiv, 2023. [Paper]
  • Examining User-Friendly and Open-Sourced Large GPT Models: A Survey on Language, Multimodal, and Scientific GPT Models, arXiv, 2023. [Paper]
  • Foundation Models for Decision Making: Problems, Methods, and Opportunities, arXiv, 2023. [Paper]
  • How Can Recommender Systems Benefit from Large Language Models: A Survey, arXiv, 2023. [Paper]
  • A Survey on Large Language Model based Autonomous Agents, arXiv, 2023. [Paper]
  • The Rise and Potential of Large Language Model Based Agents: A Survey, arXiv, 2023. [Paper]
  • A Survey on Large Language Model based Autonomous Agents, arXiv, 2023. [Paper]
  • Beyond One-Model-Fits-All: A Survey of Domain Specialization for Large Language Models, arXiv, 2023. [Paper]
  • Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveys. [Paper]

Pre-trained LLMs

General Purpose

  • T5: Exploring the limits of transfer learning with a unified text-to-text transformer, JMLR, 2020. [Paper]
  • GPT-3: Language Models are Few-Shot Learners, NeurIPS, 2020. [Paper]
  • mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer, NAACL, 2021. [Paper]
  • PanGu-alpha: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation, arXiv, 2021. [Paper]
  • CPM-2: Large-scale cost-effective pre-trained language models, AI Open, 2021. [Paper]
  • Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv, 2021. [Paper]
  • JURASSIC-1: Technical Details and Evaluation, White Paper, 2021.
  • HyperCLOVA: What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers, arXiv, 2021. [Paper]
  • Yuan 1.0: Large-scale pre-trained language model in zero-shot and few-shot learning, arXiv, 2021. [Paper]
  • Gopher: Scaling language models: Methods, analysis & insights from training gopher, arXiv, 2021. [Paper]
  • Ernie 3.0 titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation, arXiv, 2021. [Paper]
  • Gpt-neox-20b: An open-source autoregressive language model, arXiv, 2022. [Paper]
  • Opt: Open pre-trained transformer language models, arXiv, 2022. [Paper]
  • Bloom: A 176b-parameter open-access multilingual language model, arXiv, 2022. [Paper]
  • Glam: Efficient scaling of language models with mixture-of-experts, ICML, 2022. [Paper]
  • MT-NLG: Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model, arXiv, 2022. [Paper]
  • Chinchilla: Training compute-optimal large language models, arXiv, 2022. [Paper]
  • Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model, arXiv, 2022. [Paper]
  • Palm: Scaling language modeling with pathways, arXiv, 2022. [Paper]
  • U-Palm: Transcending scaling laws with 0.1% extra compute, arXiv, 2022. [Paper]
  • Ul2: Unifying language learning paradigms, ICLR, 2022. [Paper]
  • Glm-130b: An open bilingual pre-trained model, arXiv, 2022. [Paper]
  • Llama: Open and efficient foundation language models, arXiv, 2023. [Paper]
  • PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing, arXiv, 2023. [Paper]

Coding

  • Codegen: An open large language model for code with multi-turn program synthesis, arXiv, 2022. [Paper]
  • Codex: Evaluating large language models trained on code, arXiv, 2021. [Paper]
  • Alpha Code: Competition-level code generation with alphacode, Science, 2022. [Paper]
  • Codet5+: Open code large language models for code understanding and generation, arXiv, 2023. [Paper]
  • StarCoder: may the source be with you!, arXiv, 2023. [Paper]

Scientific Knowledge

  • Galactica: A large language model for science, arXiv, 2022, [Paper]

Dialog

  • Lamda: Language models for dialog applications, arXiv, 2022. [Paper]

Finance

  • Bloomberggpt: A large language model for finance, arXiv, 2023. [Paper]
  • XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters, arXiv, 2023. [Paper]

Fine-tuned LLMs

Instruction-tuning with Manually Created Datasets

  • T0: Multitask prompted training enables zero-shot task generalization, arXiv, 2021. [Paper]
  • mT0: Crosslingual generalization through multitask fine-tuning, arXiv, 2022. [Paper]
  • Tk-Instruct: Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks, arXiv, 2022. [Paper]
  • Opt-iml: Scaling language model instruction meta learning through the lens of generalization, arXiv, 2022. [Paper]
  • Flan: Scaling instruction-finetuned language models, arXiv, 2022. [Paper]
  • The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning, arXiv, 2023. [Paper]
  • From zero to hero: Examining the power of symbolic tasks in instruction tuning, arXiv, 2023. [Paper]

Instruction-tuning with LLMs Generated Datasets

  • Self-instruct: Aligning language model with self generated instructions, arXiv, 2022. [Paper]
  • Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation, arXiv, 2023. [Paper]
  • Stanford Alpaca: An Instruction-following LLaMA model, Github, 2023. [Link]
  • Vicucna: Github, 2023. [Link]
  • LLaMA-GPT-4: INSTRUCTION TUNING WITH GPT-4, arXiv, 2023. [Paper]
  • Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks, arXiv, 2023. [Paper]
  • Huatuo: Tuning llama model with chinese medical knowledge, arXiv, 2023. [Paper]
  • Wizardlm: Empowering large language models to follow complex instructions, arXiv, 2023. [Paper]
  • WizardCoder: Empowering Code Large Language Models with Evol-Instruct, arXiv, 2023. [Paper]

Aligning with Human Preferences

  • InstructGPT: Training language models to follow instructions with human feedback, NeurIPS, 2022. [Paper]
  • LLaMA-2-Chat: Llama 2: Open foundation and fine-tuned chat models, arXiv, 2023. [Paper]

Aligning with Supported Evidence

  • Webgpt: Browser-assisted question-answering with human feedback, arXiv, 2021. [Paper]
  • Sparrow: Improving alignment of dialogue agents via targeted human judgments, arXiv, 2022. [Paper]
  • GopherCite: Teaching language models to support answers with verified quotes, arXiv, 2022. [Paper]

Aligning Directly with SFT

  • DPO: Direct preference optimization: Your language model is secretly a reward model, arXiv, 2023. [Paper]
  • Raft: Reward ranked finetuning for generative foundation model alignment, arXiv, 2023. [Paper]
  • Rrhf: Rank responses to align language models with human feedback without tears, arXiv, 2023. [Paper]
  • PRO: Preference ranking optimization for human alignment, arXiv, 2023. [Paper]
  • CoH: Languages are rewards: Hindsight finetuning using human feedback, arXiv, 2023. [Paper]

Aligning with Synthetic Feedback

  • Constitutional ai: Harmlessness from ai feedback, arXiv, 2022. [Paper]
  • Alpacafarm: A simulation framework for methods that learn from human feedback, arXiv, 2023. [Paper]
  • Self-align: Principle-driven self-alignment of language models from scratch with minimal human supervision, arXiv, 2023. [Paper]

Aligning with Prompts

  • Prompting gpt-3 to be reliable, arXiv, 2022. [Paper]
  • The capacity for moral self-correction in large language models, arXiv, 2023. [Paper]

Red-Teaming Jailbreaking Adversarial Attacks

  • Red teaming language models with language models, arXiv, 2023. [Paper]
  • Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, arXiv, 2022. [Paper]
  • Jailbroken: How does llm safety training fail?, arXiv, 2023. [Paper]
  • Explore, Establish, Exploit: Red Teaming Language Models from Scratch, arXiv, 2023. [Paper]

Continue Pre-Training

  • Fine-tuned language models are continual learners, EMNLP, 2023. [Paper]
  • Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner, arXiv, 2023. [Paper]

Sample Efficiency

  • Instruction Tuned Models are Quick Learners, arXiv, 2023. [Paper]
  • Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning, arXiv, 2023. [Paper]
  • Lima: Less is more for alignment, arXiv, 2023. [Paper]

Increasing Context Window

Position Interpolation

  • Extending context window of large language models via positional interpolation, arXiv, 2023. [Paper]
  • Giraffe: Adventures in Expanding Context Lengths in LLMs, arXiv, 2023. [Paper]
  • YaRN: Efficient Context Window Extension of Large Language Models, arXiv, 2023. [Paper]

Efficient Attention Mechanism

  • LongT5: Efficient text-to-text transformer for long sequences, NAACl, 2022. [Paper]
  • Colt5: Faster long-range transformers with conditional computation, arXiv, 2023. [Paper]
  • Longnet: Scaling transformers to 1,000,000,000 tokens, arXiv, 2023. [Paper]
  • LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models, arXiv, 2023. [Paper]

Extrapolation without Training

  • LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models, arXiv, 2023. [Paper]
  • PCW: Parallel context windows for large language models, ACL, 2023. [Paper]

Augmented LLMs

Retrieval Augmented LLMs

  • Retrieval augmented language model pre-training, ICML,2020. [Paper]
  • Rationale-augmented ensembles in language models, arXiv, 2022. [Paper]
  • RETRO: Improving language models by retrieving from trillions of tokens, ICML, 2022. [Paper]
  • Learning to retrieve prompts for in-context learning, NACCL, 2022. [Paper]
  • Internet-augmented dialogue generation, ACL, 2022. [Paper]
  • Long time no see! open-domain conversation with long-term persona memory, arXiv, 2022. [Paper]
  • Internet-augmented language models through few-shot prompting for open-domain question answering, arXiv, 2022. [Paper]
  • FLARE: Active retrieval augmented generation, arXiv, 2023. [Paper]
  • In-context retrieval-augmented language models, arXiv, 2023. [Paper]
  • Repocoder: Repository-level code completion through iterative retrieval and generation, arXiv, 2023. [Paper]
  • Shall we pretrain autoregressive language models with retrieval? a comprehensive study, arXiv, 2023. [Paper]
  • Learning to Retrieve In-Context Examples for Large Language Models, arXiv, 2023. [Paper]
  • What makes good in-context examples for GPT-3?, arXiv, 2023. [Paper]
  • Learning to Retrieve In-Context Examples for Large Language Models, arXiv, 2023. [Paper]
  • Replug: Retrieval-augmented black-box language models, arXiv, 2023. [Paper]
  • RPT: Long-range Language Modeling with Self-retrieval, arXiv, 2023. [Paper]
  • Fid-light: Efficient and effective retrieval-augmented text generation, SIGIR, 2022. [Paper]
  • Augmenting Language Models with Long-Term Memory, arXiv, 2023. [Paper]
  • MemoryBank: Enhancing Large Language Models with Long-Term Memory, arXiv, 2023. [Paper]
  • Reflexion: Language Agents with Verbal Reinforcement Learning, arXiv, 2023. [Paper]
  • ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory, arXiv, 2023. [Paper]
  • Memory augmented large language models are computationally universal, arXiv, 2023. [Paper]
  • RET-LLM: Towards a General Read-Write Memory for Large Language Models, arXiv, 2023. [Paper]
  • Atlas: Few-shot Learning with Retrieval Augmented Language Models, JMLR, 2023. [Paper]

Tool Augmented LLMs

  • Talm: Tool augmented language models, arX0v, 2022. [Paper]
  • AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn, arXiv, 2023. [Paper]
  • Chameleon: Plug-and-play compositional reasoning with large language models, arXiv, 2023. [Paper]
  • Art: Automatic multi-step reasoning and tool-use for large language models, arXiv, 2023. [Paper]
  • Tool documentation enables zero-shot tool-usage with large language models, arXiv, 2023. [Paper]
  • RestGPT: Connecting Large Language Models with Real-World Applications via RESTful APIs, arXiv, 2023. [Paper]
  • ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings, arXiv, 2023. [Paper]
  • Gorilla: Large language model connected with massive apis, arXiv, 2023. [Paper]
  • On the Tool Manipulation Capability of Open-source Large Language Models, arXiv, 2023. [Paper]
  • Toolllm: Facilitating large language models to master 16000+ real-world apis, arXiv, 2023. [Paper]
  • Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface, arXiv, 2023. [Paper]
  • Gpt4tools: Teaching large language model to use tools via self-instruction, arXiv, 2023. [Paper]
  • Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis, arXiv, 2023. [Paper]
  • Vipergpt: Visual inference via python execution for reasoning, arXiv, 2023. [Paper]