/gpt_alternatives

A Survey on Language, Multimodal, and Scientific GPT Models: Examing User-Friendly and Open-Sourced Large GPT Models

Continuously updating

The original paper is released on arxiv.

Introduction

The advent of GPT models has brought about a significant transformation in the field of NLP. These models, such as GPT-4, demonstrate exceptional capabilities in various NLP tasks. However, despite their impressive capabilities, large GPT models have inherent limitations that restrict their widespread adoption, usability, and fine-tuning. The need for user-friendly, relatively small, and open-sourced alternative GPT models arises from the desire to overcome these limitations while retaining high performance. In this survey paper, we provide an examination of alternative open-sourced models of large GPTs, focusing on user-friendly and relatively small models (near 10B) that facilitate easier deployment and accessibility.

  • Investigate the architecture, design principles, and trade-offs of user-friendly and relatively small alternative GPT models, focusing on their ability to overcome the challenges posed by large GPT models.
  • Present the data collection and analyze the pre-training data source, data quality, quantity, diversity, and finetuning data including instruction data, alignment data, and also the domain-specific data for domain-specific models.
  • Survey the techniques for efficient deployment and fine-tuning of these GPT models.
  • Introduce ongoing open-source projects and initiatives for user-friendly GPT model reproduction and deployment.
  • Provide a thorough analysis of benchmark evaluations and offer human evaluations of these relatively small GPT models to give some human-liked recommendations in real usage.
  • Explore the extension of GPT models to multimodal settings, focusing on models that integrate NLP with computer vision, and also place special focus on user-friendly scientific GPT models and biomedical domains

The overview of the content is shown in Figure 1. Figure 1: Overview of the content

GPT and GPT-like models

Related papers/links for open LLMs (List is updating)

Language Domain

  1. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR 2020. [paper] [code & models] [Huggingface models]
  2. mT5: A massively multilingual pre-trained text-to-text transformer. NAACL 2021. [paper] [code & models] [Huggingface models]
  3. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. [code & models] [Huggingface models]
  4. Gpt-neox-20b: An open-source autoregressive language model. arxiv 2022. [paper] [code] [original models] [Huggingface models]
  5. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [code & models] [Huggingface models]
  6. Opt: Open pre-trained transformer language models. arxiv 2022. [paper] [code] [Huggingface models]
  7. BLOOM: A 176b-parameter open-access multilingual language model. arxiv 2022. [paper] [Huggingface models]
  8. Crosslingual Generalization through Multitask Finetuning. arxiv 2022. [paper] [Huggingface models]
  9. Glm: General language model pretraining with autoregressive blank infilling. ACL 2022. [paper] [code & models] [Huggingface models]
  10. GLM-130B: An Open Bilingual Pre-trained Model. ICLR 2023. [paper] [code & models]
  11. ChatGLM-6B [code & models] [Huggingface models]
  12. ChatGLM2-6B [code & models] [Huggingface models]
  13. LLaMA: Open and Efficient Foundation Language Models. arxiv 2023. [paper] [code & models]
  14. OpenLLaMA: An Open Reproduction of LLaMA. [code & models]
  15. Stanford Alpaca: An Instruction-following LLaMA Model. [code & models]
  16. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality. [blog] [code & models]
  17. StableLM: Stability AI Language Models. [code & models]
  18. Baize. [code & models]
  19. Koala: A Dialogue Model for Academic Research. [blog] [code & models]
  20. WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions. [code & models]
  21. Large-scale, Informative, and Diverse Multi-round Dialogue Data, and Models. [code & models]
  22. YuLan-Chat: An Open-Source Bilingual Chatbot. [code & models]
  23. Pythia: Interpreting Transformers Across Time and Scale. arxiv 2023. [paper] [code & models]
  24. Dolly. [code & models]
  25. OpenChatKit. [code & models]
  26. BELLE: Be Everyone's Large Language model Engine. [code & models]
  27. RWKV: Reinventing RNNs for the Transformer Era. arxiv 2023. [paper] [code & models] [Huggingface models]
  28. ChatRWKV. [code & models]
  29. MOSS. [code & models]
  30. RedPajama-INCITE. [blog] [Huggingface models]
  31. Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs. [blog] [code] [Huggingface models]
  32. Introducing Falcon LLM. [blog] [Huggingface models]
  33. InternLM. [code & models]
  34. Baichuan-7B. [code & models]
  35. Llama 2: Open Foundation and Fine-Tuned Chat Models. arxiv 2023. [paper] [code & models]
  36. Introducing Qwen-7B: Open foundation and human-aligned models. code & models]
  37. XVERSE-13B. [code & models]

Multimodal Domain

  1. Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS 2022. [paper]
  2. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arxiv 2023. [paper] [code]
  3. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arxiv 2023. [paper] [website, code & models]
  4. Visual Instruction Tuning. arxiv 2023. [paper] [website, code & models]
  5. mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality. arxiv 2023. [paper] [code & models]
  6. Transfer Visual Prompt Generator across LLMs. arxiv 2023. [paper] [webste, code & models]
  7. Otter: A Multi-Modal Model with In-Context Instruction Tuning. arxiv 2023. [paper] [code & models]
  8. MultiModal-GPT: A Vision and Language Model for Dialogue with Humans. arxiv 2023. [paper] [code & models]

Scientific Domain

  1. BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining. Bioinformatics 2022. [paper] [code & models]
  2. Galactica: A Large Language Model for Science. arxiv 2022. [paper] [models]
  3. BiomedGPT: A Unified and Generalist Biomedical Generative Pre-trained Transformer for Vision, Language, and Multimodal Tasks. arxiv 2023. [paper] [code & models]
  4. MolXPT: Wrapping Molecules with Text for Generative Pre-training. ACL 2023. [paper] [code & models]
  5. Translation between Molecules and Natural Language. EMNLP 2022. [paper] [code & models]

Figure 2: Model Evolution

Table 1. Statistical overview of open large language models in recent years, categorized by base models Outer pipes Cell padding No sorting

Model #Param Backbone Release Date Training Data Size
T5 (enc-dec) [github] 60M, 220M, 770M, 3B, 11B Base Model 2019-10 1T tokens
mT5 (enc-dec) [github] 300M, 580M, 1.2B, 3.7B, 13B Base Model 2020-10 1T tokens
GPT-Neo [github] 125M, 350M, 1.3B, 2.7B Base Model 2021-03 825GB
GPT-NeoX [github] 20B Base Model 2022-02 825GB
GPT-J [github] 6B Base Model 2021-06 825GB
OPT [github] 125M, 1.3B, 2.7B, 6.7B, 13B, 30B, 66B, 175B Base Model 2022-05 180B tokens
BLOOM 560M, 1.1B, 1B7, 3B, 7.1B, 176B Base Model 2022-07 366B tokens
BLOOMZ 560M, 1.1B, 1B7, 3B, 7.1B, 176B BLOOM 2022-11 -
GLM [github] 110M, 335M, 410M, 515M, 2B, 10B, 130B Base Model 2021-03
English Wikipedia -
GLM-130B [github] 130B Base Model 2022-08 -
ChatGLM [github] 6B GLM 2023-03 -
ChatGLM2 [github] 6B GLM 2023-06 -
LLaMA [github] 7B, 13B, 33B, 65B Base Model 2023-02 1.4T tokens
OpenLLaMA [github] 3B, 7B Replicate of LLaMA 2023-05
Alpaca [github] 7B LLaMA 2023-03 52K
Vicuna [github] 7B, 13B LLaMA 2023-03 70K
StableVicuna [github] 13B LLaMA Vicuna -
BAIZE [github] 7B, 13B, 30B LLaMA 2023-04 54K/57K/47K
Koala [github] 13B LLaMA 2023-04 -
WizardLM [github] 7B, 13B, 30B LLaMA 2023-06 250k/70k
UltraLM [github] 13B LLaMA 2023-06 -
Pythia [github] 70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, 12B Base Model 2023-01 299.9B tokens/207B tokens
Dolly-v2 [github] 12B Pythia 2023-04 \textasciitilde 15k
Openchatkit [github] 7B Pythia 2023-03
BELLE-7B [github] 7B Pythia 2023-03 1.5M
StableLM-Alpha [github] 3B, 7B Base Model 2023-04 1.5T tokens
StableLM-Tuned-Alpha [github] 7B StableLM 2023-04 -
RWKV [github] 169M, 430M, 1.5B, 3B, 7B, 14B Base Model - 825GB
ChatRWKV [github] 7B, 14B RWKV 2022-12 -
moss-moon-003-base [github] 16B base model 2023-04 700B tokens
moss-moon-003-sft [github] 16B moss-moon-003-base 2023-04 1.1 million
RedPajama-INCITE 3B, 7B Base Model 2023-05 1.2T tokens
MPT-7B [github] 7B Base Model 2023-05 1T tokens
MPT-7B-Chat [github] 7B MPT-7B 2023-05 -
Falcon LLM 7B, 40B Base Model 2023-06 1T tokens
InternLM [github] 7B Base Model 2023-06 trillions of tokens
InternLM Chat [github] 7B InternLM 2023-06 -
Baichuan [github] 7B Base Model 2023-06 1.2T tokens
LLAMA 2 [github] 7B, 13B, 70B Base Model 2023-07 2T tokens
LLAMA 2-CHAT [github] 7B, 13B, 70B LLAMA 2 2023-07 27,540 instruction tuning data, 2,919,326 human preference data
Qwen [github] 7B Base Model 2023-08 2.2T tokens
Qwen-Chat [github] 7B Qwen 2023-08 -

Training/fintuning Data sources

Deployment and fine-tuning technique

Efficient Deploy

Related papers

  1. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers. arxiv 2022. [paper]
  2. LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models. arxiv 2022. [paper]
  3. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. Neurips 2022. [paper]
  4. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023. [paper]
  5. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. ICML 2023. [paper]
  6. LLM-QAT: Data-Free Quantization Aware Training for Large Language Models. arxiv 2023. [paper]

3.2 Efficient Finetuning

Related papers

  1. Parameter-Efficient Transfer Learning for NLP. ICML 2019. [paper]
  2. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. [paper]
  3. The power of scale for parameter-efficient prompt tuning. EMNLP 2021. [paper]
  4. GPT Understands, Too. arxiv 2021. [paper]
  5. Prefix-Tuning: Optimizing Continuous Prompts for Generation. ACL 2021. [paper]
  6. P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks. ACL 2022. [paper]
  7. QLoRA: Efficient Finetuning of Quantized LLMs. arxiv 2023. [paper]

Open-sourced tools

TABLE 5: Overview of open-source efforts and tools development I have swapped the "tool" and "category" columns in the markdown table as requested:

Category Tool Application Released by Link
Deployment Transformers LLM training and deployment Huggingface https://huggingface.co/transformers
Colossal-AI Unified system to train and deploy large-scale models HPC-AI Tech https://colossalai.org/
GPT4all Large and personalized language models training and deployment on common hardware Nomic AI https://gpt4all.io/
PandaLM System providing automated and reproducible comparisons among various LLMs Westlake University https://github.com/WeOpenML/PandaLM
MLC LLM Solution allowing LLMs to be deployed natively MLC AI https://mlc.ai/mlc-llm/
Accelerating Deepspeed Accelerating training and inference of large-scale models Microsoft https://github.com/microsoft/DeepSpeed
Megatron-LM Accelerating training and inference of large-scale models Nvidia https://github.com/NVIDIA/Megatron-LM
Reproduction MinGPT Re-implementation of GPT which is clean, interpretable and educational Stanford University https://github.com/karpathy/minGPT
RedPajama An effort to produce reproducible and fully-open language models ETH Zurich https://together.xyz/blog/redpajama
Framework LangChain Framework for integration of LLMs with other computational sources and knowledge LangChain https://python.langchain.com/
xTuning Framework providing fast, efficient and simple fine-tuning of LLMs Stochastic https://github.com/stochasticai/xturing
Evaluation Open LLM Leaderboard LM evaluation leaderboard Huggingface https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
Framework Scikit-LLM Framework integrating LLMs into scikit-learn for enhanced text analysis tasks Tractive https://github.com/iryna-kondr/scikit-llm
AlpacaFarm Simulation framework for methods that learn from human feedback Stanford https://github.com/tatsu-lab/alpaca_farm/
h2oGPT LLM finetuning framework and chatbot UI with document(s) question-answer capabilities H2O.ai https://github.com/h2oai/h2ogpt
Software Open-Assistant Customized and personalized chat-based assistant LAION AI https://github.com/LAION-AI/Open-Assistant
MetaGPT Multi-agent framework to tackle tasks with multiple agents Open-Source Community https://github.com/geekan/MetaGPT
Finetuning PEFT Library for finetuning LLMs with only part of parameters Huggingface https://huggingface.co/docs/peft

Benchmark evaluations

Upcoming soon ...

Misc

TABLE 16. ChatGPT Alternatives on Different Applications

Field Software Backbone Url
Writing ChatSonic GPT-4 https://writesonic.com/chat
Jasper Chat GPT 3.5 and others https://www.jasper.ai/chat
Search Engines ChatSonic on Opera GPT-4 https://writesonic.com/chatsonic-opera
NeevaAI ChatGPT https://neeva.com/
Coding Copilot Codex https://github.com/features/copilot
Tabnine GPT-2 https://www.tabnine.com/
Codewhisperer - https://aws.amazon.com/cn/codewhisperer
Language Learning Elsa - https://elsaspeak.com/en
DeepL Write - https://www.deepl.com/translator
Research Elicit - https://elicit.org
ChatPDF ChatGPT https://www.chatpdf.com
Copilot in Azure Quantum GPT-4 https://quantum.microsoft.com/
Productivity (team work) CoGram - https://www.cogram.com
Otter - https://otter.ai
Chatexcel - https://chatexcel.com
AI Anywhere ChatGPT, GPT-4 https://www.ai-anywhere.com/#/dashboard
Conversation Replika A model with 774M parameters https://replika.com
Character AI GPT-4 https://beta.character.ai
Poe Multiple Models (GPT-4, LLaMA, ...) https://poe.com
Building customized AI Botsonic AI chatbot GPT-4 https://writesonic.com/botsonic

Reference

If you find our paper/repository useful, please kindly cite our paper.

@misc{gao2023examining,
      title={Examining User-Friendly and Open-Sourced Large GPT Models: A Survey on Language, Multimodal, and Scientific GPT Models}, 
      author={Kaiyuan Gao and Sunan He and Zhenyu He and Jiacheng Lin and QiZhi Pei and Jie Shao and Wei Zhang},
      year={2023},
      eprint={2308.14149},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}