/open_source_chatgpt_list

Open efforts to implement ChatGPT-like models and beyond.

open source ChatGPT and beyond

On the road to implement open-source ChatGPT-like models and beyond.

Since the accidental leak of LLaMA model weights, and the impressive performance of Stanford Alpaca, which is trained on LLaMA using data generated by GPT-3 api with the self-instruct technique, the open-source community has been excited about the promising future of reproducing ChatGPT in an open way.

This repo aims at recording this process, and providing an overview of how to get involved.

Including: base models, technologies, data, domain models, training pipelines, speed up techniques, multi-language, multi-modal, and more to go.

Base Models

contributor model/project multi-modal license language main feature
Meta LLaMA ✖ en LLaMA-13B outperforms GPT-3(175B) and LLaMA-65B is competitive to PaLM-540M.
Base model for most follow-up works.
THU ChatGLM-6B ✖ en/zh well-known Chinese model, in chat mode, and can run on single GPU.
HuggingFace-BigScience BLOOM ✖ multi an autoregressive Large Language Model (LLM) trained by HuggingFace BigScience.
HuggingFace-BigScience BLOOMZ ✖ multi instruction-finetuned version of BLOOM & mT5 pretrained multilingual language models on crosslingual task mixture.
EleutherAI GPT-J ✖ en transformer model trained using Ben Wang'sMesh Transformer JAX.
Meta OPT ✖ en Open Pre-trained Transformer Language Models, aim in developing this suite of OPT models is to enable reproducible
and responsible research at scale, and to bring more voices to the table in studying the impact of these LLMs.
Cerebras Systems Cerebras-GPT ✖ en Pretrained LLM, GPT-3 like, Commercially available, efficiently trained on theAndromeda AI supercomputer,
trained in accordance withChinchilla scaling laws (20 tokens per model parameter) which is compute-optimal.
EleutherAI pythia ✖ en combine interpretability analysis and scaling laws to understand how knowledge develops
and evolves during training in autoregressive transformers.
Stability-AI StableLM ✖ en Stability AI Language Models
FDU MOSS ✖ en/zh An open-source tool-augmented conversational language model from Fudan University.
ssymmetry & FDU BBT-2 ✖ zh 12B open-source LM.
@mlfoundations OpenFlamingo ✅ en An open-source framework for training large multimodal models.
EleutherAI GPT-NeoX-20B ✖ en Its architecture intentionally resembles that of GPT-3, and is almost identical to that ofGPT-J- 6B.
UCB OpenLLaMA ✖ Apache-2.0 en An Open Reproduction of LLaMA.
MosaicML MPT ✖ Apache-2.0 en MPT-7B is a GPT-style model, and the first in the MosaicML Foundation Series of models.
Trained on 1T tokens of a MosaicML-curated dataset, MPT-7B is open-source,
commercially usable, and equivalent to LLaMa 7B on evaluation metrics.

Domain Models

contributor model domain language base model main feature
UT Southwestern/
UIUC/OSU/HDU		 ChatDoctor medical en LLaMA Maybe the first domain-specific chat model tuned on LLaMA.
Cambridge Visual Med-Alpaca biomedical en LLaMA-7B a multi-modal foundation model designed specifically for the biomedical domain.
HIT Huatuo / ChatGLM-Med medical zh LLaMA/ChatGLM ine-tuned with Chinese medical knowledge dataset, which is generated by using gpt3.5 api.
ShanghaiTech, etc DoctorGLM medical en/zh ChatGLM-6B Chinese medical consultation model fine-tuned on ChatGLM-6B.
THU AIR BioMedGPT-1.6B biomedical en/zh - a pre-trained multi-modal molecular foundation model with 1.6B parameters that associates 2D molecular graphs with texts.
@LiuHC0428 LawGPT_zh legal zh ChatGLM-6B a general model in Chinese legal domain, trained on data generated via Reliable-Self-Instruction.
SJTU MedicalGPT-zh medical zh ChatGLM-6B a general model in Chinese medical domain, a diverse data generated via self-instruct.
SJTU PMC-LLaMA medical zh LLaMA Continue Training LLaMA on Medical Papers.

General Domain Instruction Models

contributor model/project language base model main feature
Stanford Alpaca en LLaMA/OPT use 52K instruction-following data generated by Self-Instructt techniques to fine-tune 7B LLaMA,
the resulting model,  Alpaca, behaves similarly to the text-davinci-003 model on the Self-Instruct instruction-following evaluation suite.
Alpaca has inspired many follow-up models.
LianJiaTech BELLE en/zh BLOOMZ-7B1-mt maybe the first Chinese model to follow Alpaca.
Databricks Dolly en GPT-J 6B use Alpaca data to fine-tune a 2-year-old model: GPT-J, which exhibits surprisingly high quality
instruction following behavior not characteristic of the foundation model on which it is based.
@tloen Alpaca-LoRA en LLaMA-7B trained within hours on a single RTX 4090,
reproducing the Stanford Alpaca results using low-rank adaptation (LoRA),
and can run on a Raspberry pi.
ColossalAI Coati7B en/zh LLaMA-7B a large language model developed by the ColossalChat project
Shanghai AI Lab LLaMA-Adapter en LLaMA-7B Fine-tuning LLaMA to follow instructions within 1 Hour and 1.2M Parameters
AetherCortex Llama-X en LLaMA Open Academic Research on Improving LLaMA to SOTA LLM.
TogetherComputer OpenChatKit en GPT-NeoX-20B OpenChatKit provides a powerful, open-source base to create both specialized and general purpose chatbots for various applications.
The kit includes an instruction-tuned language models, a moderation model, and an extensible retrieval system for including
up-to-date responses from custom repositories.
nomic-ai GPT4All en LLaMA trained on a massive collection of clean assistant data including code, stories and dialogue
@ymcui Chinese-LLaMA-Alpaca en/zh LLaMA-7B/13B expand the Chinese vocabulary based on the original LLaMA and use Chinese data for secondary pre-training,
further enhancing Chinese basic semantic understanding. Additionally, the project uses Chinese instruction data
for fine-tuning on the basis of the Chinese LLaMA, significantly improving the model's understanding and execution of instructions.
UC Berkley
Stanford
CMU		 Vicuna en LLaMA-13B Impressing GPT-4 with 90% ChatGPT Quality.
UCSD/SYSU baize en/zh LLaMA fine-tuned withLoRA. It uses 100k dialogs generated by letting ChatGPT chat with itself.
Alpaca's data is also used to improve its performance.
UC Berkley Koala en LLaMA Rather than maximizingquantity by scraping as much web data as possible, the team focus on collecting a small high-quality dataset.
@imClumsyPanda langchain-ChatGLM en/zh ChatGLM-6B local knowledge based ChatGLM with langchain.
@yangjianxin1 Firefly zh bloom-1b4-zh
bloom-2b6-zh		 Instruction Tuning on Chinese dataset. Vocabulary pruning, ZeRO, and tensor parallelism
are used to effectively reduce memory consumption and improve training efficiency.
microsoft GPT-4-LLM en/zh LLaMA aims to share data generated by GPT-4 for building an instruction-following LLMs with supervised learning and reinforcement learning.
Hugging Face StackLLaMA en LLaMA trained on StackExchange data and the main goal is to serve as a tutorial and walkthrough on
how to train model with RLHF and not primarily model performance.
Nebuly ChatLLaMA en - a library that allows you to create hyper-personalized ChatGPT-like assistants using your own data and the least amount of compute possible.
@juncongmoo ChatLLaMA en LLaMA LLaMA-based RLHF model, runnable in a single GPU.
@juncongmoo minichatgpt en GPT/OPT ... To Train ChatGPT In 5 Minutes with ColossalAI.
@LC1332 Luotuo-Chinese-LLM zh LLaMA/ChatGLM Instruction fine-tuned Chinese Language Models, with colab provided!
@Facico Chinese-Vicuna zh LLaMA A Chinese Instruction-following LLaMA-based Model, fine-tuned with Lora, cpp inference supported, colab provided.
@yanqiangmiffy InstructGLM en/zh ChatGLM-6B ChatGLM based instruction-following model, fine-tuned on a variety of data sources, supports deepspeed accelerating and LoRA.
alibaba Wombat en LLaMA a novel learning paradigm called RRHF, as an alternative of RLHF,  is proposed, which scores responses generated by
different sampling policies and learns to align them with human preferences through ranking loss. And the performance
is comparable to RLHF, with less models used in the process.
@WuJunde alpaca-glassoff en LLaMA a mini image-acceptable Chat AI can run on your own laptop,  based onstanford-alpaca and alpaca-lora.
@JosephusCheung Guanaco multi LLaMA-7B A Multilingual Instruction-Following Language Model.
BlinkDL ChatRWKV en/zh RNN powered by RWKV (100% RNN), Training sponsored by Stability EleutherAI.
@FreedomIntelligence LLM Zoo multi BLOOMZ/LLaMA a project that provides data, models, and evaluation benchmark for large language models.
model released: Phoenix, Chimera
SZU Linly en/zh LLaMA expand the Chinese vocabulary, full fine-tuned models, largest LLaMA-based Chinese models, aggregation of Chinese instruction data, reproduceable details..
@lamini-ai lamini multi - data generator for generating instructions to train instruction-following LLMs.
Stability-AI StableVicuna en LLaMA a further instruction fine tuned and RLHF trained version of Vicuna v0 13b, with better performance than Vicuna.
Hugging Face HuggingChat en LLaMA seems to be the first one available to access as a platform that appears similar to ChatGPT.
microsoft WizardLM en LLaMA-7B trained with 70k evolved instructions,Evol-Instruct is a novel method using LLMs instead of humans to automatically mass-produce
open-domain instructions of various difficulty levels and skills range, to improve the performance of LLMs.
FDU OpenChineseLLaMA en/zh LLaMA-7B further pretrain LLaMA on Chinese data, improving LLaMA preformance on Chinese tasks.
@chenfeng357 open-Chinese-ChatLLaMA en/zh LLaMA The complete training code of the open-source Chinese-Llama model, including the full process from pre-training instructing and RLHF.
@FSoft-AI4Code CodeCapybara en LLaMA Open Source LLaMA Model that Follow Instruction-Tuning for Code Generation.
@mbzuai-nlp LaMini-LM en LLaMA/Flan-T5 ... A Diverse Herd of Distilled Models from Large-Scale Instructions.
@dandelionsllm Panda en/zh LLaMA further pretraining on Chinese data, full-size of LLaMA models.
@hiyouga ChatGLM-Efficient-Tuning en/zh ChatGLM-6B efficient fine-tuning ChatGLM-6B with PEFT.
IBM/CMU/MIT Dromedary en LLaMA-65B Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision.

Multi-Modal

contributor project language base model main feature
BaihaiAIen/zh IDPChat en/zh LLaMA-13B
Stable Diffusion		 Open Chinese multi-modal model, single GPU runnable, easy to deploy, UI provided.
KAUST MiniGPT-4 en/zh LLaMA MiniGPT-4 aligns a frozen visual encoder from BLIP-2 with a frozen LLM, Vicuna, using just one projection layer,
and yields many emerging vision-language capabilities similar to those demonstrated in GPT-4.
UW–Madison/MSR
/Columbia University		 LLaVA en LLaMA visual instruction tuning is proposed, towards building large language and vision models with GPT-4 level capabilities.
NUS/THU VPGTrans en LLaMA/OPT/
Flan-T5/BLIP-2
...		 transferring VPG across LLMs to build VL-LLMs at significantly lower cost. The GPU hours
can be reduced over 10 times and the training data can be reduced to around 10%.
Two novel VL-LLMs are released via VPGTrans, including VL-LLaMA and VL-Vicuna.
VL-LLaMA is a multimodal version LLaMA by transferring the BLIP-2 OPT-6.7B to LLaMA via VPGTrans.
VL-Vicuna is a GPT-4-like multimodal chatbot, based on the Vicuna LLM.

Data

Pretrain Data

contributor data language main feature
TogetherComputer RedPajama-Data en An Open Source Recipe to Reproduce LLaMA training dataset.

Instruction Data

see Alpaca-CoT data collection

Synthetic Data Generation

contributor method main feature
UW, etc. self-instruct using the model's own generations to create a large collection of instructional data.
@LiuHC0428 Reliable-Self-Instruction use ChatGPT to generate some questions and answers based on a given text.
PKU Evol-Instruct a novel method, proposed inWizardLM,  by using LLMs instead of humans to automatically mass-produce open-domain
instructions of various difficulty levels and skills range, to improve the performance of LLMs.
KAUST, etc. CAMEL a novel communicative agent framework namedrole-playing is proposed, which involves using inception prompting to guide chat agents
toward task completion while maintaining consistency with human intentions.
role-playing can be used to generate conversational data in a specific task/domain.

Evaluation

contributor method main feature
- human evalation -
OpenAI GPT-4/ChatGPT -
PKU/CMU/MSRA ... PandaLM Reproducible and Automated Language Model Assessment.
UCB Chatbot Arena Chat with two anonymous models side-by-side and vote for which one is better,
then use the Elo rating system to calculate the relative performance of the models.

Framework/ToolKit/Platform

contributor project main feature
CAS Alpaca-CoT extend CoT data to Alpaca to boost its reasoning ability.
aims at building an instruction finetuning (IFT) platform with extensive instruction collection (especially the CoT datasets)
and a unified interface for various large language models.
ColossalAI ColossalChat An open-source low cost solution for cloningChatGPT with a complete RLHF pipeline.
microsoft deepspeed-chat Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales.
LAION-AI Open Assistant a project meant to give everyone access to a great chat based large language model.
HKUST LMFlow an extensible, convenient, and efficient toolbox for finetuning large machine learning models,
designed to be user-friendly, speedy and reliable, and accessible to the entire community.
UCB EasyLM EasyLM is a one stop solution for pre-training, finetuning, evaluating and serving LLMs in JAX/Flax.
EasyLM can scale up LLM training to hundreds of TPU/GPU accelerators by leveraging JAX's pjit functionality.

Alignment

contributor method used in main feature
- SFT ChatGPT Supervised Fine-Tuning.
- RLHF ChatGPT RL from Human Feedback.
Anthropic RLAIF Claude RL from AI Feedback.
alibaba RRHF Wombat a novel learning paradigm called RRHF, as an alternative of RLHF,  is proposed, which scores responses generated by
different sampling policies and learns to align them with human preferences through ranking loss. And the performance
is comparable to RLHF, with less models used in the process.
HKUST RAFT - RAFT is a new alignment algorithm, which is more efficient than conventional (PPO-based) RLHF.
IBM/CMU/MIT SELF-ALIGN Dromedary combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision.

Multi-Language

vocabulary expansion

todo

Efficient Training/Fine-Tuning

contributor method main feature
microsoft LoRA Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices
into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.
stanford Prefix Tuning a lightweight alternative to fine-tuning for natural language generation tasks, which keeps language model parameters frozen
and instead optimizes a sequence of continuous task-specific vectors, which we call the prefix.
THU P-Tuning P-tuning leverages few continuous free parameters to serve as prompts fed as the input to the pre-trained language models.
We then optimize the continuous prompts using gradient descent as an alternative to discrete prompt searching.
THU/BAAI/
Shanghai Qi Zhi Institute		 P-Tuning v2 a novel empirical finding that properly optimized prompt tuning can be comparable to fine-tuning universally across various model scales and NLU tasks.
Technically, P-tuning v2 is not conceptually novel. It can be viewed as an optimized and adapted implementation of Deep Prompt Tuning.
Google Prompt Tuning a simple yet effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks.
Prompt Tuning can be seen as a simplification of "prefix tuning".
GT/Princeton/microsoft AdaLoRA adaptively allocates the parameter budget among weight matrices according to their importance score.
In particular, AdaLoRA parameterizes the incremental updates in the form of singular value decomposition.

acknowledgement: HuggingFace Peft

Low-Cost Inference

contributor project language base model main feature
@ggerganov llama.cpp multi LLaMA c/cpp implementation for llama and some other models, using quantization.
@NouamaneTazi bloomz.cpp multi BLOOMZ C++ implementation for BLOOM inference.
@mlc-ai MLC LLM multi - a universal solution that allows any language models to be deployed natively on a diverse set of hardware backends and native applications,
plus a productive framework for everyone to further optimize model performance for their own use cases.  

Satefy

contributor method main feature
thu-coai Safety-Prompts Chinese safety prompts for evaluating and improving the safety of LLMs.

Input Length Extrapolation

contributor method main feature
UW, etc. ALiBi Instead of adding position embeddings at the bottom of the transformer stack,
ALiBi adds a linear bias to each attention score, allowing the model to be trained on,
for example, 1024 tokens, and then do inference on 2048 (or much more) tokens without any finetuning.
DeepPavlov, etc. RMT use a recurrent memory to extend the context length.
bytedance SCM unleash infinite-length input capacity for large-scale language models.