/awesome-llm-and-aigc

A collection of some awesome public projects about Large Language Model, Vision Foundation Model and AI Generated Content.

Awesome-llm-and-aigc

Awesome

🔥🔥🔥 This repository lists some awesome public projects about Large Language Model, Vision Foundation Model and AI Generated Content.

Contents

Summary

  • Frameworks

    • Official Version

      • Large Language Model

        • GPT-1 : "Improving Language Understanding by Generative Pre-Training". (cs.ubc.ca, 2018).

        • GPT-2 : "Language Models are Unsupervised Multitask Learners". (OpenAI blog, 2019). Better language models and their implications.

        • GPT-3 : "GPT-3: Language Models are Few-Shot Learners". (arXiv 2020).

        • InstructGPT : "Training language models to follow instructions with human feedback". (arXiv 2022). "Aligning language models to follow instructions". (OpenAI blog, 2022).

        • ChatGPT: Optimizing Language Models for Dialogue.

        • GPT-4: GPT-4 is OpenAI’s most advanced system, producing safer and more useful responses.

        • Whisper : Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification. "Robust Speech Recognition via Large-Scale Weak Supervision". (arXiv 2022).

        • MiniGPT-4 : MiniGPT-4: Enhancing Vision-language Understanding with Advanced Large Language Models. minigpt-4.github.io

        • Auto-GPT : Auto-GPT: An Autonomous GPT-4 Experiment. Auto-GPT is an experimental open-source application showcasing the capabilities of the GPT-4 language model. This program, driven by GPT-4, chains together LLM "thoughts", to autonomously achieve whatever goal you set. As one of the first examples of GPT-4 running fully autonomously, Auto-GPT pushes the boundaries of what is possible with AI.

        • LLaMA : Inference code for LLaMA models. "LLaMA: Open and Efficient Foundation Language Models". (arXiv 2023).

        • StableLM : StableLM: Stability AI Language Models.

        • JARVIS : JARVIS, a system to connect LLMs with ML community. "HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace". (arXiv 2023).

        • minGPT : A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training.

        • nanoGPT : The simplest, fastest repository for training/finetuning medium-sized GPTs.

        • Claude : Claude is a next-generation AI assistant based on Anthropic’s research into training helpful, honest, and harmless AI systems.

        • MicroGPT : A simple and effective autonomous agent compatible with GPT-3.5-Turbo and GPT-4. MicroGPT aims to be as compact and reliable as possible.

        • Dolly : Databricks’ Dolly, a large language model trained on the Databricks Machine Learning Platform. Hello Dolly: Democratizing the magic of ChatGPT with open models

        • LMFlow : An extensible, convenient, and efficient toolbox for finetuning large machine learning models, designed to be user-friendly, speedy and reliable, and accessible to the entire community. Large Language Model for All. optimalscale.github.io/LMFlow/

        • Open-Assistant : OpenAssistant is a chat-based assistant that understands tasks, can interact with third-party systems, and retrieve information dynamically to do so. open-assistant.io

        • Colossal-AI : Making big AI models cheaper, easier, and scalable. www.colossalai.org. "Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training". (arXiv 2021).

        • Lit-LLaMA : ⚡ Lit-LLaMA. Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

        • GPT-4-LLM : "Instruction Tuning with GPT-4". (arXiv 2023). instruction-tuning-with-gpt-4.github.io/

        • ChatGLM-6B : ChatGLM-6B：开源双语对话语言模型 | An Open Bilingual Dialogue Language Model. ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型，基于 General Language Model (GLM) 架构，具有 62 亿参数。 "GLM: General Language Model Pretraining with Autoregressive Blank Infilling". (ACL 2022). "GLM-130B: An Open Bilingual Pre-trained Model". (ICLR 2023).

        • MOSS : An open-source tool-augmented conversational language model from Fudan University. MOSS是一个支持中英双语和多种插件的开源对话语言模型，moss-moon系列模型具有160亿参数，在FP16精度下可在单张A100/A800或两张3090显卡运行，在INT4/8精度下可在单张3090显卡运行。MOSS基座语言模型在约七千亿中英文以及代码单词上预训练得到，后续经过对话指令微调、插件增强学习和人类偏好训练具备多轮对话能力及使用多种插件的能力。txsun1997.github.io/blogs/moss.html

        • 百度-文心大模型 : 百度全新一代知识增强大语言模型，文心大模型家族的新成员，能够与人对话互动，回答问题，协助创作，高效便捷地帮助人们获取信息、知识和灵感。

        • 阿里云-通义千问 : 通义千问，是阿里云推出的一个超大规模的语言模型，功能包括多轮对话、文案创作、逻辑推理、多模态理解、多语言支持。能够跟人类进行多轮的交互，也融入了多模态的知识理解，且有文案创作能力，能够续写小说，编写邮件等。

        • 商汤科技-日日新SenseNova : 日日新（SenseNova），是商汤科技宣布推出的大模型体系，包括自然语言处理模型“商量”（SenseChat）、文生图模型“秒画”和数字人视频生成平台“如影”（SenseAvatar）等。

        • GPT2-Chinese : Chinese version of GPT2 training code, using BERT tokenizer.

        • feizc/Visual-LLaMA : Open LLaMA Eyes to See the World. This project aims to optimize LLaMA model for visual information understanding like GPT-4 and further explore the potentional of large language model.

        • Lightning-AI/lightning-colossalai : Efficient Large-Scale Distributed Training with Colossal-AI and Lightning AI.

      • Vision Foundation Model

        • InternImage : "InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions". (CVPR 2023).

        • DINOv2 : "DINOv2: Learning Robust Visual Features without Supervision". (arXiv 2023).

        • Segment Anything : The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model. "Segment Anything". (arXiv 2023).

        • Track-Anything : Track-Anything is an Efficient Development Toolkit for Video Object Tracking and Segmentation, based on Segment Anything and XMem.

        • LLaVA : 🌋 LLaVA: Large Language and Vision Assistant. Visual instruction tuning towards large language and vision models with GPT-4 level capabilities. llava.hliu.cc. "Visual Instruction Tuning". (arXiv 2023).

      • AI Generated Content

        • Stable Diffusion : Stable Diffusion is a latent text-to-image diffusion model. Stable Diffusion was made possible thanks to a collaboration with Stability AI and Runway and builds upon our previous work "High-Resolution Image Synthesis with Latent Diffusion Models". (CVPR 2022).

        • Stable Diffusion Version 2 : This repository contains Stable Diffusion models trained from scratch and will be continuously updated with new checkpoints. "High-Resolution Image Synthesis with Latent Diffusion Models". (CVPR 2022).

        • Visual ChatGPT : Visual ChatGPT connects ChatGPT and a series of Visual Foundation Models to enable sending and receiving images during chatting. "Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models". (arXiv 2023).

        • AudioGPT : AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head.

        • PandasAI : Pandas AI is a Python library that adds generative artificial intelligence capabilities to Pandas, the popular data analysis and manipulation tool. It is designed to be used in conjunction with Pandas, and is not a replacement for it.

        • Midjourney : Midjourney is an independent research lab exploring new mediums of thought and expanding the imaginative powers of the human species.

        • Firefly : Adobe Firefly: Experiment, imagine, and make an infinite range of creations with Firefly, a family of creative generative AI models coming to Adobe products.

        • Jasper : Meet Jasper. On-brand AI content wherever you create.

        • Copy.ai : Whatever you want to ask, our chat has the answers.

        • Peppertype.ai : Leverage the AI-powered platform to ideate, create, distribute, and measure your content and prove your content marketing ROI.

        • ChatPPT : ChatPPT来袭命令式一键生成PPT。

    • C Implementation

    • Rust Implementation

    • Zig Implementation

      • renerocksai/gpt4all.zig : ZIG build for a terminal-based chat client for an assistant-style large language model with ~800k GPT-3.5-Turbo Generations based on LLaMa.

  • Awesome List

  • Paper and Code Overview

    • Paper Review

      • daochenzha/data-centric-AI : A curated, but incomplete, list of data-centric AI resources. "Data-centric Artificial Intelligence: A Survey". (arXiv 2023).

      • KSESEU/LLMPapers : Collection of papers and related works for Large Language Models (ChatGPT, GPT-3, Codex etc.).

    • Code Review

      • GPT4All : GPT4All: An ecosystem of open-source on-edge large language models. GTP4All is an ecosystem to train and deploy powerful and customized large language models that run locally on consumer grade CPUs.

      • LangChain : ⚡ Building applications with LLMs through composability ⚡

      • 1595901624/gpt-aggregated-edition : 聚合ChatGPT官方版、ChatGPT免费版、文心一言、Poe、chatchat等多平台，支持自定义导入平台。

      • FreedomIntelligence/LLMZoo : ⚡LLM Zoo is a project that provides data, models, and evaluation benchmark for large language models.⚡ Tech Report

      • shm007g/LLaMA-Cult-and-More : News about 🦙 Cult and other AIGC models.

      • sobelio/llm-chain : llm-chain is a collection of Rust crates designed to help you work with Large Language Models (LLMs) more effectively. llm-chain.xyz

      • ymcui/Chinese-LLaMA-Alpaca : 中文LLaMA&Alpaca大语言模型+本地CPU/GPU部署 (Chinese LLaMA & Alpaca LLMs). 本项目开源了中文LLaMA模型和指令精调的Alpaca大模型。这些模型在原版LLaMA的基础上扩充了中文词表并使用了中文数据进行二次预训练，进一步提升了中文基础语义理解能力。同时，中文Alpaca模型进一步使用了中文指令数据进行精调，显著提升了模型对指令的理解和执行能力。"Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca". (arXiv 2023).

  • Learning Resources

Prompts

Open API

Device Deployment

  • MLC LLM : Enable everyone to develop, optimize and deploy AI models natively on everyone's devices. mlc.ai/mlc-llm

  • Lamini : Lamini: The LLM engine for rapidly customizing models 🦙.

Applications

Blogs