/origin-llm

llm相关内容，包括：基础知识、八股文、面经、经典论文

llm-origin

本文将 LLM 知识分为以下七个部分，并整理了相关资源：

  • LLM Overview：汇总有关 LLM 的综述和系统课程。
  • LLM Fundamentals：学习 LLM 之前需要掌握的基础知识，包括数学、Python、深度学习框架、机器学习、深度学习和NLP等。
  • Create LLM：专注于创建和优化 LLM 所需的技术。
  • Use LLM：专注于开发基于 LLM 的应用并进行部署。
  • LLM Applications：基于 LLM 构建的应用。
  • 面试准备：整理与面试相关的八股文和面经。
  • 其他：如语音、视频等。

LLM Overview

汇总LLM相关的综述、系统课程。

Survey

Course

  • llm-course(32.1k) by mlabonne: Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. mlabonne.github.io/blog.
  • LLMs-from-scratch(17.2k): Implementing a ChatGPT-like LLM in PyTorch from scratch, step by step.
  • Awesome-LLM(15.2k): a curated list of Large Language Model.
  • llm-cookbook(9.9k): 面向开发者的 LLM 入门教程，吴恩达大模型系列课程中文版.
  • LLMSurvey(9.2k): The official GitHub page for the survey paper "A Survey of Large Language Models".
  • LLMsPracticalGuide(8.8k): A curated list of practical guide resources of LLMs (LLMs Tree, Examples, Papers)
  • llm-action(6.9k): 本项目旨在分享大模型相关技术原理以及实战经验。

LLM Fundamentals

数学

Python

深度学习框架

机器学习

深度学习

NLP

Create LLM

Transformer

Attention Is All You Need(2017), paper, arxiv.

Understand:

Use:

LLM architecture

对于Transformer架构，需要对其输入（tokens）和输出（logits）有一个良好的理解。注意力机制是另一个需要掌握的关键组件，因为其改进版本会在后续介绍。

  • 总览：理解编码器-解码器Transformer架构，特别是decoder-only GPT架构，它在每个现代大语言模型中使用。
  • 分词(Tokenization)：了解如何将原始文本数据转换为模型能够理解的格式，这涉及将文本拆分为tokens（通常是单词或子词）。
  • 注意力机制：掌握注意力机制背后的理论，包括自注意力和缩放点积注意力，这些机制使模型在生成输出时能够关注输入的不同部分。
  • 文本生成：了解模型生成输出序列的不同方式。常见的策略包括贪婪解码、束搜索(beam search)、top-k采样和核采样(nucleus sampling)。

References:

  • The Illustrated GPT-2 by Jay Alammar: focused on the GPT architecture, which is very similar to Llama's.
  • LLM Visualization by Brendan Bycroft: Incredible 3D visualization of what happens inside of an LLM.
  • nanoGPT by Andrej Karpathy: A 2h-long YouTube video to reimplement GPT from scratch (for programmers).
  • Decoding Strategies in LLMs: Provide code and a visual introduction to the different decoding strategies to generate text.

数据集(Dataset)

  • firecrawl(5.3k) by mendableai: Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
  • EasySpider(26.7k) by NaiboWang: A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
  • LLMDataHub(2.1k) by Zjh-819: A quick guide (especially) for trending instruction finetuning datasets.

预训练

  • llm.c(20k) by karpathy: LLM training in simple, raw C/CUDA. 用1000行C代码在笔记本电脑完成GPT2的训练，有助于初学者了解大模型底层的原理和知识点。

LLM汇总

  • Awesome-Chinese-LLM(12k) by HqWu-HITCS: 整理开源的中文大语言模型，以规模较小、可私有化部署、训练成本较低的模型为主，包括底座模型，垂直领域微调及应用，数据集与教程等。
  • open-llms(10.4k) by eugeneyan: A list of open LLMs available for commercial use.

Llama:

MultiModal:

  • MiniCPM-V(4.4k) by OpenBMB: MiniCPM-Llama3-V 2.5: A GPT-4V Level Multimodal LLM on Your Phone.
  • InternVL(3k) by OpenGVLab: [CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4V. 接近GPT-4V表现的可商用开源多模态对话模型.
  • CogVLM2(1k) by THUDM: GPT4V-level open-source multi-modal model based on Llama3-8B.

微调(Finetune)

  • LLaMA-Factory(23.4k) by hiyouga: Unify Efficient Fine-Tuning of 100+ LLMs.

peft

Use LLM

prompt

  • fabric github/13.3k - About fabric is an open-source framework for augmenting humans using AI. It provides a modular framework for solving specific problems using a crowdsourced set of AI prompts that can be used anywhere. 可以选择并生成上百个场景的提示词。

RAG

  • llama_index github/32.2k - LlamaIndex is a data framework for your LLM applications. 连接私有数据库和LLM。
  • Verba github/4.4k - Retrieval Augmented Generation (RAG) chatbot powered by Weaviate.
  • WrenAI github/834 - WrenAI makes your database RAG-ready. Implement Text-to-SQL more accurately and securely. getwren.ai.
  • llmware 4k - Unified framework for building enterprise RAG pipelines with small, specialized models.

agent

  • dify github/31.8k - Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production. dify.ai.
  • FinRobot github/673 - FinRobot: An Open-Source AI Agent Platform for Financial Applications using LLMs. ai4finance.org.
  • CopilotKit github/6983 - A framework for building custom AI Copilots 🤖 in-app AI chatbots, in-app AI Agents, & AI-powered Textareas. copilotkit.ai.
  • bisheng github/7.4k - Bisheng is an open LLM devops platform for next generation AI applications. 毕昇.
  • SWE-agent github/11.3k - SWE-agent takes a GitHub issue and tries to automatically fix it, using GPT-4, or your LM of choice. It solves 12.47% of bugs in the SWE-bench evaluation set and takes just 1.5 minutes to run.
  • phidata github/9.4k - Build AI Assistants with memory, knowledge and tools.
  • gpt-researcher github/12k - GPT based autonomous agent that does online comprehensive research on any given topic.
  • MemGPT github/10.4k - Create LLM agents with long-term memory and custom tools.
  • maestro github/2k - A framework for Claude Opus to intelligently orchestrate subagents.
  • anything-llm 15.3k - The all-in-one Desktop & Docker AI application with full RAG and AI Agent capabilities. useanything.com.
  • embedchain 8.7k - Personalizing LLM Responses.

workflow

Inference optimization

  • ipex-llm 6.1k - Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Baichuan, Mixtral, Gemma, Phi, etc.) on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, DeepSpeed, vLLM, FastChat, Axolotl, etc.
  • web-llm 10.9k - High-performance In-browser LLM Inference Engine. webllm.mlc.ai.

Deploying LLMs

  • mergekit 3.8k, Tools for merging pretrained large language models.
  • llm github/3.2k - Access large language models from the command-line. llm.datasette.io.
  • OpenLLM 9k, BentoML, Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint in the cloud.
  • mlc-llm 17.4k - Universal LLM Deployment Engine with ML Compilation. llm.mlc.ai.
  • self-llm 4.8k, 《开源大模型食用指南》基于Linux环境快速部署开源大模型，更适合**宝宝的部署教程

Securing LLMs

Applications

chat

search

  • khoj github/10.7k - Your AI second brain. Get answers to your questions, whether they be online or in your own notes. Use online AI models (e.g gpt4) or private, local LLMs (e.g llama3). Self-host locally or use our cloud instance. Access from Obsidian, Emacs, Desktop app, Web or Whatsapp. khoj.dev.
  • Perplexica github/6.8k - Perplexica is an AI-powered search engine. It is an Open source alternative to Perplexity AI.
  • farfalle github/1.6k - AI search engine - self-host with local or cloud LLMs. www.farfalle.dev.

agent

  • ragapp github/2.1k - The easiest way to use Agentic RAG in any enterprise.
  • GPTS - You can now create custom versions of ChatGPT that combine instructions, extra knowledge, and any combination of skills.

Virtual Human Generation

  • V-Express github/611 - V-Express aims to generate a talking head video under the control of a reference image, an audio, and a sequence of V-Kps images. 音频控制肖像生成视频。
  • MusePose github/854 - MusePose: a Pose-Driven Image-to-Video Framework for Virtual Human Generation. 姿势驱动图片生成跳舞视频。

code

  • openui github/15.6k - OpenUI let's you describe UI using your imagination, then see it rendered live.

面试

Others

tts

ChatTTS

GPT-SoVITS

  • GPT-SoVITS github/25k - 1 min voice data can also be used to train a good TTS model! (few shot voice cloning)

OpenVoice

CV