pipeline-parallelism

There are 27 repositories under pipeline-parallelism topic.

  • ColossalAI

    hpcaitech/ColossalAI

    Making large AI models cheaper, faster and more accessible

    Language:Python41.2k3901.7k4.5k

  • deepspeedai/DeepSpeed

    DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

    Language:Python40.1k3533.2k4.6k

  • bigscience-workshop/petals

    🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

    Language:Python9.8k99208572
  • torchgpipe

    kakaobrain/torchgpipe

    A GPipe implementation in PyTorch

    Language:Python854323499

  • PaddlePaddle/PaddleFleetX

    飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。

    Language:Python47522115166

  • Coobiw/MPP-LLaVA

    Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train your own 8B/14B LLaVA-training-like MLLM on RTX3090/4090 24GB.

    Language:Jupyter Notebook47263823

  • Oneflow-Inc/libai

    LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training

    Language:Python408417956

  • InternLM/InternEvo

    InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

    Language:Python407109169

  • alibaba/EasyParallelLibrary

    Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.

    Language:Python268121049

  • Shenggan/awesome-distributed-ml

    A curated list of awesome projects and papers for distributed training or inference

    24310027

  • torchpipe/torchpipe

    Serving Inside Pytorch

    Language:C++1635713

  • ai-decentralized/BloomBee

    Decentralized LLMs fine-tuning and inference with offloading

    Language:Python9813

  • xrsrke/pipegoose

    Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

    Language:Python8733219

  • AlibabaPAI/DAPPLE

    An Efficient Pipelined Data Parallel Approach for Training Large Model

    Language:Python7712017

  • ParCIS/Chimera

    Chimera: bidirectional pipeline parallelism for efficiently training large-scale models.

    Language:Python66238

  • saareliad/FTPipe

    FTPipe and related pipeline model parallelism research.

    Language:Python42357

  • gty111/gLLM

    gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling

    Language:Python41

  • MLSysU/TD-Pipe

    A Throughput-Optimized Pipeline Parallel Inference System for Large Language Models

    Language:Python37

  • nawnoes/pytorch-gpt-x

    Implementation of autoregressive language model using improved Transformer and DeepSpeed pipeline parallelism.

    Language:Python32203

  • fanpu/DynPartition

    Official implementation of DynPartition: Automatic Optimal Pipeline Parallelism of Dynamic Neural Networks over Heterogeneous GPU Systems for Inference Tasks

    Language:Python7100

  • garg-aayush/model-parallelism

    Model parallelism for NN architectures with skip connections (eg. ResNets, UNets)

    Language:Python5100
  • torchpipe.github.io

    torchpipe/torchpipe.github.io

    Docs for torchpipe: https://github.com/torchpipe/torchpipe

    Language:MDX4001

  • explcre/pipeDejavu

    pipeDejavu: Hardware-aware Latency Predictable, Differentiable Search for Faster Config and Convergence of Distributed ML Pipeline Parallelism

    Language:Jupyter Notebook3100

  • LER0ever/HPGO

    Development of Project HPGO | Hybrid Parallelism Global Orchestration

    3200

  • joe0731/hf_vram_calc

    A CLI tool for estimating GPU VRAM requirements for Hugging Face models, supporting various data types, parallelization strategies, and fine-tuning scenarios like LoRA.

    Language:Python1

  • 1set-t/ai-model

    Industrial-grade weather visualization system that transforms AI model predictions into professional meteorological plots, emphasizing operational forecasting capabilities.

  • sparklerz/multigpu-llm-finetuning

    This repository showcases hands-on projects leveraging distributed multi-GPU training to fine-tune large language models (LLMs).

    Language:Python