model-parallelism
There are 37 repositories under model-parallelism topic.
hpcaitech/ColossalAI
Making large AI models cheaper, faster and more accessible
deepspeedai/DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
kakaobrain/torchgpipe
A GPipe implementation in PyTorch
PaddlePaddle/PaddleFleetX
飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。
Oneflow-Inc/libai
LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training
kaiyuyue/torchshard
Slicing a PyTorch Tensor Into Parallel Shards
alibaba/EasyParallelLibrary
Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.
Shenggan/awesome-distributed-ml
A curated list of awesome projects and papers for distributed training or inference
xrsrke/pipegoose
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
hkproj/pytorch-transformer-distributed
Distributed training (multi-node) of a Transformer model
tanyuqian/redco
NAACL '24 (Best Demo Paper RunnerUp) / MlSys @ NeurIPS '23 - RedCoast: A Lightweight Tool to Automate Distributed Training and Inference
NERSC/sc23-dl-tutorial
SC23 Deep Learning at Scale Tutorial Material
vdutts7/dnn-distributed
Distributed training of DNNs • C++/MPI Proxies (GPT-2, GPT-3, CosmoFlow, DLRM)
NERSC/dl-at-scale-training
Deep Learning at Scale Training Event at NERSC
NERSC/dl4sci25-dl-at-scale
Deep learning for science school material 2025
ryantd/veloce
WIP. Veloce is a low-code Ray-based parallelization library that makes machine learning computation novel, efficient, and heterogeneous.
AlibabaPAI/FlashModels
Fast and easy distributed model training examples.
atakehiro/3D-U-Net-pytorch-model-parallel
PyTorch implementation of 3D U-Net with model parallel in 2GPU for large model
ShashankSubramanian/transformer-perf-estimates
Performance Estimates for Transformer AI Models in Science
Shenggan/atp
Adaptive Tensor Parallelism for Foundation Models
fanpu/DynPartition
Official implementation of DynPartition: Automatic Optimal Pipeline Parallelism of Dynamic Neural Networks over Heterogeneous GPU Systems for Inference Tasks
dlzou/computron
Serving distributed deep learning models with model parallel swapping.
garg-aayush/model-parallelism
Model parallelism for NN architectures with skip connections (eg. ResNets, UNets)
dscpesu/NetTorrent
A decentralized and distributed framework for training DNNs
explcre/pipeDejavu
pipeDejavu: Hardware-aware Latency Predictable, Differentiable Search for Faster Config and Convergence of Distributed ML Pipeline Parallelism
LER0ever/HPGO
Development of Project HPGO | Hybrid Parallelism Global Orchestration
AnveshaM/Enhancing-performance-of-big-data-machine-learning-models-on-Google-Cloud-Platform
The project is focused on parallelising pre-processing, measuring and machine learning in the cloud, as well as the evaluation and analysis of the cloud performance.
EunjuYang/distributed-tf
distributed tensorflow (model parallelism) example repository
ngrabaskas/Torch-Automatic-Distributed-Neural-Network
Torch Automatic Distributed Neural Network (TorchAD-NN) training library. Built on top of TorchMPI, this module automatically parallelizes neural network training.
sjlee25/legion-readme
Description of Framework for Efficient Fused-layer Cost Estimation, Legion (2021)
d4l3k/axe
A simple graph partitioning algorithm written in Go. Designed for use for partitioning neural networks across multiple devices which has an added cost when crossing device boundaries.
zhuangsc/altsplit
An MPI-based distributed model parallelism technique for MLP
ankahira/chainermnx
Extended ChainerMN
joelrorseth/HyperTune
A fully distributed hyperparameter optimization tool for PyTorch DNNs
olk/mnist-performance
performance test of MNIST hand writings usign MXNet + TF
sparklerz/petals-llama2-70b
Decentralised inference + Prompt tuning of LLaMA-2-70B with Petals (swarm model-parallelism). Meta-repo with summary and links to the two write-ups.