tensor-parallelism

There are 9 repositories under tensor-parallelism topic.

bigscience-workshop/petals
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
Language:Python9.3k 95 203525
InternLM/InternEvo
InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.
Language:Python310 10 8452
kaiyuyue/torchshard
Slicing a PyTorch Tensor Into Parallel Shards
Language:Python296 11 815
xrsrke/pipegoose
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
Language:Python80 4 3218
aniquetahir/JORA
JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)
Language:Python29 2 11
fattorib/transformer_shmap
Tensor Parallelism with JAX + Shard Map
Language:Python11 1 01
AlibabaPAI/FlashModels
Fast and easy distributed model training examples.
Language:Python10 10 04
InternLM/InternEvo-HFModels
Democratizing huggingface model training with InternEvo
Language:Python5 3 04
NiuHuangxiaozi/Deep-Learning-Parallelism
This repository outlines a comprehensive guide for training a distributed deep learning model.
Language:Python1 2 00