Chimera is novel pipeline parallelism approach, which is proposed for efficiently training large-scale neural network models (e.g., BERT, GPT-2/3) on parallel machines (e.g., GPU clusters). The key idea of Chimera is to reduce the number of bubbles in the pipeline, without introducing staleness in the training process. Our implementation (SC'21) was based on PyTorch and adapted from the PipeDream. We use GLOO as the distributed backend.
A new (concise and also fully-fledged) verion of Chimera will be added in the Chimera-BERT branch.
chimera/chimera_bert
Bert in Chimera.
chimera/chimera_gpt2
GPT-2 in Chimera.
chimera/chimera_pipes
Chimera generalized to more than two pipelines.
chimera/performance_model
Performance modelling for communications.
To install the required Python modules:
conda create --name py37 python=3.7
source activate py37
pip install -r requirements.txt
We run experiments on GPU clusters with SLURM job scheduler. For example, one can submit a job to the job queue by
cd ./job_scripts
sbatch daint_bert48_32nodes_chimera_4w8d.sh
Chimera is pulished in SC'21, Best Paper Finalist. See the paper and the video talk for more details. To cite our work:
@inproceedings{li143,
author = {Li, Shigang and Hoefler, Torsten},
title = {Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines},
year = {2021},
isbn = {9781450384421},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3458817.3476145},
doi = {10.1145/3458817.3476145},
booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
articleno = {27},
numpages = {14},
location = {St. Louis, Missouri},
series = {SC '21}
}
See LICENSE.