Add Pod Orchestration
cfoster0 opened this issue · 1 comments
cfoster0 commented
https://github.com/kingoflolz/mesh-transformer-jax
Use the pod orchestation code from here. Effectively, we should borrow everything and modify the transformer_shard and tfrecord_loader files.
cfoster0 commented
For small (<< 1B parameter) models, it was decided that model parallelism isn't needed. Closing for now.