google-research/text-to-text-transfer-transformer

Question about cross-node(multi-node) data parallelism on GPU

hwyFighting opened this issue · 1 comments

hwyFighting commented

Hi! I would like to ask how to get t5 to train data in parallel across nodes on GPUs?

Is there a corresponding solution in the case of mesh_tensorflow based? Or there is no corresponding design yet?

Thank you for your positive and exact reply!