pytorch/torchtitan

Wrong mesh order

ad8e opened this issue · 1 comments

ad8e commented

torchtitan/torchtitan/parallelisms/__init__.py

Line 46 in d442743

[self.dp, self.tp, self.pp], ["dp", "tp", "pp"], strict=True

Should be PP DP TP. This matters for NUMA across nodes.

wanchaol commented

good catch! submitting a fix soon