Zero3 train: Invalidate trace cache @ step 391: expected module 25, but got module 5
Zhiyuan-Fan opened this issue · 0 comments
Zhiyuan-Fan commented
{'loss': 0.8527, 'learning_rate': 1.282051282051282e-07, 'epoch': 0.0}
{'loss': 0.9276, 'learning_rate': 2.564102564102564e-07, 'epoch': 0.0}
0%| | 2/5194 [00:26<18:22:29, 12.74s/it]
Invalidate trace cache @ step 391: expected module 25, but got module 5