[Feature] Support for microbatches
PicoCreator opened this issue · 0 comments
PicoCreator commented
Current the official trainer will outperform infctx-trainer. In part because it supports microbatches.
While infctx has huge compatibility issues with microbatches, it is possible for microbatches to be implemented for non BPTT training runs. Helping mitigate this issue