how to acquire the real whole batch sequenece training loss(reduction_mode=mean) ?

[littttttlebird]

in the train.py, the loss return from main process is the loss of one sequence block, not the whole sequence loss.

[jzhang38]

EasyContext/train.py

Line 150 in 01a9360

gathered_loss = accelerator.reduce(loss.clone().detach(), "mean")

It is the whole sequence loss?

[littttttlebird]

3Q very much