TBPTT

[shenqiwang27]

Hi Magehrig could you tell me where is the TBPTT dataloader in your code? I didn't find it

[magehrig]

This function creates the 'streaming' datasets.

• If you create the training dataset with this function, it will eventually use the ConcatStreamingDataPipe.
• If you create the validation dataset with this function, it will eventually use the ShardedStreamingDataPipe.

If you look further into the code you can find the DataModule that is use by Pytorch-Lightning to handle the dataloading. The datamodule that I wrote can return 2 dataloaders during training. One dataloader returns random samples (for BPTT) and another one streams sequences (for TBPTT). For evaluation, we only use the streaming dataloader.
Finally, in the training loop, we merge the batches from both dataloaders here.

Let me know if I answered your question.

[shenqiwang27]

Thanks a lot for your reply, I'll close the issue