About preprocessing before input into BART
Junpliu opened this issue · 1 comments
Junpliu commented
Hi, I noticed that utterances are truncated after 1024 tokens as described in (Part D, Implementation Details).
I want to know do I need to add role information when con-cating utterances into one sentence? or do the 1024 tokens includes the role information? Please tell me more details about the preprocessing.
Thanks so much!
zcgzcgzcg1 commented
The truncation happens after role information is added.