Closed this issue a year ago · 1 comments
llm-foundry enjoys a benefit when the corpus provides a sequence of token IDs.
Include the token_ids field in the resultant corpus.
token_ids
93b66db