mlfoundations/open_lm

Standardize tokenization for json and txt datasets

sagadre opened this issue · 0 comments

sagadre commented
Standardize tokenization for json and txt datasets