数据集过大导致加载数据集时内存爆掉，似乎在哪里看到可以直接加载tokenize之后的数据进行训练
Closed this issue · 1 comments
Victoriaheiheihei commented
Reminder
- I have read the README and searched the existing issues.
Reproduction
想要直接加载tokenized 之后的文本进行训练，但是招不到具体使用方法写在哪里了，请问有具体的地址吗
Expected behavior
No response
System Info
No response
Others
No response