数据处理部分可以提供更详细的说明吗？
wowfingerlicker opened this issue · 1 comments
wowfingerlicker commented
谢谢！很好的工作！
pie33000 commented
My apologies for the delay in responding. I utilized the following example to process the fine web dataset: https://github.com/karpathy/build-nanogpt/blob/master/fineweb.py