huggingface/datasets

Concurrent loading in `load_from_disk` - `num_proc` as a param

Closed this issue · 0 comments

Feature request

#6464 mentions a num_proc param while loading dataset from disk, but can't find that in the documentation and code anywhere

Motivation

Make loading large datasets from disk faster

Your contribution

Happy to contribute if given pointers