large-scale-data-processing

There are 2 repositories under large-scale-data-processing topic.

NVIDIA-NeMo/Curator
Scalable data pre processing and curation toolkit for LLMs
Language:Python1.2k 19 300187
data-prep-kit/data-prep-kit
Open source project for data preparation for GenAI applications
Language:HTML847 19 519226