semantic-deduplication

There are 2 repositories under semantic-deduplication topic.

NVIDIA-NeMo/Curator
Scalable data pre processing and curation toolkit for LLMs
Language:Python1.2k 18 262180
MinishLab/semhash
Fast Semantic Text Deduplication & Filtering
Language:Python810 6 2150