NVIDIA/NeMo-Curator
Scalable data pre processing and curation toolkit for LLMs
Jupyter NotebookApache-2.0
Issues
- 0
Default int_to_str_id error in jaccard shuffle
#150 opened by yyu22 - 2
Running into OOM with add id
#142 opened by yyu22 - 2
- 1
Add HuggingFace support for Quality Classifier
#167 opened by sarahyurick - 3
- 2
- 2
- 1
- 2
Reading compressed json files
#145 opened by yyu22 - 0
- 5
- 0
- 0
Remove text field requirement from Download and Extract
#158 opened by ryantwolf - 0
handling inconsistent metadata
#146 opened by yyu22 - 0
KeyError in Map Buckets
#149 opened by yyu22 - 1
- 3
Unable to install[BUG]
#76 opened by mike2463 - 0
Add pytests for semantic dedup
#141 opened by ayushdg - 3
Broken Tutorial: single_gpu_tutorial.ipynb
#105 opened by axelmagn - 1
- 0
[FEA] Support an installation path using RAPIDS nightly packages & source installs of Curator
#133 opened by randerzander - 1
- 0
[FEA] Improve download and extract utility
#80 opened by ryantwolf - 0
Import fails on cpu
#109 opened by yyu22 - 0
[FEA] Add nemo_curator.__version__ and also display it in command line when invoked --help
#132 opened by miguelusque - 1
- 1
FastTextQualityFilter model file release
#121 opened by simplew2011 - 2
- 3
shuffle datasets
#115 opened by simplew2011 - 0
README.md broken links
#116 opened by miguelusque - 2
- 0
Remove Numpy<2.0 pin
#120 opened by ayushdg - 6
exact_deduplication.py out_of_memory
#112 opened by simplew2011 - 0
Update tutorial & Slurm documentation
#108 opened by ayushdg - 0
fuzzy dedup in cpu
#101 opened by simplew2011 - 3
- 1
[FEA] Support dask query planning
#73 opened by ayushdg - 0
- 1
- 0
- 1
find_pii_and_deidentify example fails
#85 opened by randerzander - 1
[FEA] Raise a warning when creating FuzzyDuplicatesConfig with non-empty cache_dir
#84 opened by randerzander - 2
- 0
- 0
[FEA] Update read_json to work with s3 paths.
#66 opened by ayushdg - 1
[BUG] OOM errors while running Deduplication
#59 opened by vipulraheja - 0
- 0
[BUG] Jaccard Shuffle error if shuffled_docs.parquet data already exists and has been written.
#51 opened by ayushdg - 1
Better mimic DocumentDataset's `read_*` functions to Dask's `read_*` functions
#50 opened by sarahyurick - 0