NVIDIA/NeMo-Curator

Issues

• Default int_to_str_id error in jaccard shuffle

  #150 opened 2 months ago by yyu22
  0

• Running into OOM with add id

  #142 opened 2 months ago by yyu22
  2

• [BUG] Jaccard Shuffle error if merge result is empty

  #49 opened 4 months ago by ayushdg
  2

• Add HuggingFace support for Quality Classifier

  #167 opened a month ago by sarahyurick
  1

• Error starting dask cuda worker with nemo framework container

  #143 opened a month ago by yyu22
  3

• Given JSON input is too large when using "cudf" backend

  #174 opened a month ago by Edfame
  2

• Incorrect Shuffle results with dask-cuda 24.06 & above

  #134 opened 2 months ago by ayushdg
  2

• [FEA] Add support for Multiple Model Quality Classification

  #70 opened 4 months ago by sarahyurick
  1

• Reading compressed json files

  #145 opened a month ago by yyu22
  2

• [IMP] Hard code labels for the quality and domain classifier

  #71 opened a month ago by VibhuJawa
  0

• [FEA] Make classifiers work with HuggingFace directly

  #72 opened a month ago by VibhuJawa
  5

• Remove duplicated assigments in GeneralCommentToCodeFilter

  #161 opened a month ago by miguelusque
  0

• Remove text field requirement from Download and Extract

  #158 opened a month ago by ryantwolf
  0

• handling inconsistent metadata

  #146 opened 2 months ago by yyu22
  0

• KeyError in Map Buckets

  #149 opened 2 months ago by yyu22
  0

• [FEA] Add batched files reading to separate_by_metadata.py

  #53 opened 4 months ago by miguelusque
  1

• Unable to install[BUG]

  #76 opened 2 months ago by mike2463
  3

• Add pytests for semantic dedup

  #141 opened 2 months ago by ayushdg
  0

• Broken Tutorial: single_gpu_tutorial.ipynb

  #105 opened 2 months ago by axelmagn
  3

• [FEA] Refactor how arguments are added to scripts

  #69 opened 2 months ago by miguelusque
  1

• [FEA] Support an installation path using RAPIDS nightly packages & source installs of Curator

  #133 opened 2 months ago by randerzander
  0

• ValueError: More than one filename found in partition

  #89 opened 2 months ago by aasthajh
  1

• [FEA] Improve download and extract utility

  #80 opened 2 months ago by ryantwolf
  0

• Import fails on cpu

  #109 opened 2 months ago by yyu22
  0

• [FEA] Add nemo_curator.__version__ and also display it in command line when invoked --help

  #132 opened 2 months ago by miguelusque
  0

• [BUG] Fuzzy deduplication fails on datasets with no duplicates

  #67 opened 4 months ago by Maghoumi
  1

• FastTextQualityFilter model file release

  #121 opened 3 months ago by simplew2011
  1

• Where to obtain domain and quality model files?

  #86 opened 3 months ago by randerzander
  2

• shuffle datasets

  #115 opened 3 months ago by simplew2011
  3

• README.md broken links

  #116 opened 3 months ago by miguelusque
  0

• NnoeAlphaNumericFilter has impelemented wrong[BUG]

  #52 opened 3 months ago by zahramahani
  2

• Remove Numpy<2.0 pin

  #120 opened 3 months ago by ayushdg
  0

• exact_deduplication.py out_of_memory

  #112 opened 3 months ago by simplew2011
  6

• Update tutorial & Slurm documentation

  #108 opened 3 months ago by ayushdg
  0

• fuzzy dedup in cpu

  #101 opened 3 months ago by simplew2011
  0

• Update download documentation to include client creation

  #100 opened 3 months ago by moutasemalakkad
  3

• [FEA] Support dask query planning

  #73 opened 3 months ago by ayushdg
  1

• [DOC] Incorrect reference to domain_classifier_example.py

  #91 opened 3 months ago by miguelusque
  0

• [FEA] Add examples showing how to use both CPU & GPU modules together

  #65 opened 4 months ago by ayushdg
  1

• [DOC] Add rapids nightly install instructions for curator

  #87 opened 3 months ago by VibhuJawa
  0

• find_pii_and_deidentify example fails

  #85 opened 3 months ago by randerzander
  1

• [FEA] Raise a warning when creating FuzzyDuplicatesConfig with non-empty cache_dir

  #84 opened 3 months ago by randerzander
  1

• [FEA] Add --meta parameter to explicitly specify the jsonl field dtypes

  #63 opened 3 months ago by miguelusque
  2

• [BUG] Better error/checks around input types being CPU/GPU

  #79 opened 3 months ago by ayushdg
  0

• [FEA] Update read_json to work with s3 paths.

  #66 opened 4 months ago by ayushdg
  0

• [BUG] OOM errors while running Deduplication

  #59 opened 4 months ago by vipulraheja
  1

• [BUG] importing spacy before cluster creation leads on only 1 GPU being used.

  #64 opened 4 months ago by ayushdg
  0

• [BUG] Jaccard Shuffle error if shuffled_docs.parquet data already exists and has been written.

  #51 opened 4 months ago by ayushdg
  0

• Better mimic DocumentDataset's `read_*` functions to Dask's `read_*` functions

  #50 opened 4 months ago by sarahyurick
  1

• Fuzzy dedup error if partition wise indices do not start from 0

  #48 opened 4 months ago by ayushdg
  0