NVIDIA/NeMo-Curator

Issues

• Graceful handling when no LSH duplicates found.

  #381 opened 20 days ago by davzoku
  3

• Installation Error with mamba-ssm while Installing nemo-curator

  #399 opened 20 days ago by piyushkashyap07
  4

• Add Jupyter notebook tutorials for data classifiers

  #406 opened 21 days ago by sarahyurick
  0

• Add CI tests for Hugging Face classifiers

  #405 opened 21 days ago by sarahyurick
  0

• Translation example with ctranslate2's Translator.

  #246 opened 3 months ago by uahmed93
  5

• Add Trafilatura text extraction

  #400 opened 22 days ago by sarahyurick
  0

• Retire text_bytes_aware_shuffle and directly use shuffle

  #240 opened a month ago by VibhuJawa
  1

• Connected Components Speedup breaks tutorial examples

  #380 opened a month ago by davzoku
  2

• Improve Pytorch Model Performence

  #329 opened 2 months ago by VibhuJawa
  1

• Update `columns` documentation

  #378 opened a month ago by sarahyurick
  0

• Use CrossFit for `TokenizerFertilityFilter`

  #377 opened a month ago by sarahyurick
  0

• Add GPU test with NeMo 2.0

  #376 opened a month ago by sarahyurick
  0

• [IMP] Decrease Merge Peak Memory Usage of ConnectedComponents

  #375 opened a month ago by VibhuJawa
  0

• Add more comprehensive `DocumentDataset` PyTests

  #371 opened a month ago by sarahyurick
  0

• Data Curation Failure in NVIDIA NeMo Docker Container (nvcr.io/nvidia/nemo:24.07) Using DataCurator with JSONL Files

  #348 opened a month ago by MalekSolta
  4

• Cannot run `gpu_exact_dups` due to deprecated flag

  #350 opened a month ago by davzoku
  1

• Regarding the issue of AEGIS model labels

  #349 opened a month ago by xiafeng-nb
  2

• GitHub workflows improvements

  #259 opened 3 months ago by sarahyurick
  5

• Zyda2 tutorial - key error when running compute_counts script

  #345 opened 2 months ago by ronjer30
  0

• Zyda2 tutorial - TypeError when initializing Dask CPU cluster

  #344 opened 2 months ago by ronjer30
  0

• Remove hardcoded "text" column in `_run_classifier_helper` function

  #333 opened 2 months ago by sarahyurick
  0

• Japanese support for get_word_splitter

  #257 opened 2 months ago by oumugai
  2

• semantic_dedupe runs into IndexError: list index out of range

  #341 opened 2 months ago by ruchaa-apte
  1

• Faster/More efficient duplicate removal for exact/fuzzy dedup.

  #335 opened 2 months ago by ayushdg
  1

• Standardize `text_field`, `id_field`, etc. terminology

  #342 opened 2 months ago by sarahyurick
  0

• Deprecate `max_text_bytes_per_part`

  #331 opened 2 months ago by sarahyurick
  1

• Make `max_text_bytes_per_part` configurable

  #233 opened 2 months ago by ayushdg
  1

• Check Pytorch cuda context is valid across GPUs

  #284 opened 2 months ago by VibhuJawa
  2

• Improve NeMo Curator Experience for Pytorch Models (with crossfit)

  #288 opened 2 months ago by VibhuJawa
  1

• Add `memory_limit` to `get_client` function

  #324 opened 2 months ago by sarahyurick
  1

• ADD DCO signing as a pre-commit check

  #290 opened 2 months ago by VibhuJawa
  1

• Write to file w/o including `filename` column

  #293 opened 2 months ago by joshwyatt
  0

• Use `overwrite=True` when writing out intermediate files

  #315 opened 2 months ago by praateekmahajan
  0

• [FEA] Assert that on connecting to cluster you have at least > 1 workers

  #216 opened 2 months ago by VibhuJawa
  0

• Curator should support numpy > 2

  #287 opened 2 months ago by praateekmahajan
  1

• NEMO Curator Not Extracting Thai Language Content

  #281 opened 2 months ago by ThittipatBeam
  0

• Unmanaged memory is high and frozen execution

  #295 opened 2 months ago by pappagari
  3

• Unable to run distributed_data_classification tutorial

  #296 opened 2 months ago by pappagari
  4

• Cleanup Github Branches

  #289 opened 2 months ago by VibhuJawa
  1

• [DOC] Add documentation for fineweb edu classifier

  #248 opened 2 months ago by VibhuJawa
  0

• Resuming the job on slurm after it gets cancelled.

  #297 opened 2 months ago by uahmed93
  0

• Expand RMM options for Python API

  #260 opened 2 months ago by sarahyurick
  1

• Semantic Dedup doesn't work with UCX

  #283 opened 3 months ago by praateekmahajan
  1

• Installation error: pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000

  #278 opened 3 months ago by pappagari
  3

• [FEA] Improve separate_by_metadata performance when dealing with jsonl files

  #255 opened 3 months ago by miguelusque
  0

• Unknown affinity parameter in SDG-Retriever_Eval_Tutorial notebook

  #264 opened 3 months ago by miguelusque
  0

• Exact/fuzzy deduplication bug

  #252 opened 3 months ago by yyu22
  1

• Grammar and punctuation nits in Jupyter Notebooks

  #228 opened 4 months ago by sarahyurick
  3

• Re-add `test_uneven_common_crawl_range` PyTest

  #236 opened 3 months ago by sarahyurick
  0

• Explore Dask jobque's slurm runner for multi node slurm setups.

  #215 opened 4 months ago by ayushdg
  1