modelscope/data-juicer

Issues

• Heavy dependency of Data-Juicer

  #398 opened 11 days ago by BeachWang
  4

• [Feat] Enhance type hints and parameter validation

  #416 opened 25 days ago by drcege
  1

• [Bug]: undefined symbol: _ZN3c104cuda9SetDeviceE

  #419 opened a month ago by lh61500
  1

• 执行 python tools/process_data.py --config train.yaml 命令

  #425 opened a month ago by abchbx
  1

• [Bug]: 去重的hash计算卡在100%上，一直不过滤

  #387 opened a month ago by xiafeng-nb
  6

• AssertionError

  #420 opened a month ago by abchbx
  1

• [Feat] Data-Juicer as a Service

  #417 opened a month ago by drcege
  1

• [Feat]: Add Ray actor support

  #371 opened a month ago by drcege
  2

• Efficient processing OPs for scanned images and pdf

  #375 opened a month ago by yxdyc
  3

• [Feat] Automatically Handle `BrokenPipeError` Caused by Limited Memory

  #377 opened a month ago by yxdyc
  3

• 什么鬼呀，不管是你们huggingface空间还是自己起个服务都运行不起来，demo也运行不起来

  #404 opened a month ago by coder4nlp
  2

• analyzer or analyzer?

  #409 opened a month ago by lilqz66
  1

• [Feat] Support `dj_batched_group_ops` that allows for the configuration and application of multiple operators in smaller, manageable batches

  #413 opened a month ago by yxdyc
  0

• [Feat] Support `PythonCodesOperator` and `BashCodesOperator` that wraps an existing python file, or some code snippets to be executed, such as the existing DJ tools.

  #412 opened a month ago by yxdyc
  0

• 是否可以为一个op设置多个text_key

  #380 opened 2 months ago by lihongxiacream
  1

• Guidance for OP with multiple data fields to be processed

  #411 opened a month ago by yxdyc
  0

• How to download the YoukuMPLUG video dataset?

  #341 opened a month ago by lucasjinreal
  10

• [Bug]: video_split_by_duration_mapper RuntimeError: can't start new thread

  #362 opened a month ago by kike-0304
  3

• [Bug]: librosa not work with np>1

  #372 opened a month ago by drcege
  2

• potential bug of checkpointing

  #337 opened a month ago by drcege
  3

• [Bug]: MODEL_ZOO is not reused in subprocesses

  #370 opened a month ago by drcege
  1

• cache files in /tmp/hf_datasets-*

  #328 opened 2 months ago by simplew2011
  4

• [Bug]: 运行sandbox的时候显示ModuleNotFoundError: No module named 'tools.mm_eval'

  #379 opened 2 months ago by Snow0111
  3

• alphanumeric_filter算子清洗疑问

  #267 opened 6 months ago by echo-valor
  5

• [Bug]: Loading checkpoint shards:的时候直接kill了是什么，是内存不够了吗

  #388 opened 2 months ago by ZHJ19970917
  1

• Confused with the meaning of 'preprocess' time-consuming in the `reproduced_redpajama /README.md`

  #383 opened 2 months ago by flyflypeng
  2

• 图片去重之后保留的是数组前面出现的图片吗？

  #342 opened 2 months ago by HalcyonLiang
  3

• Unable to process data with ray executor on Kubernetes Ray Cluster:

  #345 opened 2 months ago by Fatima-0SA
  14

• [Bug]: 使用图片相关算子在显存充足的情况下 报OOM

  #378 opened 2 months ago by tian969
  4

• [Bug]: Memory leak in video OP

  #369 opened 2 months ago by BeachWang
  0

• 请问如何在自己的文档数据上通过GPT-4o/Claude3.5-Sonnet预标注，清洗，最后制作成一个VQA的数据集？

  #360 opened 2 months ago by moyans
  2

• 如何根据算子提前准备好需要资源?

  #376 opened 2 months ago by tian969
  0

• How to speed up the processing speed of image-text similarity filtering？

  #363 opened 2 months ago by qyx1121
  1

• video_ocr_area_ratio_filter may cause cuda out of memory

  #353 opened 3 months ago by ycwfs
  2

• [Bug]: RuntimeError: SimpleQueue objects should only be shared between processes through inheritance

  #356 opened 3 months ago by MingdongHe
  4

• ram_plus_swin_large_14m.pth invalid

  #350 opened 3 months ago by arturia-Xayah
  2

• How to add a metric to parse from HELM output in wandb_writer.py?

  #330 opened 3 months ago by Mr-lonely0
  2

• [Bug]: bug when use the video_motion_score_filter

  #344 opened 3 months ago by ycwfs
  0

• [Bug]: language_id_score_filter affect the speed of other operators

  #340 opened 3 months ago by simplew2011
  3

• 报”error: Unrecognized arguments: -B -S -I -c“

  #315 opened 4 months ago by HaleYang
  4

• About Quality Classifier

  #321 opened 4 months ago by koanho
  2

• stopwords_filter 为什么是过滤掉小于某个阈值的样本

  #307 opened 4 months ago by noforit
  4

• filter是否支持batch处理，以及怎么设置batch_size？

  #285 opened 5 months ago by Yang-QW
  6

• hash calculate in ray deduplicator

  #286 opened 5 months ago by simplew2011
  5

• 为什么大部分的refined recipe都是用simhash去重？

  #292 opened 5 months ago by sherrytonger
  3

• A Compatibility Issue in Environment Installation of DJ-Sandbox

  #305 opened 5 months ago by lingzhq
  1

• [Bug]: 运行tools/analyze_data.py报错，出现 KeyError: 'text'

  #296 opened 6 months ago by promisecc
  2

• [Question] Can't find evalutor.yaml on the path of `/workspace/data-juicer/demos`

  #298 opened 6 months ago by BenWu11
  1

• Absolute path to relative path for multi-source

  #278 opened 6 months ago by BeachWang
  0

• [Bug]: process on ray occur "TypeError: 'str' object cannot be interpreted as an integer"

  #281 opened 6 months ago by laolv421
  8