google-research/deduplicate-text-datasets

Issues

• How to dedup subtring in one dataset?

  #7 opened 3 years ago by lan2016286
  10

• called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }

  #51 opened 7 months ago by bingkunyao
  1

• [Question] An error with the same repo guideline

  #36 opened a year ago by manel-hikk
  5

• Distributed running

  #49 opened 9 months ago by jordane95
  0

• does finish_dedup_wiki40b.py has some wrong?

  #48 opened 9 months ago by mathCrazyy
  0

• does this tool can process Chinese?

  #47 opened 9 months ago by mathCrazyy
  1

• [Bug] Out of range error when counting occurrences on a custom suffix array

  #45 opened 9 months ago by WWWonderer
  1

• Accessing the duplicates and their counts

  #20 opened 2 years ago by yanaiela
  13

• Question: Upper Bound

  #44 opened 10 months ago by bezir
  1

• Count_occurrence does not work with tokenizer?

  #43 opened 10 months ago by WWWonderer
  2

• question about wstring_equal function

  #41 opened 10 months ago by WWWonderer
  2

• 是否可以提供一个纯python版本的，相信很多研究者在服务器上没有权限安装gcc

  #42 opened 10 months ago by gongye19
  1

• when i use tokenizer , I obtained many patterns that span across the data, which is quite strange.

  #39 opened a year ago by gawei1995
  4

• customized dataset deduplication

  #38 opened a year ago by zengyangjie
  1

• where the data is?

  #40 opened a year ago by jianshu93
  1

• Adjust TensorFlow version to fix cuDNN, cuFFT, cuBLAS errors.

  #33 opened a year ago by longxudou
  4

• what is the input dataset format for custom dataset?

  #31 opened a year ago by mittsommer
  1

• how to deduplicate huggingface datasets

  #21 opened a year ago by StephennFernandes
  7

• cargo build error.Could you upload cargo.lock file?

  #30 opened a year ago by jinzhuer
  2

• Incomplete Sentences

  #34 opened a year ago by MiladMolazadeh
  1

• remove_ex in finish_dedup_wiki40b

  #35 opened a year ago by wead-hsu
  1

• How to restore the result data after deduplication (remove invisible characters)

  #29 opened a year ago by greenriver777
  1

• Fix to issue #17 limits cmd_merge to be single-threaded

  #19 opened a year ago by kleinj
  3

• Error when running the code

  #12 opened a year ago by MatthewCYM
  15

• Retain one instance per duplicate

  #32 opened a year ago by RobinQrtz
  2

• RAM crash when use collect method

  #18 opened 3 years ago by acul3
  2

• Inplementation of NearDup(approximate match)

  #27 opened 2 years ago by Yaoming95
  1

• [Paper Question] Why use w-shingles over k-shingles?

  #28 opened 2 years ago by micimize
  1

• Simple test

  #26 opened 2 years ago by KeremTurgutlu
  0

• Off-by-1 error in `collect`?

  #24 opened 2 years ago by ola13
  0

• question about deduplication cluster size

  #23 opened 2 years ago by everks
  2

• one bug when I use

  #17 opened 3 years ago by flyingwaters
  2

• Should newline char be removed

  #16 opened 3 years ago by cperiz
  1

• Unexpected behavior with ending symbols

  #15 opened 3 years ago by mitya52
  2

• "failed to fill whole buffer" errors

  #14 opened 3 years ago by mitya52
  2

• Error with table size not being divisible by text size

  #11 opened 3 years ago by jinyongyoo
  7

• Can the tool run on plain text files?

  #8 opened 3 years ago by m-resta
  20

• false positives

  #9 opened 3 years ago by ChenghaoMou
  1

• How to dedup between two datasets?

  #3 opened 3 years ago by mralexis1
  7

• Error on self deduplication

  #5 opened 3 years ago by zijwang
  10

• Why not use Simhash?

  #4 opened 3 years ago by Ethan-yt
  3