ChenghaoMou/text-dedup

Issues

• Questions on MinHash Deduplication

  #106 opened 10 days ago by XChen-Zero
  2

• Performance issues

  #105 opened 24 days ago by bowspider-man
  3

• minhash deduplication error

  #104 opened 24 days ago by bowspider-man
  2

• text-dedup

  #96 opened a month ago by HungHoangDinh
  6

• Improve the interpretability of the scripts and results

  #97 opened a month ago by linear
  1

• Improve the readability of the documentation

  #98 opened a month ago by linear
  1

• Improve the usability of the scripts

  #99 opened a month ago by linear
  1

• Fingerprint computation

  #100 opened a month ago by linear
  1

• Clustering

  #101 opened a month ago by linear
  1

• how to dedup short text?

  #103 opened a month ago by varuy322
  1

• JSON File

  #102 opened 3 months ago by qxuanson
  0

• Run MinHash dedup on Multi-Nodes

  #92 opened 3 months ago by alielfilali01
  5

• text-dedup 去重效果怎么样

  #91 opened 3 months ago by maoxiangyi
  2

• Output Detail

  #95 opened 3 months ago by Dodero10
  2

• When I run

  #94 opened 3 months ago by Dodero10
  0

• AttributeError: 'DatasetDict' object has no attribute 'shard' when running SimHash deduplication

  #93 opened 3 months ago by Dodero10
  2

• Can we use it for Arabic text?

  #90 opened 3 months ago by hahmad2008
  2

• Can we consider using dask for distributed deduplication

  #89 opened 4 months ago by simplew2011
  1

• Deduplication results vary depending on whether Spark is used or not.

  #88 opened 4 months ago by cjmp1
  6

• OSError: Memory mapping file failed: Cannot allocate memory

  #87 opened 4 months ago by siebeniris
  4

• 数据读取失败

  #86 opened 5 months ago by programmerLY
  5

• minhash_spark.py [UNABLE_TO_INFER_SCHEMA]

  #85 opened 5 months ago by Yang-QW
  3

• no module named numpy._typing

  #83 opened 6 months ago by Leoooooo123
  2

• Little refactor to allow imports from python instead of cli/subprocess

  #82 opened 6 months ago by wuodar
  3

• how about make a ray executor to deduplication

  #81 opened 6 months ago by simplew2011
  2

• Consistently seeing more rows being dropped in minhash_spark.py compared to minhash.py

  #71 opened 7 months ago by prikmm
  8

• Failed to install using `pip install text-dedup`, but succeeded using `pip install -e .`

  #84 opened 7 months ago by hancheolcho
  1

• The max_iteration for small star and large star in minhash_spark.py

  #80 opened 7 months ago by Jason3900
  3

• Can we accelerate the groupByKey operation by md5 hashing for the Minhash spark version?

  #76 opened 8 months ago by 311dada
  2

• Add back support for across-similar in suffix array

  #42 opened 8 months ago by ChenghaoMou
  1

• boundaries of sub-strings

  #73 opened 8 months ago by MiladMolazadeh
  2

• the effect of min_length in minhash_spark.py/minhash.py

  #77 opened 10 months ago by Jason3900
  4

• Error when running on Windows10

  #50 opened a year ago by soonjune
  5

• Papers, Datasets that use this repo

  #65 opened a year ago by chris-ha458
  3

• FileNotFoundError when run `Finding clusters` step.

  #67 opened a year ago by Ox0400
  2

• many duplicate pairs were not actually similar using minhash_spark.py

  #70 opened a year ago by dbc1040
  1

• PySpark without DataProc

  #64 opened a year ago by scheiblr
  4

• Deduplication of union find clusters explained

  #62 opened a year ago by ZJaume
  2

• Python 3.9 compatibility

  #59 opened a year ago by ZJaume
  2

• refactor hash related code

  #29 opened a year ago by chris-ha458
  3

• Why using Set instead of List for list ngrams in Minhash?

  #52 opened a year ago by Flamentt
  1

• Open up more avenues for discussion

  #46 opened a year ago by chris-ha458
  2

• Suffix array collect src/main.rs:174 assertion failed: input.len() % size_width == 0

  #47 opened a year ago by leoMesss
  2

• Load data from serveral workers with minhash_pyspark.py

  #43 opened a year ago by nguyenhuuthuat09
  7

• Is that minhash oom error normal ?

  #41 opened a year ago by BillZid
  3

• FileNotFoundError: [Errno 2] No such file or directory: 'output/temp_text.txt.part.0-7320579'

  #38 opened a year ago by listentomi
  4

• User-controlled cache files clean-up

  #39 opened a year ago by ChenghaoMou
  0

• Can the suffix_array here be used for Chinese encoded in UTF-8?

  #33 opened a year ago by BillZid
  11

• Should min_hash and min_hash_spark return the same result?

  #28 opened a year ago by nguyenhuuthuat09
  2

• suffix array "No such file or directory" exception

  #27 opened a year ago by yusufcakmakk
  2