allenai/dolma

Issues

• dedupe.documents.attribute_name does not work

  #166 opened a month ago by mathCrazyy
  0

• make_wikipedia.py fails on linux

  #58 opened 8 months ago by peterbjorgensen
  10

• Running paragraph level deduplication on c4

  #150 opened 2 months ago by andrewhojel
  2

• Clarification Needed on "C4 NoPunc" in Data Processing

  #162 opened a month ago by codefly13
  1

• Need help for installing dolma

  #158 opened 2 months ago by mihara-bot
  2

• Duplicate ids in Dolma v1.7

  #157 opened 2 months ago by Vedaad-Shakib
  0

• dtype option is not working as expected

  #152 opened 2 months ago by tokenizer-decode
  1

• Inquiry about Web Pipeline Availability

  #151 opened 2 months ago by codefly13
  1

• Simplify how rules in the mixer are provided

  #50 opened 9 months ago by soldni
  4

• S3 mixer doesn't start

  #143 opened 3 months ago by marcopasqua
  2

• deduplication examples does not work

  #96 opened 3 months ago by TTTTao725
  5

• make_wikipedia in getting_started.md

  #125 opened 3 months ago by leeparkuky
  3

• Is there a way to intergratge Dolma toolkit to Spark?

  #136 opened 3 months ago by DangoWang
  1

• Some race condition in url taggers

  #138 opened 3 months ago by peterbjorgensen
  1

• Data out of bounds when using ‘dolma tokens --dtype uint32’

  #142 opened 3 months ago by Jackwaterveg
  1

• Possible bug in `local_shuffle`?

  #139 opened 3 months ago by hwijeen
  2

• A Question about the meaning of dolma_v1.6_cc_en

  #134 opened 3 months ago by aleien95
  1

• make_wikipedia.py: long running time

  #121 opened 5 months ago by chschroeder
  3

• Support providing streams into mixer via CLI

  #130 opened 4 months ago by soldni
  0

• tagger_modules do not work in current git version

  #128 opened 4 months ago by peterbjorgensen
  1

• Add option to exclude empty documents in the mixer output

  #51 opened 4 months ago by soldni
  0

• How does Exact paragraph deduplication performed?

  #111 opened 4 months ago by silverriver
  1

• not_alphanum_paragraph_v1 tagger takes forever to run on certain inputs.

  #123 opened 4 months ago by peterbjorgensen
  1

• Can I use the dolma toolkit to process my own datasets?

  #127 opened 4 months ago by Tendo33
  1

• Provenance license?

  #108 opened 5 months ago by boxabirds
  1

• Tokenizer name or path must be found error

  #110 opened 5 months ago by RohitRathore1
  3

• Data sheet link in README is broken

  #106 opened 5 months ago by simonw
  1

• Only the attributes written by the last tagger in the tagger list gets written in version 1.0.0

  #113 opened 5 months ago by peterbjorgensen
  0

• Progress Bar may use more resources than necessary

  #77 opened 5 months ago by soldni
  0

• Latest version is not on PyPi

  #78 opened 5 months ago by KennethEnevoldsen
  3

• When will a version that can add tagger be released?

  #93 opened 7 months ago by suolyer
  0

• The Law School Admission Council | LSAC

  #89 opened 7 months ago by hannahzacharski55
  0

• Child Support Division | LouisvilleKY.gov

  #88 opened 7 months ago by hannahzacharski55
  0

• AllenAI

  #87 opened 7 months ago by hannahzacharski55
  0

• Dolma stat crashes because number of bins overflows python integer

  #70 opened 7 months ago by peterbjorgensen
  0

• Hells Angels infinite loop

  #85 opened 7 months ago by hannahzacharski55
  0

• New Albany Business, Family Law and Criminal Defense Lawyer Aaron Johnson

  #83 opened 7 months ago by hannahzacharski55
  0

• Hells Angels infinite loop save view open

  #84 opened 7 months ago by hannahzacharski55
  0

• $open create sudo port forward import for https://www.mattoxandwilson.com>>all

  #82 opened 7 months ago by hannahzacharski55
  0

• $open Allen Wolf (infinite_loop)

  #80 opened 7 months ago by hannahzacharski55
  0

• 911

  #81 opened 7 months ago by hannahzacharski55
  0

• Git - gitk Documentation

  #79 opened 7 months ago by hannahzacharski55
  0

• Terminal20141030.zip - Google Drive

  #75 opened 8 months ago by hannahzacharski55
  1

• ZIP Extractor - Free App for Opening and Creating ZIP Files

  #76 opened 8 months ago by hannahzacharski55
  1

• make_wikipedia.py hardcoded to simple

  #57 opened 8 months ago by peterbjorgensen
  0

• Jessie ©

  #48 opened 9 months ago by hannahzacharski55
  1

• Titles

  #47 opened 9 months ago by hannahzacharski55
  1

• Ruby

  #46 opened 9 months ago by hannahzacharski55
  1

• Adam Burden, I love you!

  #45 opened 9 months ago by hannahzacharski55
  1

• Adam Burden

  #44 opened 9 months ago by hannahzacharski55
  1