ropensci/tokenizers

Issues

• New Maintainer Welcome :-)

  #86 opened 7 months ago by maelle
  3

• implicit conversion of character input to UTF-8

  #87 opened 9 months ago by ablaette
  0

• Add strip_url option to tokenize_words()

  #85 opened 2 years ago by fschaffner
  1

• tokenize_tweets replacement

  #84 opened 2 years ago by alanault
  2

• Twitter tokenizing logic broken by upcoming ICU 72 breaking change ('@' no longer splits)

  #82 opened 2 years ago by MichaelChirico
  14

• Low-level parallelism with RcppParallel

  #51 opened 2 years ago by lmullen
  3

• keeping punctuation

  #80 opened 2 years ago by Legallois
  0

• Add strip_numeric = TRUE option to tokenize_tweets()

  #79 opened 2 years ago by fschaffner
  1

• Possible CRAN release

  #81 opened 2 years ago by EmilHvitfeldt
  2

• tokenize_tweets doesn't separate emojis with no spaces between them

  #68 opened 3 years ago by EmilHvitfeldt
  4

• Inconsistent behavior of tokenize_tweets() when filtering stopwords with punctuation

  #76 opened 4 years ago by syumet
  2

• Output ngram might consider punctuation separation?

  #74 opened 5 years ago by hope-data-science
  11

• Why could tokenizers split Chinese as well?

  #73 opened 5 years ago by hope-data-science
  1

• tokenize_tweets and single word strings

  #70 opened 6 years ago by juliasilge
  2

• Submit paper to JOSS

  #39 opened 7 years ago by lmullen
  14

• Update DESCRIPTION prior to release

  #62 opened 7 years ago by lmullen
  0

• Specify encoding in C++ code for skip_ngrams

  #58 opened 7 years ago by patperry
  12

• Comply with text interchange format, perhaps also adding vignette

  #49 opened 7 years ago by lmullen
  7

• integration into quanteda as a core tokenizer

  #25 opened 7 years ago by kbenoit
  14

• Depend on the stopwords package instead of providing that functionality

  #46 opened 8 years ago by randomgambit
  3

• Add a pkgdown website

  #61 opened 7 years ago by lmullen
  0

• Update README and vignettes for new release

  #43 opened 7 years ago by lmullen
  0

• Add function to chunk texts into smaller segments

  #30 opened 7 years ago by lmullen
  3

• Add Jockers stopwords

  #53 opened 7 years ago by lmullen
  0

• Strip punctuation option for tokenize_ngrams

  #57 opened 7 years ago by alanault
  0

• Tokenize sentences starting with a number

  #59 opened 7 years ago by ekstroem
  2

• Lower level C++ api with external pointers

  #50 opened 7 years ago by dselivanov
  4

• Installation Error

  #55 opened 7 years ago by rlumor
  3

• Error: could not find function "%>%"

  #54 opened 7 years ago by fahadshery
  1

• Inconsistent tokenizing when numbers are followed by a period

  #52 opened 8 years ago by rajkorde
  1

• Punctuation options

  #48 opened 8 years ago by lmullen
  2

• Way of committing to repo

  #45 opened 8 years ago by dselivanov
  2

• can we use alternative lexicons?

  #47 opened 8 years ago by randomgambit
  4

• Deprecate tokenize_regex()

  #42 opened 8 years ago by lmullen
  0

• Word counting

  #36 opened 8 years ago by lmullen
  1

• Trouble installing package "tokenizers" on R v3.2.3

  #41 opened 8 years ago by sanjesk1
  7

• Incorrect skipgrams

  #24 opened 8 years ago by koheiw
  47

• Add keyboard interrupts

  #37 opened 8 years ago by Ironholds
  0

• Add stopwords argument to `tokenize_skip_ngrams()`

  #31 opened 8 years ago by lmullen
  3

• NA support?

  #33 opened 8 years ago by Ironholds
  4

• Remove requirement for C++11

  #26 opened 8 years ago by statspro1
  30

• Pass argument by reference using raw pointer

  #20 opened 8 years ago by lmullen
  0

• Ideas for other tokenizers

  #27 opened 8 years ago by lmullen
  2

• `tokenize_ngram` deal with colon(:) inconsistently across different platforms

  #32 opened 8 years ago by everdark
  2

• Installation failed on Microsoft R Server

  #28 opened 8 years ago by kevinbsc
  3

• R-devel / Travis

  #23 opened 8 years ago by maelle
  2

• hunspell

  #19 opened 8 years ago by jeroen
  1

• Character level tokenizers

  #22 opened 8 years ago by dselivanov
  5

• Using long vectors

  #21 opened 8 years ago by lmullen
  1

• Error while installing tokenizers package

  #18 opened 8 years ago by harshakap
  2