mideind/Tokenizer

Issues

• OrderedDict not needed, and question and comment

  #53 opened 24 days ago by PallHaraldsson
  0

• A dot added in dates

  #52 opened 2 months ago by starkadur
  7

• detokenize and correct_spaces problem with hyphens and En dashes

  #51 opened 4 months ago by atlijas
  0

• pkg_resources is deprecated

  #46 opened 4 months ago by sveinbjornt
  0

• correct_spaces incorrectly inserts spaces into abbreviations

  #44 opened 2 years ago by atlijas
  1

• The tokenizer is slow when the input string is long.

  #40 opened 3 years ago by HaukurPall
  2

• Two dots

  #36 opened 3 years ago by starkadur
  1

• Spaces deleted

  #21 opened 3 years ago by starkadur
  3

• split_into_sentences changes sentences

  #24 opened 3 years ago by bnika
  7

• Character omitted

  #32 opened 3 years ago by starkadur
  0

• Adda mánuð

  #37 opened 3 years ago by sigurdurb
  3

• Support colon-separated duration?

  #31 opened 4 years ago by sveinbjornt
  0

• Bigger ordinal numbers in the tokenizer

  #28 opened 4 years ago by helga-lvl
  1

• The tokenizer is missing some abbreviations

  #25 opened 4 years ago by helga-lvl
  3

• Twitter handles and @usernames can contain periods (@mat­ur.a.mbl) but are broken into sentences

  #18 opened 4 years ago by sveinbjornt
  1

• Not enough test coverage

  #23 opened 4 years ago by peturorri
  0

• Bandstrik skilin frá orði

  #15 opened 4 years ago by starkadur
  1

• Support for citation characters

  #16 opened 5 years ago by sveinbjornt
  0

• Can this tokenizer be used for English Language also?

  #14 opened 5 years ago by Dhanachandra
  1

• Detokenization adds spaces to "o.s.frv."

  #13 opened 5 years ago by HaukurPall
  3

• UnboundLocalError: local variable 'unit' referenced before assignment

  #12 opened 5 years ago by HaukurPall
  1

• Inconsistent application of abbreviation expansion

  #11 opened 5 years ago by HaukurPall
  3

• Tokeniize() options

  #4 opened 5 years ago by pallih
  2

• KeyError for unknown abbreviations

  #1 opened 7 years ago by sverrirab
  1