NMZivkovic/BertTokenizers

Issues

• [BUG] BertUncasedBaseTokenizer ran forever with input "SixGe1−xH"

  #28 opened 3 months ago by darren-zdc
  2

• Wrong vocabulary index after white space

  #27 opened 6 months ago by wegylexy
  0

• This does not match behavior of Huggingface's Python version

  #18 opened 2 years ago by gevorgter
  9

• Word piece tokenizer never exits if a sub-word token doesn't exist

  #26 opened a year ago by matteocontrini
  1

• Custom vocabulary classes naming error

  #22 opened a year ago by tsepton
  0

• Strings with linux line endings break the tokenizer

  #24 opened a year ago by palenshus
  0

• Looks for Vocabularies in source dir instead of (e.g.) bin/release/net6/

  #21 opened a year ago by ctwardy
  0

• The tokenization for Korean text seems not correct.

  #20 opened a year ago by terryqj0107
  0

• different behavior: hugging face bert-base-uncased vs. BERT Base Uncased

  #12 opened 2 years ago by PaulCalot
  3

• Words surrounded by backwards quotation marks causing inaccurate tokenization results

  #17 opened 2 years ago by rghavimi
  0

• Multilingual vocab not code properly

  #15 opened 2 years ago by zhipenghan
  0

• Support for loading a custom vocab.txt?

  #13 opened 2 years ago by BrainSlugs83
  3

• Multilingual model tokenization differs from Python

  #4 opened 2 years ago by ADD-eNavarro
  3

• BertTokenizers in .NET 3.1 and/or .NET 6.0

  #2 opened 2 years ago by ADD-eNavarro
  2

• CI/CD Pipeline - Automaticly publishing NuGet Package

  #8 opened 2 years ago by NMZivkovic
  0

• Issue using the code

  #1 opened 2 years ago by bentoo
  1