sugarme/tokenizer

Issues

• When Support Unigram?

  #44 opened 3 months ago by Doloxetine
  0

• regexp dont support ?i

  #43 opened 3 months ago by xuxiaoxia96
  1

• panic: runtime error: invalid memory address or nil pointer dereference

  #37 opened 5 months ago by runarheggset
  0

• remove log from init

  #39 opened 9 months ago by jackielii
  2

• Loading tokenizer trained with Huggingface implementation

  #38 opened 9 months ago by mazzma12
  0

• please fix this bug: int overflow

  #36 opened 10 months ago by HgLiJiahao
  2

• Compile to wasm target

  #35 opened a year ago by tiero
  1

• OpenAI CLIP tokenization support?

  #34 opened a year ago by kristofmaar
  0

• panic:fatal error: concurrent map writes

  #32 opened a year ago by ZeroYuJie
  2

• panic: assignment to entry in nil map

  #31 opened a year ago by AlamoTNT
  6

• Missing roberta-base-vocab.json

  #26 opened a year ago by johntrob14
  2

• BOS/EOS tokens

  #17 opened a year ago by trpstra
  2

• Bump version?

  #18 opened a year ago by hugbubby
  1

• How to load self-made vocab.txt

  #24 opened a year ago by lierik
  4

• Performance / Parallelization Support

  #20 opened 2 years ago by JoeREISys
  3

• fatal error: concurrent map read and map write

  #19 opened 3 years ago by daudcanugerah
  2

• int64

  #15 opened 4 years ago by sugarme
  0

• Truncation and Padding not working properly

  #14 opened 4 years ago by sugarme
  0

• Wordpiece Decoder

  #13 opened 4 years ago by sugarme
  1

• Decoder using pointer to interface

  #12 opened 4 years ago by sugarme
  0

• Add optional `addSpecialTokens` when encoding

  #11 opened 4 years ago by sugarme
  0

• pretrained tokenizers

  #9 opened 4 years ago by sugarme
  0

• Encode - Truncation working incorrectly with DualEncodeInput

  #5 opened 4 years ago by sugarme
  1

• Update License, README, Travis CI, Go report badge and bump version

  #3 opened 4 years ago by sugarme
  0

• pointer receiver

  #4 opened 4 years ago by sugarme
  0

• serialization

  #2 opened 4 years ago by sugarme
  0