microsoft/Tokenizer

Issues

• Support Llama3.1 tokenization

  #56 opened a month ago by joyceerhl
  0

• Tokenizing a file with a lot of nested open arrays takes almost a minute

  #55 opened 2 months ago by kimadeline
  0

• Error: ENOENT: no such file or directory, mkdir '/var/model'

  #48 opened 6 months ago by sbernadsky
  1

• Support for text-embedding-3-large

  #47 opened 7 months ago by veedi
  1

• Tokenizer TS Web Browser Compatability

  #45 opened 8 months ago by Wolfleader101
  1

• TS Optimization: make the LRU cache pluggable and faster

  #37 opened 8 months ago by connor4312
  0

• TS Optimization: avoid allocations and array slicing in byte pair encoding

  #39 opened 8 months ago by connor4312
  0

• TS Optimization: avoid allocations when encoding text

  #38 opened 8 months ago by connor4312
  0

• TS Optimization: see if there's a minimum length for LRU entries

  #43 opened 8 months ago by connor4312
  1

• `encodeTrim*` functions behave weirdly with some (special?) tokens

  #25 opened a year ago by nicocanali
  0

• Token count accuracy questions with GPT3.5

  #20 opened a year ago by jsypkens
  2

• Can I use the SentencePiece Model file?

  #15 opened a year ago by tylike
  1

• Example of how to pre-download the BPE rank file

  #18 opened a year ago by KyleMit
  2