zurawiki/tiktoken-rs

Issues

• Is there a way to cache the bpe when using `num_tokens_from_messages`

  #81 opened 3 months ago by xd009642
  3

• Good points. How are you benchmarking these times today?

  #78 opened 3 months ago by darksonblade
  0

• Great to hear! Issues that can be worked on are listed here in the GitHub issues. I'd recommend tackling each issue one at a time. You can comment on each issue that you're interested in working on.

  #79 opened 3 months ago by darksonblade
  0

• hgx

  #80 opened 3 months ago by darksonblade
  0

• Support for other LLMs?

  #62 opened 9 months ago by carlca
  0

• async-openai 0.18 support

  #56 opened 10 months ago by ChristopherJMiller
  1

• Update to async-openai 0.16.2

  #50 opened a year ago by Dreaming-Codes
  3

• BPE memory leak

  #39 opened a year ago by venual
  7

• Add `gpt-4-1106-preview`

  #46 opened a year ago by JosephTLyons
  3

• Make encode token generic, accepting usize and u32

  #45 opened a year ago by mokeyish
  1

• Expose `_decode_native`?

  #27 opened a year ago by Palmik
  4

• Missing gpt2 encoder

  #8 opened 2 years ago by ceifa
  1

• Token counts are incorrect when request includes function messages

  #40 opened a year ago by kylewlacy
  1

• Don't arbitrarily enforce versions please

  #37 opened a year ago by aristotaloss
  1

• [Question] Difference between `encode_with_special_tokens` and `encode_ordinary`

  #30 opened a year ago by kojix2
  0

• tiktoken-c and tiktoken-cr

  #29 opened a year ago by kojix2
  0

• How could you add parallelism to make the encoding faster?

  #7 opened 2 years ago by smahm006
  6

• Incomplete utf-8 byte sequence from index 0

  #23 opened a year ago by BohuTANG
  1

• Use case: splitting text into tokens.

  #16 opened 2 years ago by jackbackes
  0

• Any chance of isolating some dependencies with features?

  #10 opened 2 years ago by IAPark
  3

• Make the crate independent of async_openai

  #13 opened 2 years ago by thedenisnikulin
  1

• use tiktoken encode throw error.

  #5 opened 2 years ago by misssonder
  1

• cl100k usage

  #1 opened 2 years ago by mkualquiera
  1