guillaume-be/rust-tokenizers

Issues

• Evaluate Profile-Guided Optimization (PGO) performance benefits for the library

  #103 opened 7 months ago by zamazan4ik
  0

• Port of 'rust-tokenizer' to C#/.NET

  #102 opened 7 months ago by vermorel
  0

• Slight Error in Readme?

  #96 opened a year ago by ToluClassics
  2

• sentencepiece is not the same

  #89 opened 2 years ago by igor-yusupov
  0

• Reading SentencePieceVocab from text file

  #51 opened 4 years ago by MikaelCall
  5

• Can't create `XLMRobertaTokenizer` from xlm-roberta dataset

  #65 opened 3 years ago by mlodato517
  2

• [Question] How is this library compared to huggingface's tokenizer?

  #49 opened 4 years ago by failable
  2

• Using tokenizers within threads is a little cumbersome

  #39 opened 4 years ago by epwalsh
  10

• Issues with clean_up_tokenization() function?

  #16 opened 4 years ago by proycon
  2

• Tokenize non-breaking space

  #30 opened 4 years ago by sbeckeriv
  2

• Character offset information

  #14 opened 5 years ago by proycon
  10