ankane/tokenizers-ruby

Issues

• Add optional punctuation cleanup during decoding - clean_up_tokenization equivalent

  #33 opened 2 months ago by w-zygmuntowicz
  5

• Clarify how to load tokenizers from files

  #32 opened 6 months ago by datasciencedavid
  1

• Error when loading tokenizers gem

  #31 opened 8 months ago by crohr
  5

• Issues when deploying to Ubuntu 20.04

  #30 opened a year ago by TheBrockEllis
  6

• Cannot install tokenizer 0.3.2

  #28 opened a year ago by pribadi1st
  5

• error when using in docker alpine

  #27 opened a year ago by haanhduclinh
  5

• Issue with CharBPETokenizer and pile_tokenizer.json

  #25 opened a year ago by max-fry-apps
  3

• Ruby 2.7 didn't have JSON.load_file method

  #26 opened a year ago by elct9620
  1

• Using tiktoken

  #12 opened a year ago by ScotterC
  1

• Unable to run Rails when gem is installed

  #10 opened a year ago by marckohlbrugge
  6

• More flexible handling of special tokens

  #8 opened a year ago by petergoldstein
  6

• Version 0.2 relying on an old version of libssl

  #6 opened a year ago by kwi
  4

• Support to Ruby 3.2.0 (release 0.2.1)

  #5 opened a year ago by vickymadrid03
  3

• I get an error when installing

  #3 opened 2 years ago by kojix2
  5

• Error compiling

  #2 opened 2 years ago by ur5us
  1