ankane/tokenizers-ruby

Issues

• Cannot install tokenizer 0.3.2

  #28 opened 2 years ago by pribadi1st
  8

• Using gem on Alpine and ARM-based CPUs

  #36 opened 5 months ago by robinroestenburg
  2

• Add optional punctuation cleanup during decoding - clean_up_tokenization equivalent

  #33 opened 10 months ago by w-zygmuntowicz
  5

• Clarify how to load tokenizers from files

  #32 opened a year ago by datasciencedavid
  1

• Error when loading tokenizers gem

  #31 opened a year ago by crohr
  5

• Issues when deploying to Ubuntu 20.04

  #30 opened 2 years ago by TheBrockEllis
  6

• error when using in docker alpine

  #27 opened 2 years ago by haanhduclinh
  5

• Issue with CharBPETokenizer and pile_tokenizer.json

  #25 opened 2 years ago by max-fry-apps
  3

• Ruby 2.7 didn't have JSON.load_file method

  #26 opened 2 years ago by elct9620
  1

• Using tiktoken

  #12 opened 2 years ago by ScotterC
  1

• Unable to run Rails when gem is installed

  #10 opened 2 years ago by marckohlbrugge
  6

• More flexible handling of special tokens

  #8 opened 2 years ago by petergoldstein
  6

• Version 0.2 relying on an old version of libssl

  #6 opened 2 years ago by kwi
  4

• Support to Ruby 3.2.0 (release 0.2.1)

  #5 opened 2 years ago by vickymadrid03
  3

• I get an error when installing

  #3 opened 2 years ago by kojix2
  5

• Error compiling

  #2 opened 3 years ago by ur5us
  1