mlverse/hftokenizers

Issues

• please implement the `from_pretrained` tokenizer method

  #12 opened 3 years ago by cregouby
  0

• installation error `could not compile hftokenizer` due to missing argument in `truncate()`

  #10 opened 3 years ago by cregouby
  0

• Support for trainers in tokenizers$fit

  #3 opened 4 years ago by dfalbel
  0

• BPE trainer

  #2 opened 4 years ago by dfalbel
  0

• Export some pre-tokenizer objects

  #4 opened 4 years ago by dfalbel
  0

• implement full decoder API

  #1 opened 4 years ago by dfalbel
  0