Hk669/bpetokenizer

Issues

• use go for multithreading<> increase the performance

  #16 opened 5 months ago by Hk669
  2

• Deprecate the save/load mode= "file" for the tokenizer.

  #4 opened 5 months ago by Hk669
  0

• Add batch preprocessing if the training dataset is huge.

  #1 opened 5 months ago by Hk669
  8

• `special_tokens` in the encode method doesn't work for the BPETokenizer

  #12 opened 5 months ago by Hk669
  0

• start-endtime and throughput required to estimate the time required to train the tokenizer.

  #9 opened 5 months ago by Hk669
  1

• deprecate the __version__ check when loading the tokenizer.

  #7 opened 5 months ago by Hk669
  1

• make the `from_pretrained` method available to load the tokenizers from pretrained.

  #5 opened 5 months ago by Hk669
  1