dhbrojas/tokengeex

Issues

• Comprehensive ablation study of all tokeniser parameters

  #12 opened 9 months ago by dhbrojas
  5

• Update exact, general, idiomatic regexes

  #36 opened 9 months ago by dhbrojas
  0

• Build database of added & suggested tokens

  #35 opened 9 months ago by dhbrojas
  0

• Add `Tokenizer::sample_encode` method

  #8 opened 9 months ago by dhbrojas
  0

• Add node bindings and web-based visualiser

  #11 opened a year ago by dhbrojas
  0

• Add many more suggested and added tokens

  #22 opened a year ago by dhbrojas
  3

• Implement the HuggingFace `AutoTokenizer` interface

  #18 opened a year ago by dhbrojas
  1

• Use log frequencies on a per-document basis

  #33 opened a year ago by dhbrojas
  0

• Add Python methods to `PyTokenizer`

  #9 opened a year ago by dhbrojas
  1

• Add typing support for Python bindings

  #27 opened a year ago by dhbrojas
  1

• Fallible methods should throw exceptions or return errors but not panic

  #23 opened a year ago by dhbrojas
  2

• Add support for "added tokens"

  #26 opened a year ago by dhbrojas
  1

• Research how compression and byte-level tokenisation could play together

  #15 opened a year ago by dhbrojas
  2

• Automated evaluation pipeline and vocabulary hub

  #29 opened a year ago by dhbrojas
  4

• Add support for vocabulary "extension packs"

  #3 opened a year ago by dhbrojas
  3

• Benchmark hot paths in the crate

  #6 opened a year ago by dhbrojas
  2

• Improve the `Unigram::VocabularyGenerator` interface

  #2 opened a year ago by dhbrojas
  2

• Enable training on >5GB on a 16GB laptop

  #16 opened a year ago by dhbrojas
  0

• Add capcode as a separate, optional processing step

  #10 opened a year ago by dhbrojas
  2

• Investigate less than 3 byte compression for Chinese

  #14 opened a year ago by dhbrojas
  1

• Add base CodeGeeX vocabulary to TokenGeeX

  #13 opened a year ago by dhbrojas
  1

• Add tracing support through the `tracing` Crate

  #5 opened a year ago by dhbrojas
  2

• Find a solution for having to generate `"D "` on every generation

  #25 opened a year ago by dhbrojas
  1

• Prune low frequency tokens after training

  #31 opened a year ago by dhbrojas
  1

• Collect most frequent tokens from un-strict vocab and add them to strict vocab

  #28 opened a year ago by dhbrojas
  1

• Measure compression on production data

  #4 opened a year ago by dhbrojas
  0

• Tokenizer should operate on bytes

  #7 opened a year ago by dhbrojas
  2

• Make capcode lossless

  #24 opened a year ago by dhbrojas
  1

• Error `"invalid type: null, expected f64"` when loading cached vocab

  #20 opened a year ago by dhbrojas
  0

• Add support for special tokens

  #17 opened a year ago by dhbrojas
  0

• Investigate the need for a "splitting" step before encoding

  #21 opened a year ago by dhbrojas
  1

• Add utility scripts to compute compression over a given dataset

  #19 opened a year ago by dhbrojas
  0