/bpe-greek

Byte-pair encoding Greek

Primary LanguageR

bpe-greek

Byte-pair encoding Greek Byte-pair encoding Greek

  • Determine maximal needed tokens.
  • If token length does not change much, it is 'saturated'