fasttokenizer

There are 1 repositories under fasttokenizer topic.

  • NemoTokenizer

    Fast wordpiece, sentencepiece tokenizer by Trie, OpenMP, SIMD, MemoryPool

    Language:C++1