twinnydotdev/toxe

SentencePiece tokenizer for cross-encoders

JavaScriptMIT

toxe

npm i toxe

import { Toxe } from 'toxe';

const toxe = new Txe('./spm.model');

const ids = await toxe.encode("a", [
  "a b",
  "a b c",
]);

Support passing config.