tokeniser

There are 12 repositories under tokeniser topic.

  • dragonofmercy/Tokenize2

    Tokenize2 is a plugin which allows your users to select multiple items from a predefined list or ajax, using autocompletion as they type to find each item. You may have seen a similar type of text entry when filling in the recipients field sending messages on facebook or tags on tumblr.

    Language:JavaScript8367725

  • LanguageMachines/ucto

    Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules for several languages and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog, our Dutch morpho-syntactic processor. http://ilk.uvt.nl/ucto --

    Language:C++69129314
  • taibun

    andreihar/taibun

    Taiwanese Hokkien Transliterator and Tokeniser

    Language:Python331101

  • jonsafari/tok-tok

    A fast, simple, multilingual tokenizer

    Language:Python29513

  • ben-sb/jisu

    JavaScript Parser

    Language:TypeScript10100

  • kuhumcst/rtfreader

    Text segmenter and tokeniser for Danish, English and other languages. Reads an RTF or flat text file and outputs the text, one line per sentence & optionally tokenized.

    Language:C++8324

  • ztjhz/word-piece-tokenizer

    A Lightweight Word Piece Tokenizer

    Language:Python6310

  • phughesmcr/happynodetokenizer

    Javascript port of HappyFunTokenizer.py by Christopher Potts and HappierFunTokenizing.py by H. Andrew Schwartz

    Language:TypeScript5100
  • taibun.js

    andreihar/taibun.js

    Taiwanese Hokkien Transliterator and Tokeniser

    Language:JavaScript2100

  • TomHarte/bas2uef

    Converts BBC BASIC 2 source code into UEF files.

    Language:C++210

  • P0u4a/casio-parser

    Find out useful information about your Casio watch

    Language:Rust00

  • Abhigyan126/SentencePiece-Tokenisation

    A python and rust implementation of SentencePiece (A language-independent subword tokeniser and de-tokeniser developed by Google)

    Language:Rust10