tokenization
There are 1107 repositories under tokenization topic.
explosion/spaCy
💫 Industrial-strength Natural Language Processing (NLP) in Python
AgentOps-AI/tokencost
Easy token price estimates for 400+ LLMs. TokenOps.
NVIDIA/Cosmos-Tokenizer
A suite of image and video neural tokenizers
lunasec-io/lunasec
LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/
securitybunker/databunker
Secure Vault for Customer PII/PHI/PCI/KYC Records
RavenProject/Ravencoin
Ravencoin Core integration/staging tree
VKCOM/YouTokenToMe
Unsupervised text tokenizer focused on computational efficiency
explosion/spacy-streamlit
👑 spaCy building blocks and visualizers for Streamlit apps
AmoDinho/datacamp-python-data-science-track
All the slides, accompanying code and exercises all stored in this repo. 🎈
nlp-uoregon/trankit
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
cbaziotis/ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
alasdairforsythe/tokenmonster
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
adobe/NLP-Cube
Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing
yooper/php-text-analysis
PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language
macmade/ClangKit
ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.
daac-tools/vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
WorksApplications/sudachi.rs
Sudachi in Rust 🦀 and new generation of SudachiPy
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
SaberaTalukder/TOTEM
The official code 👩💻 for - TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis
FoundationVision/OmniTokenizer
[NeurIPS 2024]OmniTokenizer: one model and one weight for image-video joint tokenization.
natasha/razdel
Rule-based token, sentence segmentation for Russian language
CodeChain-io/codechain
CodeChain's official implementation in Rust.
SmartTokenLabs/TokenScript
TokenScript schema, specs and paper
zjukg/MyGO
[Paper][AAAI 2025] (MyGO)Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation
janlukasschroeder/nlp-cheat-sheet-python
NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
milaan9/Python_Natural_Language_Processing
This repository consists of a complete guide on natural language processing (NLP) in Python where we'll learn various techniques for implementing NLP including parsing & text processing and understand how to use NLP for text feature engineering.
adbar/simplemma
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
GlitchedPolygons/l8w8jwt
Minimal, OpenSSL-less and super lightweight JWT library written in C.
THUDM/icetk
A unified tokenization tool for Images, Chinese and English.
rth/vtext
Simple NLP in Rust with Python bindings
gautierdag/bpeasy
Fast bare-bones BPE for modern tokenizer training
cohere-ai/magikarp
Code for the paper "Fishing for Magikarp"
ash-01xor/bpe.c
Simple Byte pair Encoding mechanism used for tokenization process . written purely in C
jshuadvd/LongRoPE
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
bminixhofer/zett
Code for Zero-Shot Tokenizer Transfer