tokenization
There are 1379 repositories under tokenization topic.
explosion/spaCy
💫 Industrial-strength Natural Language Processing (NLP) in Python
toon-format/toon
🎒 Token-Oriented Object Notation (TOON) – Compact, human-readable, schema-aware JSON for LLM prompts. Spec, benchmarks, TypeScript SDK.
AgentOps-AI/tokencost
Easy token price estimates for 400+ LLMs. TokenOps.
NVIDIA/Cosmos-Tokenizer
A suite of image and video neural tokenizers
lunasec-io/lunasec
LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/
securitybunker/databunker
Secure Vault for Customer PII/PHI/PCI/KYC Records
RavenProject/Ravencoin
Ravencoin Core integration/staging tree
VKCOM/YouTokenToMe
Unsupervised text tokenizer focused on computational efficiency
AmoDinho/datacamp-python-data-science-track
All the slides, accompanying code and exercises all stored in this repo. 🎈
explosion/spacy-streamlit
👑 spaCy building blocks and visualizers for Streamlit apps
nlp-uoregon/trankit
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
cbaziotis/ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
alasdairforsythe/tokenmonster
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
adobe/NLP-Cube
Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing
yooper/php-text-analysis
PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language
alejoacosta74/blockchain-bike-rental
Solidity based "BIKE RENTAL SHOP" on Ethereum network.
WorksApplications/sudachi.rs
Sudachi in Rust 🦀 and new generation of SudachiPy
daac-tools/vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
macmade/ClangKit
ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.
SaberaTalukder/TOTEM
The official code 👩💻 for - TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
FoundationVision/OmniTokenizer
[NeurIPS 2024]OmniTokenizer: one model and one weight for image-video joint tokenization.
natasha/razdel
Rule-based token, sentence segmentation for Russian language
zjukg/MyGO
[Paper][AAAI 2025] (MyGO)Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation
CodeChain-io/codechain
CodeChain's official implementation in Rust.
janlukasschroeder/nlp-cheat-sheet-python
NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
SmartTokenLabs/TokenScript
TokenScript schema, specs and paper
ImadSaddik/Train_Your_Language_Model_Course
Train a language model to chat like you using your personal conversations from WhatsApp, Telegram, Signal, or other platforms.
milaan9/Python_Natural_Language_Processing
This repository consists of a complete guide on natural language processing (NLP) in Python where we'll learn various techniques for implementing NLP including parsing & text processing and understand how to use NLP for text feature engineering.
adbar/simplemma
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
cohere-ai/magikarp
Code for the paper "Fishing for Magikarp"
gautierdag/bpeasy
Fast bare-bones BPE for modern tokenizer training
GlitchedPolygons/l8w8jwt
Minimal, OpenSSL-less and super lightweight JWT library written in C.
rth/vtext
Simple NLP in Rust with Python bindings
THUDM/icetk
A unified tokenization tool for Images, Chinese and English.