/tiny_tokenizer

A word-level tokenizer for TinyStories data

Primary LanguagePythonMIT LicenseMIT

tiny_tokenizer

A word-level tokenizer for TinyStories data

Made with help and thoughts from https://github.com/tdooms, Dan Braun, Juan Diego Rodriguez, and Mat Allen.