/tokenizer-in-python

This is a repo where I am trying to code a tokenizer

Primary LanguageCMIT LicenseMIT

tokenizer-in-python

This is a repo where I am trying to code a tokenizer. This is work in Progress. This is not useful right now in practical applications. I am writing this tokenizer form scratch without any previous baseline code. Use this at your own risk