stringertheory/twitter-tokenize

This evolved from this project, but only retained the tokenization stuff.

Install with:

pip install git+https://github.com/stringertheory/twitter-tokenize.git#egg=twitter-tokenize

and run with:

import twokenize

twokenize.tokens("Here's the thing: yo mama ;)")
# [u'here', u"'s", u'the', u'thing', u':', u'yo', u'mama', u';)']