mideind/Tokenizer

Support for citation characters

Opened this issue · 0 comments

The tokenizer should support superscripted citation characters. This will also help with GreynirCorrect, which I assume will be heavily used to read student essays and academic papers.

Screen Shot 2020-06-30 at 23 14 20