Add tests
Opened this issue · 1 comments
serin-delaunay commented
Please.
serin-delaunay commented
In particular, test:
- line splitting with various characters
- inclusion/exclusion of line delimiters
- tokenisation
- ngramisation
- trascery symbol names
- expected output for simple source texts (
ababa, etc.) - correct treatment of various characters in text