Not enough test coverage

[peturorri]

We could use having better code coverage of tests.

There is an easy way to get the coverage: There is a pytest plugin called pytest-cov that generates coverage reports.

pip install pytest pytest-cov
pytest --cov=src/tokenizer --cov-report=html

The current results look like this:

----------- coverage: platform linux, python 3.8.6-final-0 -----------
Name                           Stmts   Miss  Cover
--------------------------------------------------
src/tokenizer/__init__.py          7      0   100%
src/tokenizer/abbrev.py          157     12    92%
src/tokenizer/definitions.py     121      9    93%
src/tokenizer/main.py            103    103     0%
src/tokenizer/tokenizer.py      1168    200    83%
--------------------------------------------------
TOTAL                           1556    324    79%

and a html-report that highlights uncovered lines is generated in the folder htmlcov