sm-Fifteen/markovich

Add a fast path to populate the database in one go

sm-Fifteen opened this issue · 0 comments

Populating the database is currently very slow since feeding text to the main learning functions one line at a time causes very large I/O overhead. Ideally, a mass populate function could be written using a Counter dict keyed by word pairs, where an entire document is broken down into word pairs and then everything in that dictionnary is saved in one go, with no collisions to account for.