Add a fast path to populate the database in one go
sm-Fifteen opened this issue · 0 comments
sm-Fifteen commented
Populating the database is currently very slow since feeding text to the main learning functions one line at a time causes very large I/O overhead. Ideally, a mass populate function could be written using a Counter
dict keyed by word pairs, where an entire document is broken down into word pairs and then everything in that dictionnary is saved in one go, with no collisions to account for.