Why not use unique words to build prefix tree?
SRHyd opened this issue · 2 comments
SRHyd commented
Hello,
Why not use unique words to build prefix tree?
In LanguageModel.py at line 53
self.tree.addWords(words) # add all unique words to tree
But I think 'words' contains duplicates, shouldn't it be 'uniqueWords' ?
Thanks
SR
githubharald commented
thanks for the hint - I'll have a look at it.
githubharald commented
the tree is now created using the list of unique words - this should speed up the building process a little bit.