vocabulary set to empty during tranform method in DeepcutTokenizer
titipata opened this issue · 0 comments
titipata commented
Somehow after transforming the document using
tokenizer.transform(raw_documents, new_document=True)The tokenizer.vocabulary_ is set to empty. It might be an error in the transform method where we accidentally set the vocabulary to empty.