Tokenize error

[Kaimary]

When I executed the tokenize_pt_corpus.py script to tokenize the corpus, it encountered an error,

[lihaoyang-ruc]

This error likely occurred due to insufficient memory available during the execution of tokenize_pt_corpus.py.

[Kaimary]

Thanks. I solved the problem finally.