pre-processing memory error
kimkwan93 opened this issue · 0 comments
kimkwan93 commented
If you guys have memory allocation error when pre-processing the whole dataset, try to open each json or tok file and process it with line by line multiprocessing. In this case no memory error occurred and the processing time is similar.