`fastBPE` fix path
nmd2k opened this issue · 0 comments
nmd2k commented
I against assertion AssertionError: failed to learn bpe on /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.tok.shuf.50gb, command: /home/dungnm/CodeGen/fastBPE/fast learnbpe 50000 /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.tok.shuf.50gb > /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.codes
It turn out the command itself was not right. The fastBPE path will be located at "codegen_sources/model/tools/fastBPE/fast" according to install_env.sh instead of "fastBPE/fast"
Suggest file codegen_sources/preprocessing/bpe_modes/fast_bpe_mode.py change
FAST = str(Path(__file__).parents[3].joinpath("fastBPE/fast"))to
FAST = str(Path(__file__).parents[3].joinpath("codegen_sources/model/tools/fastBPE/fast"))