Training error after use BPE
happygirl123456 opened this issue · 2 comments
Hello ,after I use BPE, I meet the issue ,Before using BPE, the program ran very well，what's wrong with it ,could you give me some help, Thanks!
The error :
/usr/local/bin/luajit: /usr/local/share/lua/5.1/threads/threads.lua:183: [thread 1 callback] /usr/local/share/lua/5.1/threads/threads.lua:183: [thread 1 callback] .../local/share/lua/5.1/torchnet/dataset/indexeddataset.lua:97: inconsistent index data size
stack traceback:
[C]: in function 'assert'
.../local/share/lua/5.1/torchnet/dataset/indexeddataset.lua:97: in function '__init'
/usr/local/share/lua/5.1/torch/init.lua:91: in function </usr/local/share/lua/5.1/torch/init.lua:87>
[C]: in function 'IndexedDataset'
/usr/local/share/lua/5.1/fairseq/torchnet/data.lua:546: in function </usr/local/share/lua/5.1/fairseq/torchnet/data.lua:545>
/usr/local/share/lua/5.1/fairseq/torchnet/data.lua:308: in function 'closure'
...are/lua/5.1/torchnet/dataset/paralleldatasetiterator.lua:79: in function <...are/lua/5.1/torchnet/dataset/paralleldatasetiterator.lua:78>
[C]: in function 'xpcall'
/usr/local/share/lua/5.1/threads/threads.lua:234: in function 'callback'
/usr/local/share/lua/5.1/threads/queue.lua:65: in function </usr/local/share/lua/5.1/threads/queue.lua:41>
[C]: in function 'pcall'
/usr/local/share/lua/5.1/threads/queue.lua:40: in function 'dojob'
[string " local Queue = require 'threads.queue'..."]:13: in main chunk
stack traceback:
[C]: in function 'error'
/usr/local/share/lua/5.1/threads/threads.lua:183: in function 'dojob'
/usr/local/share/lua/5.1/threads/threads.lua:264: in function 'synchronize'
/usr/local/share/lua/5.1/threads/threads.lua:142: in function 'specific'
/usr/local/share/lua/5.1/threads/threads.lua:125: in function 'Threads'
...are/lua/5.1/torchnet/dataset/paralleldatasetiterator.lua:85: in function '__init'
/usr/local/share/lua/5.1/torch/init.lua:91: in function </usr/local/share/lua/5.1/torch/init.lua:87>
[C]: in function 'ParallelDatasetIterator'
/usr/local/share/lua/5.1/fairseq/torchnet/data.lua:334: in function 'closure'
...hare/lua/5.1/fairseq/torchnet/SingleParallelIterator.lua:28: in function <...hare/lua/5.1/fairseq/torchnet/SingleParallelIterator.lua:27>
[C]: in function 'xpcall'
/usr/local/share/lua/5.1/threads/threads.lua:234: in function 'callback'
/usr/local/share/lua/5.1/threads/queue.lua:65: in function </usr/local/share/lua/5.1/threads/queue.lua:41>
[C]: in function 'pcall'
/usr/local/share/lua/5.1/threads/queue.lua:40: in function 'dojob'
[string " local Queue = require 'threads.queue'..."]:13: in main chunk
stack traceback:
[C]: in function 'error'
/usr/local/share/lua/5.1/threads/threads.lua:183: in function 'dojob'
/usr/local/share/lua/5.1/threads/threads.lua:264: in function 'synchronize'
/usr/local/share/lua/5.1/threads/threads.lua:142: in function 'specific'
/usr/local/share/lua/5.1/threads/threads.lua:125: in function 'Threads'
...hare/lua/5.1/fairseq/torchnet/SingleParallelIterator.lua:34: in function '__init'
/usr/local/share/lua/5.1/torch/init.lua:91: in function </usr/local/share/lua/5.1/torch/init.lua:87>
[C]: in function 'makeDataIterator'
/usr/local/share/lua/5.1/fairseq/torchnet/data.lua:482: in function 'makeDataIterators'
/usr/local/share/lua/5.1/fairseq/torchnet/data.lua:527: in function 'loadCorpus'
/usr/local/share/lua/5.1/fairseq/scripts/train.lua:160: in main chunk
[C]: in function 'require'
/usr/local/lib/luarocks/rocks/fairseq/scm-1/bin/fairseq:17: in main chunk
[C]: at 0x00406020
How did you apply BPE exactly? Does the error appear right at the start or later on? How long is the longest sequence in your training data after applying BPE?