build dataset and torch load()

Question

build dataset and torch load()

Opened this issue 5 years ago · 10 comments

Excuse me... I have another error when I build dataset.
I set the PYTHONPATH and run ./buildData.sh and no vocab.data.sql file in workdir but vocab.sql
then I renamed this file to vocab.data.sql and run ./run.sh sql
and I getting this error now..

(py2) linux@ubuntu1604:~/codenn/src/model$ ./run.sh sql
/home/linux/torch/install/bin/luajit: /home/linux/torch/install/share/lua/5.1/torch/File.lua:375: unknown object
stack traceback:
[C]: in function 'error'
/home/linux/torch/install/share/lua/5.1/torch/File.lua:375: in function 'readObject'
/home/linux/torch/install/share/lua/5.1/torch/File.lua:409: in function 'load'
main.lua:102: in function 'main'
main.lua:215: in main chunk
[C]: in function 'dofile'
...inux/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50
{
beam_size : 10
batch_size : 100
max_code_length : 100
max_nl_length : 100
layers : 1
max_length : 20
rnn_size : 400
}
/home/linux/torch/install/bin/luajit: /home/linux/torch/install/share/lua/5.1/torch/File.lua:375: unknown object
stack traceback:
[C]: in function 'error'
/home/linux/torch/install/share/lua/5.1/torch/File.lua:375: in function 'readObject'
/home/linux/torch/install/share/lua/5.1/torch/File.lua:409: in function 'load'
predict.lua:213: in function 'main'
predict.lua:251: in main chunk
[C]: in function 'dofile'
...inux/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50

It's seemd that torch load() can't load this file .
Thanks for your help.

Answer 1 · 2019-03-30T07:45:22.000Z

I think you might have been using python3, i think u will need to use python 2.7 for this

Answer 2 · 2019-03-30T08:47:42.000Z

I think you might have been using python3, i think u will need to use python 2.7 for this
I use python 2.7 environment by anaconda.

Answer 3 · 2019-03-30T18:37:26.000Z

You shouldn't need to rename anything. Can you paste the output of buildData?

Answer 4 · 2019-03-31T05:46:16.000Z

buildData create these files in workdir:
dev.txt.csharp eval.txt.sql train.txt.sql vocab.csharp
dev.txt.sql test.txt valid.txt.csharp vocab.sql
eval.txt.csharp train.txt.csharp valid.txt.sql

the output of buildData:
(py2) linux@ubuntu1604:~/codenn/src/model$ ./buildData.sh
Total size = 23970
Total skipped = 1701
Total size = 3110
Total skipped = 216
Total size = 111
Total skipped = 0
Total size = 100
Total skipped = 0
line 3:4 token recognition error at: '$'
line 1:55 token recognition error at: '@]'
line 1:62 token recognition error at: '@]'
line 1:81 token recognition error at: '@/'
line 1:83 token recognition error at: '$'
line 4:61 token recognition error at: '$'
line 1:10 token recognition error at: '$'
line 1:34 token recognition error at: '$'
line 3:41 token recognition error at: '' line 3:47 token recognition error at: ''
line 3:55 token recognition error at: '' line 3:64 token recognition error at: ''
line 4:6 token recognition error at: '' line 4:15 token recognition error at: ''
line 3:17 token recognition error at: '$'
line 3:22 token recognition error at: '$'
line 3:27 token recognition error at: '$'
line 1:54 token recognition error at: '$'
line 1:30 token recognition error at: '$'
line 1:53 token recognition error at: '@'
line 1:64 token recognition error at: '@('
line 2:44 token recognition error at: '$'
line 2:47 token recognition error at: '$'
line 2:50 token recognition error at: '$'
line 1:15 token recognition error at: '$'
line 3:20 token recognition error at: '@1'
line 22:0 token recognition error at: '$'
line 1:17 token recognition error at: '$'
line 1:13 token recognition error at: '$'
line 1:51 token recognition error at: '$'
line 2:2 token recognition error at: '$'
line 4:8 token recognition error at: '$'
line 11:88 token recognition error at: '$'
line 11:93 token recognition error at: '$'
line 1:55 token recognition error at: '$'
line 1:48 token recognition error at: '@;'
line 1:6 token recognition error at: '$'
line 1:48 token recognition error at: '$'
line 1:75 token recognition error at: '$'
line 2:6 token recognition error at: '$'
line 2:48 token recognition error at: '$'
line 2:74 token recognition error at: '$'
line 1:108 token recognition error at: '$'
line 1:31 token recognition error at: '$'
line 1:24 token recognition error at: '$'
line 1:32 token recognition error at: '$'
line 1:51 token recognition error at: '$'
line 1:9 token recognition error at: '$'
line 1:27 token recognition error at: '$'
line 1:60 token recognition error at: '$'
line 1:73 token recognition error at: '$'
line 2:87 token recognition error at: '$'
line 1:29 token recognition error at: '$'
line 1:116 token recognition error at: '' line 1:125 token recognition error at: ''
line 1:127 token recognition error at: '' line 1:136 token recognition error at: ''
line 1:139 token recognition error at: '' line 1:145 token recognition error at: ''
line 2:4 token recognition error at: '$'
line 4:0 token recognition error at: '$'
line 5:58 token recognition error at: '@'
line 1:39 token recognition error at: '$'
line 1:31 token recognition error at: '$'
line 1:42 token recognition error at: '' line 1:64 token recognition error at: '$' line 1:75 token recognition error at: ''
line 1:85 token recognition error at: '@\n'
line 3:10 token recognition error at: '$'
line 3:67 token recognition error at: '$'
line 5:17 token recognition error at: '$'
line 1:33 token recognition error at: '$'
line 2:11 token recognition error at: '$'
line 4:40 token recognition error at: '$'
line 5:15 token recognition error at: '$'
line 7:109 token recognition error at: '$'
line 9:127 token recognition error at: '$'
line 10:15 token recognition error at: '$'
line 11:11 token recognition error at: '$'
line 3:26 token recognition error at: '@0'
line 3:4 token recognition error at: '$'
line 1:55 token recognition error at: '@]'
line 1:62 token recognition error at: '@]'
line 1:81 token recognition error at: '@/'
line 1:83 token recognition error at: '$'
line 4:61 token recognition error at: '$'
line 1:10 token recognition error at: '$'
line 1:34 token recognition error at: '$'
line 3:41 token recognition error at: '' line 3:47 token recognition error at: ''
line 3:55 token recognition error at: '' line 3:64 token recognition error at: ''
line 4:6 token recognition error at: '' line 4:15 token recognition error at: ''
line 3:17 token recognition error at: '$'
line 3:22 token recognition error at: '$'
line 3:27 token recognition error at: '$'
line 1:54 token recognition error at: '$'
line 1:30 token recognition error at: '$'
line 1:53 token recognition error at: '@'
line 1:64 token recognition error at: '@('
line 2:44 token recognition error at: '$'
line 2:47 token recognition error at: '$'
line 2:50 token recognition error at: '$'
line 1:15 token recognition error at: '$'
line 3:20 token recognition error at: '@1'
line 22:0 token recognition error at: '$'
line 1:17 token recognition error at: '$'
line 1:13 token recognition error at: '$'
line 1:51 token recognition error at: '$'
line 2:2 token recognition error at: '$'
line 4:8 token recognition error at: '$'
line 11:88 token recognition error at: '$'
line 11:93 token recognition error at: '$'
line 1:55 token recognition error at: '$'
line 1:48 token recognition error at: '@;'
line 1:6 token recognition error at: '$'
line 1:48 token recognition error at: '$'
line 1:75 token recognition error at: '$'
line 2:6 token recognition error at: '$'
line 2:48 token recognition error at: '$'
line 2:74 token recognition error at: '$'
line 1:108 token recognition error at: '$'
line 1:31 token recognition error at: '$'
line 1:24 token recognition error at: '$'
line 1:32 token recognition error at: '$'
line 1:51 token recognition error at: '$'
line 1:9 token recognition error at: '$'
line 1:27 token recognition error at: '$'
line 1:60 token recognition error at: '$'
line 1:73 token recognition error at: '$'
line 2:87 token recognition error at: '$'
line 1:29 token recognition error at: '$'
line 1:116 token recognition error at: '' line 1:125 token recognition error at: ''
line 1:127 token recognition error at: '' line 1:136 token recognition error at: ''
line 1:139 token recognition error at: '' line 1:145 token recognition error at: ''
line 2:4 token recognition error at: '$'
line 4:0 token recognition error at: '$'
line 5:58 token recognition error at: '@'
line 1:39 token recognition error at: '$'
line 1:31 token recognition error at: '$'
line 1:42 token recognition error at: '' line 1:64 token recognition error at: '$' line 1:75 token recognition error at: ''
line 1:85 token recognition error at: '@\n'
line 3:10 token recognition error at: '$'
line 3:67 token recognition error at: '$'
line 5:17 token recognition error at: '$'
line 1:33 token recognition error at: '$'
line 2:11 token recognition error at: '$'
line 4:40 token recognition error at: '$'
line 5:15 token recognition error at: '$'
line 7:109 token recognition error at: '$'
line 9:127 token recognition error at: '$'
line 10:15 token recognition error at: '$'
line 11:11 token recognition error at: '$'
line 3:26 token recognition error at: '@0'
Total size = 46106
Total skipped = 6706
line 1:49 token recognition error at: '$'
line 1:93 token recognition error at: '$'
line 1:105 token recognition error at: '$'
line 3:72 token recognition error at: '$'
line 3:84 token recognition error at: '$'
line 5:60 token recognition error at: '$'
line 5:72 token recognition error at: '$'
Total size = 5795
Total skipped = 806
Total size = 100
Total skipped = 0
Total size = 108
Total skipped = 0

Thanks !

Answer 5 · 2019-03-31T05:48:41.000Z

That looks ok, what happens when you execute run.sh sql after this?

Answer 6 · 2020-02-15T21:41:01.000Z

Hi,
I am running ./buildData.sh, its been going on for some time now ( over 2 hours)
Total size = 23970
Total skipped = 1701
Total size = 3110
Total skipped = 216
Total size = 111
Total skipped = 0
Total size = 100
Total skipped = 0
line 3:4 token recognition error at: '$'
line 1:55 token recognition error at: '@]'
line 1:62 token recognition error at: '@]'
line 1:81 token recognition error at: '@/'
line 1:83 token recognition error at: '$'
line 4:61 token recognition error at: '$'

Is this normal behavior ? Any idea how much time it usually takes to run.
My machine is using Ubuntu 18.04 with 16GB Ram and intel i7

Answer 7 · 2020-02-15T22:51:47.000Z

Try using antlr-python2-runtime 4.5.3 instead of the lates one. I think its a lot faster for some people. The script should finish in a few minutes.

Answer 8 · 2020-02-16T06:25:09.000Z

Oh my bad, I switched to TPU and it finished within few minutes :). Thanks. Great work on the paper.

Answer 9 · 2021-05-31T06:29:29.000Z

Excuse.
When i execute "./buildData.sh" ,there is "No module named sql.SqlTemplate".

Answer 10 · 2021-06-09T16:36:53.000Z

@Doctopus003 Your PYTHONPATH should include the folder sql/SqlTemplate