Several detailed questions about reproduction
KennardWang opened this issue · 2 comments
Hi, I am really interested in this ICSE paper published by you guys. Recently I have planned to reproduce the experiments but I face some difficulties.
- I have reproduced the standard data set under both attendgru and ast-attendgru method. Due to the limit of my devices and network, I only implement model training for 5 epochs under attendgru (about 11 hours) and 3 epochs under ast-attendgru (about 6 hours). I see the original code provided in GitHub using 100 epochs, which means high accuracy. So I am not sure that my BLEU results is OK or not. Here is my results:
Model | Ba | B1 | B2 | B3 | B4 |
---|---|---|---|---|---|
attendgru, E05 | 19.14 | 37.88 | 21.4 | 14.66 | 11.3 |
attendgru, E03 | 19.24 | 38.65 | 21.77 | 14.66 | 11.12 |
ast-attendgru, E03 | 19.37 | 38.74 | 21.88 | 14.75 | 11.27 |
-
I still plan to do the challenge data set, but I do not find anything on website given by you. I only find a data set named "sbt", which only contains coms.tok and dats.tok. In your paper, you say "The challenge dataset contains two elements for each
method: 1) the pre-processed comment, and 2) the SBT-AO representation of the Java code". So I guess "sbt" is the challenge data set. If I make a mistake, could you please tell me where to download the challenge data set, thx. -
I see that you guys provide the final ast-attendgru trained model file (.h5) for both standard and challenge data. But to load the model acquires corresponding history configuration file, which I do not find. So I cannot do the prediction for the next step.
My development environment:
- Google Colab env, GPU with high RAM
- Keras==2.2.5
- tensorflow-gpu==1.14
- h5py==2.10.0
- We usually see BLEU scores settling in the 6-10 epoch range on the full FUNCOM dataset. Your scores generally seem to be in line with what we get.
- The challenge dataset is the sbt data with the dats data set to zeros - so only the SBT AO is used to train the model
- I am able to load the model without the history files using keras.models.load_model("{path_to_model.h5}"), I am not sure what issue you are running into.
Thanks for your reply. I think I need to explain a little bit detailed for my 3rd problem: In the line 175 of predict.py, the code is config = pickle.load(open(outdir+'/histories/'+modeltype+'_conf_'+timestart+'.pkl', 'rb'))
, which acquires the history configuration .pkl file. As the result, I cannot make the prediction due to lacking this file. I know that after model training, the history file will be generated automatically. But in the worst case, perhaps I can use your results directly if it costs too much on both time and computer resources. That's why I really need the history configuration file.
Another question is still about the 2nd problem. I am not sure how to train the model by using sbt dataset and challenge dataset. I just omit several lines about "sml" in train.py because sbt data set does not contain "sml.tok" like the standard dataset:
tdatvocabsize = tdatstok.vocab_size
comvocabsize = comstok.vocab_size
#smlvocabsize = smltok.vocab_size // omit
...
print('tdatvocabsize %s' % (tdatvocabsize))
print('comvocabsize %s' % (comvocabsize))
#print('smlvocabsize %s' % (smlvocabsize)) // omit
...
config['tdatvocabsize'] = tdatvocabsize
config['comvocabsize'] = comvocabsize
#config['smlvocabsize'] = smlvocabsize // omit
...
config['tdatlen'] = len(list(seqdata['dtrain'].values())[0])
config['comlen'] = len(list(seqdata['ctrain'].values())[0])
#config['smllen'] = len(list(seqdata['strain'].values())[0]) // omit
My training command: python train.py --model-type=attendgru --gpu=0
And now I am training attendgru model by using sbt dataset (the root directory changes to the sbt) without modifying any other code. I do not know the result until 6 hours later but I am worried about that my method is incorrect.