Preprocess.sh error (division by Zero)

Question

Preprocess.sh error (division by Zero)

ShaliniR11 opened this issue 2 years ago · 14 comments

Hi Dr.Alon,
I have my own Java dataset and I am trying to preprocess it with the given Script. I have changed the path directories in the script.I get the following output:
shali@DESKTOP-JNLA5ED MINGW64 ~/Documents/Git/code2vec (master)
$ sh preprocess.sh
preprocess.sh: line 21: C:/Users/shali/Documents/Git/code2vec/data/javadata/train/: Is a directory
Extracting paths from validation set...
Finished extracting paths from validation set
Extracting paths from test set...
Finished extracting paths from test set
Extracting paths from training set...
Finished extracting paths from training set
Creating histograms from the training data
File: my_dataset.test.raw.txt
Traceback (most recent call last):
File "C:\Users\shali\Documents\Git\code2vec\preprocess.py", line 133, in
num_examples = process_file(file_path=data_file_path, data_file_role=data_role, dataset_name=args.output_name,
File "C:\Users\shali\Documents\Git\code2vec\preprocess.py", line 69, in process_file
print('Average total contexts: ' + str(float(sum_total) / total))
ZeroDivisionError: float division by zero.

My System Requirements:
I am using GitBash on Visual studio code to run the script.
OS: Windows 11
Java : java --version
openjdk 17.0.3 2022-04-19
OpenJDK Runtime Environment Temurin-17.0.3+7 (build 17.0.3+7)
OpenJDK 64-Bit Server VM Temurin-17.0.3+7 (build 17.0.3+7, mixed mode, sharing)
Python: python --version
Python 3.10.4
CUDA:
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_19:00:59_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0
below is my preprocess.sh in txt format
preprocess.txt

Please let me know how to proceed further.

Answer 1 · 2022-07-21T16:53:52.000Z

Hi @ShaliniR11 ,
Sorry for the delayed response.

Did you notice that you have a "space" token in line 21, right before the path? can you delete this space and see if it helps?

Additionally, do you have subdirectories in the directory C:/Users/shali/Documents/Git/code2vec/data/javadata/train/? The code looks for subdirectories in the training path.

Best,
Uri

Answer 2 · 2022-07-24T19:27:16.000Z

Hi Dr. Alon, I have tried removing the space, the error still exists.
I have a single sub directory in each of train,test and val like

and inside these subdirectories I have java files like this:

Answer 3 · 2022-07-25T00:51:05.000Z

Can you try running the java process directly, e.g.,:

java -cp JavaExtractor/JPredict/target/JavaExtractor-0.0.1-SNAPSHOT.jar JavaExtractor.App --max_path_length 8 --max_path_width 2 --dir JavaExtractor/JPredict/src/main

?

Answer 4 · 2022-07-25T01:01:15.000Z

Can you try running the java process directly, e.g.,:
java -cp JavaExtractor/JPredict/target/JavaExtractor-0.0.1-SNAPSHOT.jar JavaExtractor.App --max_path_length 8 --max_path_width 2 --dir JavaExtractor/JPredict/src/main
?
This is the console output:

Answer 5 · 2022-07-25T01:09:50.000Z

OK so the base java process is running fine, it looks like the problem is in some input/output redirection because of Windows.
Can you try running that on a linux machine, or BashOnWindows?

Answer 6 · 2023-02-08T10:17:15.000Z

Hi @urialon I am facing the same issue. @ShaliniR11 could you solve it?
(env_tensor) zunaira@snps-ubo9fomrduif C2V3 % source preprocess.sh
Extracting paths from validation set...
Finished extracting paths from validation set
Extracting paths from test set...
Finished extracting paths from test set
Extracting paths from training set...
Finished extracting paths from training set
Creating histograms from the training data
File: my_dataset.test.raw.txt
Traceback (most recent call last):
File "/Users/zunaira/Downloads/C2V3/preprocess.py", line 133, in
num_examples = process_file(file_path=data_file_path, data_file_role=data_role, dataset_name=args.output_name,
File "/Users/zunaira/Downloads/C2V3/preprocess.py", line 69, in process_file
print('Average total contexts: ' + str(float(sum_total) / total))
ZeroDivisionError: float division by zero

Answer 7 · 2023-02-08T11:19:40.000Z

Hi @zunairazaman2021 ,
Thank you for your interest in our work!

Can you try running the java process directly, as instructed earlier in this thread?

Answer 8 · 2023-02-08T11:57:53.000Z

@urialon Yes I did

Answer 9 · 2023-02-08T11:59:08.000Z

@urialon I tried this as well #109 but it didn't work
Note: I am using Mac M1 chip, and I just changed directories here as:
TRAIN_DIR=/Users/zunaira/Downloads/C2V3/tmp/train
VAL_DIR=/Users/zunaira/Downloads/C2V3/tmp/validation
TEST_DIR=/Users/zunaira/Downloads/C2V3/tmp/test

Answer 10 · 2023-02-08T12:25:57.000Z

Using #109 I get data stored in a tmp directory but still c2v and raw.txt files are empty. :(

Answer 11 · 2023-02-08T12:34:55.000Z

Nevermind, Solved it with #109 :) Thanks

Answer 12 · 2023-02-08T12:49:55.000Z

Great to hear :-)

…

On Wed, Feb 8, 2023 at 07:35 zunaira zaman ***@***.***> wrote: Nevermind, Solved it with #109 <#109> :) Thanks — Reply to this email directly, view it on GitHub <#158 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADSOXMGIFB6ZK2EUUI4XUDTWWOHHVANCNFSM534W3PCQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Answer 13 · 2023-03-13T08:13:45.000Z

+1 on #109 solution, I had the same issue and the PR from gOATiful made it work

Answer 14 · 2023-03-13T12:34:25.000Z

Thanks @Lufedi and @zunairazaman2021 , I merged that PR.