Time for HDF creation step
Opened this issue · 1 comments
Hello,
I am currently running the HDF dataset creation step and am curious how long I should expect this to take. Apologies this is the first time I am working with this file type.
I have run the code to create the train and test datasets on a cluster through slurm. I allocated ~5 days of run time for both runs and 120GB of ram and they don't seem to complete the step, the files slowly keep growing as the script runs but there doesn't seem to be a clear completion of the step. For example, when I look at the output of the validation step while it is running, each line of output has a number that occasionally increases. Currently, it is reading "11775" and has been running for over 5 days.
In my batch file, I am loading these modules then activating the bee4exp environment that contains all of the dependencies that are listed in the bee4exp.yml file.
module load intel-python3
module load gnu/6.3.0 cuda-11.4
module load hdf5/gnu-9.3.0
conda activate bee4exp
Does anyone have any comments on how long this process takes? Or any idea if any of these modules I am loading are blocking required programs from the bee4exp.yml? There seems to be an issue as I am allocating far more resources than I would expect to be needed, as mentioned in issue #4.
Thank you!
HI, same issue
@vladan-stojnic