Fails to create test.h5 and eval.h5 in DataExplorationAndPreparation
ykkimhgu opened this issue · 2 comments
ykkimhgu commented
Problem description
Stops with error in generating test/validation dataset of .h5 files in DataExplorationAndPreparation
It seems it creates train.h5 but does not create test.h5 and eval.h5
Problem details
When running this code in DataExplorationAndPreparation
train_eval_test_split = [0.7, 0.2, 0.1]
full_path_raw_folders = [os.path.join(RAW_DATA_DIR, f) for f in DATA_FOLDERS]
Cooking.cook(full_path_raw_folders, COOKED_DATA_DIR, train_eval_test_split)
Iteration stops with this output message.
It seems it creates train.h5 but does not create test.h5 and eval.h5
Codes before this line worked fine.
Reading data from ../dataset/EndToEndLearningRawData/data_raw/normal_1...
Reading data from ../dataset/EndToEndLearningRawData/data_raw/normal_2...
Reading data from ../dataset/EndToEndLearningRawData/data_raw/normal_3...
Reading data from ../dataset/EndToEndLearningRawData/data_raw/normal_4...
Reading data from ../dataset/EndToEndLearningRawData/data_raw/normal_5...
Reading data from ../dataset/EndToEndLearningRawData/data_raw/normal_6...
Reading data from ../dataset/EndToEndLearningRawData/data_raw/swerve_1...
Reading data from ../dataset/EndToEndLearningRawData/data_raw/swerve_2...
Reading data from ../dataset/EndToEndLearningRawData/data_raw/swerve_3...
Processing ../dataset/data_cooked/train.h5...
---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
~\Git_Program\AirSim\ADCookBook\Tutorial_E2EDeepLearning\Cooking.py in generatorForH5py(data_mappings, chunk_size)
129 raise StopIteration
--> 130 raise StopIteration
131
StopIteration:
The above exception was the direct cause of the following exception:
RuntimeError Traceback (most recent call last)
<ipython-input-13-bce568193587> in <module>
1 train_eval_test_split = [0.7, 0.2, 0.1]
2 full_path_raw_folders = [os.path.join(RAW_DATA_DIR, f) for f in DATA_FOLDERS]
----> 3 Cooking.cook(full_path_raw_folders, COOKED_DATA_DIR, train_eval_test_split)
~\Git_Program\AirSim\ADCookBook\Tutorial_E2EDeepLearning\Cooking.py in cook(folders, output_directory, train_eval_test_split)
192 for i in range(0, len(split_mappings), 1):
193 print('Processing {0}...'.format(output_files[i]))
--> 194 saveH5pyData(split_mappings[i], output_files[i])
195 print('Finished saving {0}.'.format(output_files[i]))
196
~\Git_Program\AirSim\ADCookBook\Tutorial_E2EDeepLearning\Cooking.py in saveH5pyData(data_mappings, target_file_path)
162 dset_previous_state[:] = previous_state_chunk
163
--> 164 for image_names_chunk, label_chunk, previous_state_chunk in gen:
165 image_chunk = np.asarray(readImagesFromPath(image_names_chunk))
166
RuntimeError: generator raised StopIteration
Experiment/Environment details
- Tutorial used: AirSimE2EDeepLearning-> DataExplorationAndPreparation
- Environment used: landscape
- Versions of artifacts used (if applicable): Python 3.7.7 Keras 2.3.1 Using Tensorflow backened, h5py
sudoboi commented
I have the exact same problem.