issue: dataloader error

Question

issue: dataloader error

inhopark94 opened this issue 2 years ago · 1 comments

dataset: SyntheticDataset_gaussian
2022-08-18 11:16:31 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_lines.
2022-08-18 11:16:31 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_lines.tar.gz
2022-08-18 11:16:32 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_polygon.
2022-08-18 11:16:32 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_polygon.tar.gz
2022-08-18 11:16:32 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_multiple_polygons.
2022-08-18 11:16:32 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_multiple_polygons.tar.gz
2022-08-18 11:16:34 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_ellipses.
2022-08-18 11:16:34 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_ellipses.tar.gz
2022-08-18 11:16:34 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_star.
2022-08-18 11:16:34 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_star.tar.gz
2022-08-18 11:16:34 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_checkerboard.
2022-08-18 11:16:34 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_checkerboard.tar.gz
2022-08-18 11:16:35 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_stripes.
2022-08-18 11:16:35 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_stripes.tar.gz
2022-08-18 11:16:35 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_cube.
2022-08-18 11:16:35 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_cube.tar.gz
2022-08-18 11:16:35 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive gaussian_noise.
2022-08-18 11:16:35 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\gaussian_noise.tar.gz
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_lines.
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_lines.tar.gz
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_polygon.
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_polygon.tar.gz
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_multiple_polygons.
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_multiple_polygons.tar.gz
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_ellipses.
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_ellipses.tar.gz
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_star.
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_star.tar.gz
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_checkerboard.
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_checkerboard.tar.gz
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_stripes.
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_stripes.tar.gz
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive draw_cube.
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\draw_cube.tar.gz
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO Extracting archive for primitive gaussian_noise.
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO tar_path: datasets\synthetic_shapes_v6\gaussian_noise.tar.gz
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO == train split size 66048 in 1032 batches
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO == val split size 5312 in 83 batches
Load Train_model_heatmap!!
check config!! {'train_iter': 200000, 'save_interval': 2000, 'tensorboard_interval': 1000, 'model': {'subpixel': {'enable': False}, 'name': 'SuperPointNet_gauss2', 'params': {}, 'detector_loss': {'loss_type': 'softmax'}, 'batch_size': 64, 'eval_batch_size': 16, 'learning_rate': 0.001, 'kernel_reg': 0.0, 'detection_threshold': 0.001, 'nms': 4, 'lambda_loss': 0, 'dense_loss': {'enable': False, 'params': {'descriptor_dist': 4, 'lambda_d': 800}}, 'sparse_loss': {'enable': True, 'params': {'num_matching_attempts': 1000, 'num_masked_non_matches_per_match': 100, 'lamda_d': 1, 'dist': 'cos', 'method': '2d'}}, 'other_settings': 'train 2d, gauss 0.5'}, 'data': {'gaussian_label': {'enable': False, 'params': {'GaussianBlur': {'sigma': 0.2}}}, 'dataset': 'SyntheticDataset_gaussian', 'primitives': 'all', 'truncate': {'draw_ellipses': 0.3, 'draw_stripes': 0.2, 'gaussian_noise': 0.1}, 'cache_in_memory': True, 'suffix': 'v6', 'add_augmentation_to_test_set': False, 'preprocessing': {'blur_size': 21, 'resize': [120, 160]}, 'augmentation': {'photometric': {'enable': True, 'enable_train': True, 'enable_val': False, 'primitives': ['random_brightness', 'random_contrast', 'additive_speckle_noise', 'additive_gaussian_noise', 'additive_shade', 'motion_blur'], 'params': {'random_brightness': {'max_abs_change': 75}, 'random_contrast': {'strength_range': [0.3, 1.8]}, 'additive_gaussian_noise': {'stddev_range': [0, 15]}, 'additive_speckle_noise': {'prob_range': [0, 0.0035]}, 'additive_shade': {'transparency_range': [-0.5, 0.8], 'kernel_size_range': [50, 100]}, 'motion_blur': {'max_kernel_size': 7}}}, 'homographic': {'enable': True, 'enable_train': True, 'enable_val': False, 'params': {'translation': True, 'rotation': True, 'scaling': True, 'perspective': True, 'scaling_amplitude': 0.2, 'perspective_amplitude_x': 0.2, 'perspective_amplitude_y': 0.2, 'patch_ratio': 0.8, 'max_angle': 1.57, 'allow_artifacts': True, 'translation_overflow': 0.05}, 'valid_border_margin': 2}}, 'warped_pair': {'enable': False, 'params': {'translation': True, 'rotation': True, 'scaling': True, 'perspective': True, 'scaling_amplitude': 0.2, 'perspective_amplitude_x': 0.2, 'perspective_amplitude_y': 0.2, 'patch_ratio': 0.85, 'max_angle': 1.57, 'allow_artifacts': True}, 'valid_border_margin': 3}}, 'front_end_model': 'Train_model_heatmap', 'retrain': True, 'reset_iter': True, 'validation_interval': 1000, 'validation_size': 10, 'train_show_interval': 1000, 'seed': 0}
use sparse_loss!
========== check!!! ==========
learning_rate: 0.001
lambda_loss: 0
detection_threshold: 0.001
batch_size: 64
========== descriptor: sparse ==========
num_matching_attempts : 1000
num_masked_non_matches_per_match : 100
lamda_d : 1
dist : cos
method : 2d

set writer
set train loader
set train loader
model: SuperPointNet_gauss2
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO => creating model: SuperPointNet_gauss2
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO => setting adam solver
adam optimizer
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO New model
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO reset iterations to 0
=== Let's use 1 GPUs!
adam optimizer
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO n_iter: 0
2022-08-18 11:16:36 DESKTOP-OE57NE6 root[4788] INFO max_iter: 200000
epoch: 0
get dataloader
Traceback (most recent call last):
File "train4.py", line 141, in
args.func(config, output_dir, args)
File "train4.py", line 41, in train_base
return train_joint(config, output_dir, args)
File "train4.py", line 93, in train_joint
train_agent.train()
File "C:\Users\Hanwha\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Python 3.8\test4\pytorch-superpoint-master\pytorch-superpoint-master\Train_model_frontend.py", line 275, in train
for i, sample_train in tqdm(enumerate(self.train_loader)):
File "C:\Users\Hanwha\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Python 3.8\test4\pytorch-superpoint-master\pytorch-superpoint-master\python37\lib\site-packages\torch\utils\data\dataloader.py", line 444, in iter
return self._get_iterator()
File "C:\Users\Hanwha\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Python 3.8\test4\pytorch-superpoint-master\pytorch-superpoint-master\python37\lib\site-packages\torch\utils\data\dataloader.py", line 390, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "C:\Users\Hanwha\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Python 3.8\test4\pytorch-superpoint-master\pytorch-superpoint-master\python37\lib\site-packages\torch\utils\data\dataloader.py", line 1077, in init
w.start()
File "C:\Users\Hanwha\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "C:\Users\Hanwha\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Hanwha\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Users\Hanwha\AppData\Local\Programs\Python\Python37\lib\multiprocessing\popen_spawn_win32.py", line 65, in init
reduction.dump(process_obj, to_child)
File "C:\Users\Hanwha\AppData\Local\Programs\Python\Python37\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
File "C:\Users\Hanwha\AppData\Local\Programs\Python\Python37\lib\multiprocessing\pool.py", line 535, in reduce
'pool objects cannot be passed between processes or pickled'
NotImplementedError: pool objects cannot be passed between processes or pickled

(python37) C:\Users\Hanwha\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Python 3.8\test4\pytorch-superpoint-master\pytorch-superpoint-master>Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Hanwha\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\Users\Hanwha\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

I met some error about dataloader.
Do you have any solution about the error?

An execution code was "python train4.py train_base configs/magicpoint_shapes_pair.yaml magicpoint_synth --eval"

Answer 1 · 2022-08-18T07:13:59.000Z

It was a dataloader error.
Window environment can make the error.
So I changed the num_worker of dataloader to 0 and made a lot of empty space at C drive.