Missing point cloud in dataset
yasasa opened this issue · 4 comments
yasasa commented
Hi, I used the script provided to download the dataset and run the training code. There seems to be one point cloud missing in the data though, it's pretty hard for me to pinpoint which one:
$ torchrun --nnodes=1 --nproc_per_node=2 --max_restarts=1 --rdzv_id=42353467 --rdzv_backend=c10d train.py --id train_id_000 --batch_size 8 --setting 02_05_withheld --root_dir /media/ssd/users/yasasa/data2/ --logdir ./ --use_tp 1 --continue_epoch 1 --cpu_cores 4 --num_repetitions 3 --backbone aim --use_semantic 0 --use_depth 0 --use_semantic 0 --use_bev_semantic 0 --detect_boxes 0
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
Start method of multiprocessing: fork
Start method of multiprocessing: fork
RANK, LOCAL_RANK and WORLD_SIZE in environ: 0/0/2
RANK, LOCAL_RANK and WORLD_SIZE in environ: 1/1/2
Rank: 1 Device: cuda:1 Num GPUs on node: 3 Num CPUs on node: 4 Num workers: 1
Setting: 02_05_withheld
Rank: 0 Device: cuda:0 Num GPUs on node: 3 Num CPUs on node: 4 Num workers: 1
Setting: 02_05_withheld
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 159/159 [00:11<00:00, 14.00it/s]
Loading 541944 lidars from 159 folders
Total amount of routes: 6919
Crashed routes: 1
Perfect routes: 6538
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 51/51 [00:04<00:00, 12.68it/s]
0it [00:00, ?it/s]
Loading 0 lidars from 0 folders
Total amount of routes: 0
Crashed routes: 0
Perfect routes: 0
Loading 208536 lidars from 51 folders
Total amount of routes: 2202
Crashed routes: 0
Perfect routes: 2142
Target speed weights: [0.866605263873406, 7.4527377240841775, 1.2281629310898465, 0.5269622904065803]
Angle weights: [204.25901201602136, 7.554315623148331, 0.21388916461734406, 5.476446162657503, 207.86684782608697]
0it [00:00, ?it/s]
Loading 0 lidars from 0 folders
Total amount of routes: 0
Crashed routes: 0
Perfect routes: 0
Total trainable parameters: 27949776
Adjusting learning rate of group 0 to 3.0000e-04.
Adjusting learning rate of group 0 to 3.0000e-04.
86%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 29105/33871 [2:37:26<25:46, 3.08it/s]
Traceback (most recent call last):
File "train.py", line 1019, in <module>
main()
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
return f(*args, **kwargs)
File "train.py", line 624, in main
trainer.train()
File "train.py", line 883, in train
for i, data in enumerate(tqdm(self.dataloader_train, disable=self.rank != 0)):
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/tqdm/std.py", line 1195, in __iter__
for obj in iterable:
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
data = self._next_data()
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1376, in _next_data
return self._process_data(data)
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1402, in _process_data
data.reraise()
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/_utils.py", line 461, in reraise
raise exception
laspy.errors.LaspyException: Caught LaspyException in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/yasasa/carla_garage/team_code/data.py", line 365, in __getitem__
las_object = laspy.read(str(lidars[i], encoding='utf-8'))
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/laspy/lib.py", line 186, in read_las
with open_las(source, closefd=closefd, laz_backend=laz_backend) as reader:
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/laspy/lib.py", line 117, in open_las
return LasReader(stream, closefd=closefd, laz_backend=laz_backend)
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/laspy/lasreader.py", line 49, in __init__
self.header = LasHeader.read_from(source)
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/laspy/header.py", line 525, in read_from
stream = io.BytesIO(cls._prefetch_header_data(stream))
File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/laspy/header.py", line 820, in _prefetch_header_data
raise LaspyException(f"Source is empty")
laspy.errors.LaspyException: Source is empty
Kait0 commented
Can you wrap the LiDAR loading into a try block and tell me the file name that is broken?
"/home/yasasa/carla_garage/team_code/data.py", line 365"
try: las_object = laspy.read(str(lidars[i], encoding='utf-8')) except Exception: print(str(lidars[i], encoding='utf-8'))
yasasa commented
s4_dataset_2023_05_10/Routes_Town04_Scenario4_Repetition2/Town04_Scenario4_route85_05_11_23_07_17/lidar/0099.laz
Kait0 commented
Seems like that file got corrupted when I downloaded it from my compute cluster.
I uploaded the folder again with the file.
Try re-downloading the s4 folder.
https://s3.eu-central-1.amazonaws.com/avg-projects-2/jaeger2023arxiv/dataset/s4_dataset_2023_05_10.zip
yasasa commented
Works now, thanks!