drprojects/superpoint_transformer

generating a 'processed' folder for a personalized dataset that has a structure similar to the S3DIS dataset

Closed this issue · 3 comments

kaoutar-elmouh commented

Dear authors,
First of all, thank you for this excellent work. I have a question regarding the data preprocessing to be included in the 'processed' folder in ''https://github.com/drprojects/superpoint_transformer/blob/master/docs/datasets.md'' for the S3DIS dataset. Could you mention how to generate it?

Best regards,

drprojects commented

Hi @kaoutar-elmouh. If you look at the code, you will see that the 'processed' folder is created and populated automatically. The user is only required to properly organize the data in the raw directory as specified in https://github.com/drprojects/superpoint_transformer/blob/master/docs/datasets.md

kaoutar-elmouh commented

Thank you for your response.

narges-tk commented

Hi @kaoutar-elmouh. If you look at the code, you will see that the 'processed' folder is created and populated automatically. The user is only required to properly organize the data in the raw directory as specified in https://github.com/drprojects/superpoint_transformer/blob/master/docs/datasets.md

I have customized the 'raw' file of S3DIS according to what you mentioned. However, I am getting the following error. Also I have changed the s3dis_config as follows: Thanks in advance for your help.

Changed s3dis_config :
ROOM_TYPES = {
"conferenceRoom": 0,
"auditorium": 1,
"roby": 2

}

VALIDATION_ROOMS = [
"roby_1",'conferenceRoom_1']

ROOMS = {
"Area_1": [
"conferenceRoom_1",
"conferenceRoom_2"],
"Area_2": [
"auditorium_1",
"auditorium_2"
],
"Area_3": [
"roby_1"]
}

Error:
**Extracting /home/ntakhtke/superpoint_transformer/data/s3dis/Stanford3dDataset_v1.2.zip
Processing...
30%|███████████████████████████████████▋ | 3/10 [01:34<03:40, 31.52s/it]
[2024-06-04 20:03:11,064][src.utils.utils][ERROR] -
Traceback (most recent call last):
File "/home/ntakhtke/superpoint_transformer/src/utils/utils.py", line 45, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "src/eval.py", line 105, in evaluate
trainer.test(model=model, datamodule=datamodule, ckpt_path=cfg.ckpt_path)
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 754, in test
return call._call_and_handle_interrupt(
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 794, in _test_impl
results = self._run(model, ckpt_path=ckpt_path)
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 947, in _run
self._data_connector.prepare_data()
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
call._call_lightning_datamodule_hook(trainer, "prepare_data")
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 179, in _call_lightning_datamodule_hook
return fn(*args, **kwargs)
File "/home/ntakhtke/superpoint_transformer/src/datamodules/base.py", line 144, in prepare_data
self.dataset_class(
File "/home/ntakhtke/superpoint_transformer/src/datasets/s3dis.py", line 275, in init
super().init(*args, val_mixed_in_train=True, **kwargs)
File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 223, in init
super().init(root, transform, pre_transform, pre_filter)
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in init
super().init(root, transform, pre_transform, pre_filter, log)
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 97, in init
self._process()
File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 647, in _process
self.process()
File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 682, in process
self._process_single_cloud(p)
File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 698, in _process_single_cloud
data = self.sanitized_read_single_raw_cloud(raw_path)
File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 800, in sanitized_read_single_raw_cloud
data = self.read_single_raw_cloud(raw_cloud_path)
File "/home/ntakhtke/superpoint_transformer/src/datasets/s3dis.py", line 392, in read_single_raw_cloud
return read_s3dis_area(
File "/home/ntakhtke/superpoint_transformer/src/datasets/s3dis.py", line 75, in read_s3dis_area
batch = Batch.from_data_list(starmap_with_kwargs(
File "/home/ntakhtke/superpoint_transformer/src/data/data.py", line 925, in from_data_list
batch = super().from_data_list(
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/torch_geometric/data/batch.py", line 80, in from_data_list
add_batch=not isinstance(data_list[0], Batch),
IndexError: list index out of range
[2024-06-04 20:03:11,067][src.utils.utils][INFO] - Closing loggers...
Error executing job with overrides: ['experiment=semantic/s3dis', 'datamodule.fold=5', 'ckpt_path=spt-2_s3dis_fold5.ckpt']
Traceback (most recent call last):
File "src/eval.py", line 117, in main
evaluate(cfg)
File "/home/ntakhtke/superpoint_transformer/src/utils/utils.py", line 48, in wrap
raise ex
File "/home/ntakhtke/superpoint_transformer/src/utils/utils.py", line 45, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "src/eval.py", line 105, in evaluate
trainer.test(model=model, datamodule=datamodule, ckpt_path=cfg.ckpt_path)
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 754, in test
return call._call_and_handle_interrupt(
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 794, in _test_impl
results = self._run(model, ckpt_path=ckpt_path)
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 947, in _run
self._data_connector.prepare_data()
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
call._call_lightning_datamodule_hook(trainer, "prepare_data")
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 179, in _call_lightning_datamodule_hook
return fn(*args, **kwargs)
File "/home/ntakhtke/superpoint_transformer/src/datamodules/base.py", line 144, in prepare_data
self.dataset_class(
File "/home/ntakhtke/superpoint_transformer/src/datasets/s3dis.py", line 275, in init
super().init(*args, val_mixed_in_train=True, kwargs)
File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 223, in init
super().init(root, transform, pre_transform, pre_filter)
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in init
super().init(root, transform, pre_transform, pre_filter, log)
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 97, in init
self._process()
File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 647, in _process
self.process()
File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 682, in process
self._process_single_cloud(p)
File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 698, in _process_single_cloud
data = self.sanitized_read_single_raw_cloud(raw_path)
File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 800, in sanitized_read_single_raw_cloud
data = self.read_single_raw_cloud(raw_cloud_path)
File "/home/ntakhtke/superpoint_transformer/src/datasets/s3dis.py", line 392, in read_single_raw_cloud
return read_s3dis_area(
File "/home/ntakhtke/superpoint_transformer/src/datasets/s3dis.py", line 75, in read_s3dis_area
batch = Batch.from_data_list(starmap_with_kwargs(
File "/home/ntakhtke/superpoint_transformer/src/data/data.py", line 925, in from_data_list
batch = super().from_data_list(
File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/torch_geometric/data/batch.py", line 80, in from_data_list
add_batch=not isinstance(data_list[0], Batch),
IndexError: list index out of range