Training the model on my own dataset
ImaneTopo opened this issue · 2 comments
@drprojects
Hi, I want to train the model on my own dataset wish is similar to the format of s3dis, but I face this error :
[2024-06-22 11:09:24,068][__main__][INFO] - Starting training!
Processing...
0% 0/90 [00:18<?, ?it/s]
File "/usr/local/lib/python3.10/dist-packages/torch_geometric/utils/coalesce.py", line 93, in coalesce
nnz = edge_index.size(1)
TypeError: 'int' object is not callable
[2024-06-22 12:01:07,631][src.utils.utils][INFO] - Closing loggers...
[2024-06-22 12:01:07,631][src.utils.utils][INFO] - Closing wandb!
wandb:
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./logs/train/runs/2024-06-22_12-00-44/wandb/run-20240622_120047-2hq7r3g0/logs
wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with wandb.require("core")! See https://wandb.me/wandb-core for more information.
Error executing job with overrides: ['experiment=semantic/s3dis_11g', 'datamodule.fold=5']
Traceback (most recent call last):
File "/content/drive/MyDrive/superpoint_model/src/train.py", line 139, in main
metric_dict, _ = train(cfg)
File "/content/drive/MyDrive/superpoint_model/src/utils/utils.py", line 48, in wrap
raise ex
File "/content/drive/MyDrive/superpoint_model/src/utils/utils.py", line 45, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "/content/drive/MyDrive/superpoint_model/src/train.py", line 114, in train
trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 543, in fit
call._call_and_handle_interrupt(
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 946, in _run
self._data_connector.prepare_data()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 93, in prepare_data
call._call_lightning_datamodule_hook(trainer, "prepare_data")
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 181, in _call_lightning_datamodule_hook
return fn(*args, **kwargs)
File "/content/drive/MyDrive/superpoint_model/src/datamodules/base.py", line 144, in prepare_data
self.dataset_class(
File "/content/drive/MyDrive/superpoint_model/src/datasets/s3dis.py", line 271, in __init__
super().__init__(*args, val_mixed_in_train=True, **kwargs)
File "/content/drive/MyDrive/superpoint_model/src/datasets/base.py", line 223, in __init__
super().__init__(root, transform, pre_transform, pre_filter)
File "/usr/local/lib/python3.10/dist-packages/torch_geometric/data/in_memory_dataset.py", line 57, in __init__
super().__init__(root, transform, pre_transform, pre_filter, log)
File "/usr/local/lib/python3.10/dist-packages/torch_geometric/data/dataset.py", line 97, in __init__
self._process()
File "/content/drive/MyDrive/superpoint_model/src/datasets/base.py", line 647, in _process
self.process()
File "/content/drive/MyDrive/superpoint_model/src/datasets/base.py", line 682, in process
self._process_single_cloud(p)
File "/content/drive/MyDrive/superpoint_model/src/datasets/base.py", line 710, in _process_single_cloud
nag = self.pre_transform(data)
File "/usr/local/lib/python3.10/dist-packages/torch_geometric/transforms/compose.py", line 24, in __call__
data = transform(data)
File "/content/drive/MyDrive/superpoint_model/src/transforms/transforms.py", line 23, in __call__
return self._process(x)
File "/content/drive/MyDrive/superpoint_model/src/transforms/graph.py", line 666, in _process
nag = self._process_edge_features_for_single_level(
File "/content/drive/MyDrive/superpoint_model/src/transforms/graph.py", line 681, in _process_edge_features_for_single_level
edge_index, se_point_index, se_id = subedges(
File "/content/drive/MyDrive/superpoint_model/src/utils/graph.py", line 144, in subedges
edge_index = to_trimmed(edge_index)
File "/content/drive/MyDrive/superpoint_model/src/utils/graph.py", line 410, in to_trimmed
edge_index = coalesce(edge_index)
File "/usr/local/lib/python3.10/dist-packages/torch_geometric/utils/coalesce.py", line 93, in coalesce
nnz = edge_index.size(1)
TypeError: 'int' object is not callable
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Hi @ImaneTopo
This suggests your edge_index
(ie the graph RadiusHorizontalGraph
is trying to build) has spurious shape (ie probably no edge at all). Based on your initial error that you modified, I guess one of your point clouds is quite small and there is not enough superpoints at a given partition level. If one of the partition levels has only 1 superpoint, problematic downstream effects might occur.
So, investigate the size of your partition levels and make sure there is never only 1 superpoint. To this end you have two options:
- adjust the regularization parameters of your partition (eg
pcp_regularization
,pcp_cutoff
,pcp_spatial_weight
). Have a look at the documentation, past issues, and code, for how those work and suggestions on how to adjust those - remove the 3rd partition level altogether if this is the one causing errors, as we do not use it in practice (it seems your initial error was connected to this)
If this error persists even after you have made sure no partition level contains only 1 superpoint, this means some partition level is unable to build any connection between superpoints (ie superpoints are unable to see far enough). To fix this, you must adjust graph_gap
to allow superpoints to be connected to farther neighbors.
PS: If you ❤️ or use this project, don't forget to give it a ⭐, it means a lot to us !
PS2: I will be giving a live tutorial on SPT on Thursday, you might want to attend: https://www.linkedin.com/events/superpointtransformersfor3dpoin7209130538110963712