Issues with S3DIS Dataset

[lostcolony]

So I'm attempting to replicate the S3DIS dataset results. I've endeavored to set things up to where they are listed in the readme, with the only things I know to be different are I'm using Python 3.6.4, latest Pytorch (since readme says to pull off of master), and Boost 1.67 (since readme says 1.63 or newer).

In running against the unzipped dataset, I'm getting, on Area 1 -

Voxelization into 199 x 206 x 116 grid
Reduced from 1535040 to 188713 points (12.29%)
95% done
Traceback (most recent call last):
File "partition/partition.py", line 155, in
write_features(fea_file, geof, xyz, rgb, graph_nn, labels)
File "/home/ubuntu/dev/spgraph/partition/provider.py", line 344, in write_features
data_file.create_dataset('rgb', data=rgb, dtype='uint8')
File "/home/ubuntu/anaconda3/envs/py360/lib/python3.6/site-packages/h5py/_hl/group.py", line 109, in create_dataset
self[name] = dset
File "/home/ubuntu/anaconda3/envs/py360/lib/python3.6/site-packages/h5py/_hl/group.py", line 277, in setitem
h5o.link(obj.id, self.id, name, lcpl=lcpl, lapl=self._lapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5o.pyx", line 202, in h5py.h5o.link
RuntimeError: Unable to create link (name already exists)

I went ahead and stuck the call in h50.link in a try/except block just to allow things to move on (rather than trying to diagnose what was going on, just to see if maybe it didn't matter), and that allowed things to progress; however, then on Area 3 I got -

Voxelization into 53 x 549 x 92 grid
Reduced from 178968 to 22650 points (12.65%)
88% done
computing the superpoint graph...
minimal partition...
L0-CUT PURSUIT WITH L2 FIDELITY
PARAMETERIZATION = FAST
Graph 22652 vertices and 543600 edges and observation of dimension 7
Iteration 1 - 58 components - Saturation 0.0 % - Quadratic Energy 46.465 % - Timer 0.226557
Iteration 2 - 194 components - Saturation 0.7 % - Quadratic Energy 28.743 % - Timer 0.493281
Iteration 3 - 350 components - Saturation 4.3 % - Quadratic Energy 23.606 % - Timer 0.78682
Iteration 4 - 489 components - Saturation 23.6 % - Quadratic Energy 21.411 % - Timer 1.08708
Iteration 5 - 571 components - Saturation 44.1 % - Quadratic Energy 20.574 % - Timer 1.33033
Max number of iteration reached
computation of the SPG...
Timer : 132.3 / 621.9 / 998.5
2 / 23---> hallway_2
creating the feature file...
Traceback (most recent call last):
File "partition/partition.py", line 123, in
xyz, rgb, labels = read_s3dis_format(data_file)
File "/home/ubuntu/dev/spgraph/partition/provider.py", line 162, in read_s3dis_format
nn = NearestNeighbors(1, algorithm='kd_tree').fit(xyz)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/neighbors/base.py", line 803, in fit
return self._fit(X)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/neighbors/base.py", line 204, in _fit
X = check_array(X, accept_sparse='csr')
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py", line 453, in check_array
_assert_all_finite(array)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py", line 44, in _assert_all_finite
" or a value too large for %r." % X.dtype)
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').

Any help on what might be different would be appreciated.

[loicland]

Hi,

the first issue is the same as issue #25. It should now be fixed with the latest commit, let me know if it fixes the second bug as well.

loic

[lostcolony]

It doesn't appear to. Pulled the latest, cleared prior clouds, features, and superpoint_graphs output directories, undid the try/except I stuck in, re-ran. THe changed code did indeed address the first issue, but I got the second error again, same area (Area 3).

[loicland]

I remember a similar error. It could be due to either typos in the s3dis dataset or packet error during the downloading.

Check the xyz value you got from reading the file area3/hallway2, and find the nan value (with np.isnan and np.where). Once you found the faulty line in question check the txt in /data, there should be an unrecognized character. Fix it and run partition again.

[lostcolony]

Thanks! Yeah, there's apparently a single unprintable character (non-breaking space in between -9.1 and a trailing 0) in hallway_2.txt that causes choking. Interestingly, using np.fromfile will load it fine, but numpy.loadtxt will not.

Using sed to remove unprintable characters will clean the data -

sed -i 's/[^[:print:]]//g' hallway_2.txt

[loicland]

Great, thanks for the tip. I'll add it to the doc soon.

[lostcolony]

Just going to post here any other data corruption issues I encounter, along with fix -

Area_5/hallway_6/Annotations,ceiling1.txt, line 180389 has a null byte instead of a space (I just cleaned it using a text editor, since most of those will show null bytes)

[lostcolony]

Area_6/copyRoom_1/copy_Room_1.txt needs to be moved to Area_6/copyRoom_1/copyRoom_1.txt

[Amrsaeed]

Applying the patch results in an error in Area_3/hallway_2/hallway_2.txt
ValueError: Some errors were detected ! Line #926337 (got 7 columns instead of 6)
The line after applying the batch is 19.302 -9.1 0 1.785 146 137 106 where it should be only 6 columns.
The removed corrupted character was supposedly a number. So needed to make third column into -9.160 in keeping with the pattern in this column