Memory error with realsense occupancygrid

[ldg810]

Hi, I am trying to use cupoch to mapping with L515 Lidar sensor.

I am using python script 'realsense_occupancygrid_viewer.py'

My goal is using cupoch in Jetson Nano Developer Kit but now I am working in laptop.

My hardware environment (DELL XPS Laptop) is

• GTX 1050 Ti with 4042MiB memory
• nvidia driver 460.32.03
• CUDA Version 11.2

At first, the default voxel_size and resolution seems too high (ocgd = cph.geometry.OccupancyGrid(0.05, 512)).
With default configuration, I got error

MemoryError: std::bad_alloc: RMM failure at:/home/runner/work/cupoch/cupoch/third_party/rmm/includermm/mr/device/pool_memory_resource.hpp:188: Maximum pool size exceeded.

So I changed it to ocgd = cph.geometry.OccupancyGrid(0.1, 128).

The script looks working but memory-usage(check with nvidia-smi) surge very quickly and got same memory error in temp = temp.voxel_down_sample(0.03)

I could not found memory related parameters. Can you suggest any solution to solve the problem?

Thank you.

[neka-nat]

Thank you for the question!
Since realsense_occupancygrid_viewer uses a pool allocator, it is recommended to reduce the size of the pool allocator.

import cupoch as cph
cph.initialize_allocator(cph.PoolAllocation, 500000000)  # original size 1000000000

Or maybe the 128 resolution occupancy-grid is still large.

[ldg810]

1000000000

Hi! Thanks for reply. I tried change two factors as you said.

1. size of the pool allocator (initial 1000000000 -> try 500000000 (1/2), 100000000 (1/10), 20000000 (1/50))
2. occupancy-grid resolution (128, 64, 32)

All combination even lowest combination (20000000, 32) results in same problem..

My laptop can't endure even 10 ~ 15 sec.. GPU memory usage surges very quickly.

Can you guess any problem about it?

[neka-nat]

Thanks for trying!
Is that error happening with voxel_down_sampling ? Or is it OccupancyGrid ?

[ldg810]

Thanks for trying!
Is that error happening with voxel_down_sampling ? Or is it OccupancyGrid ?

I checked where error comes.

The error occured in voxel_down_sampling.

[neka-nat]

If you omit voxel_down_sampling and run it, do you get an error?

[ldg810]

If you omit voxel_down_sampling and run it, do you get an error?

If I omit temp = temp.voxel_down_sample(0.03), the same error occur as

Traceback (most recent call last):
  File "realsense_occupancygrid_viewer.py", line 106, in <module>
    ocgd.insert(temp, np.zeros(3))
MemoryError: std::bad_alloc: RMM failure at:/home/runner/work/cupoch/cupoch/third_party/rmm/include/rmm/mr/device/pool_memory_resource.hpp:188: Maximum pool size exceeded

If I also omit ocgd.insert(temp, np.zeros(3)), no error occur and FPS is much faster but no occupancy grid is generated.

If I only omit ocgd.insert(temp, np.zeros(3)) and make temp = temp.voxel_down_sample(0.03) work, it makes same error as I firstly wrote (MemoryError: std::bad_alloc: RMM failure at:/home/runner/work/cupoch/cupoch/third_party/rmm/include/rmm/mr/device/pool_memory_resource.hpp:188: Maximum pool size exceeded)

[neka-nat]

I can't reproduce it in my environment.

• Ubuntu 20.04
• CUDA11.1
• python3.7

What happens if you disable the use of pool memory?

import cupoch as cph
# cph.initialize_allocator(cph.PoolAllocation, 1000000000)

[ldg810]

I can't reproduce it in my environment.

• Ubuntu 20.04
• CUDA11.1
• python3.7

What happens if you disable the use of pool memory?

import cupoch as cph
# cph.initialize_allocator(cph.PoolAllocation, 1000000000)

I didn't think disabling Pool memory allocation. If I remove the line, it works.

realsense_occupancygrid_viewer.py works well with small number of occupancy-grid resolution (256, 128, etc..) but realsense_occupancygrid_with_vo.py make many errors related to CUDA and memory.

With small occupancy-grid resolution(128), works well initially but if I move L515 around(less than 20 sec..), the error

• error 1

Error: operation not supported	phong_shader.cu:332

• error 2

Traceback (most recent call last):
  File "realsense_occupancygrid_with_vo.py", line 95, in <module>
    res, odo_trans, _ = cph.odometry.compute_rgbd_odometry(
MemoryError: std::bad_alloc: CUDA error at: /home/runner/work/cupoch/cupoch/third_party/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:69: cudaErrorMemoryAllocation out of memory

occured. (When error occurs, allocated memory in nvidia-smi is quickly surge to maximum memory)

I previously used octomap for 3D mapping and the result of current cupoch grid is not quietly good since I cannot recognize simple objects(like fan, laptop, monitor) in viewer.

What is the recommended environment for 3d occupancy grid? (like super-heavy desktop with 4ea 2080TI?)

I want to mapping around 100m x 100m x 10m 3d area with drone and computing device is Jetson Xavier NX (the resolution can be very low such as 1.0m 1.0m 1.0m). Do you think cupoch is appropriate for this kind of usage?

[neka-nat]

Perhaps there is not enough memory.
The occupancy grid consumes too much memory for the following things.

• Generating a map with large resolusion
• Generating maps for objects that are farther away.

In your current experiment, the distance to the target object may be too far.
The Jetson NX has 8GB of memory, so it should be more stable than your current system.

[ldg810]

Perhaps there is not enough memory.
The occupancy grid consumes too much memory for the following things.

• Generating a map with large resolusion
• Generating maps for objects that are farther away.

In your current experiment, the distance to the target object may be too far.
The Jetson NX has 8GB of memory, so it should be more stable than your current system.

Thank you for your kind reply. (I tested in laptop environment which is XPS 15 with GTX 1050 Ti and 32GB memory..)

I will try more experiments with cupoch. I think using CUDA for calculating occupancy grid is essential since it consumes lots of computing power if I use only CPU.

I will close this issue. Thank you again.