/PointCloud2Unity

A Unity project for displaying pointclouds with different shape (using Quads, Cubes, Spheres...) and shaders coming from ROS

Primary LanguageC#

PointCloud2Unity

A Unity package containing a collection of possible implementations for visualizing point clouds. This package contains also a script to decode the PointCloud2 message coming from ROS/ROS2.

Proposed methods:

  1. Creating a single mesh with all the points. Then each point will render a triangle or a disc using the geometry shader. [1]
  2. Exploit GPU Instancing and render directly each points as a single mesh (quad, cube, sphere, ...). [2]
  3. Variant of method 2 in which ComputeBuffers are used to store the data and then render the points.[2,3]
  4. Use of the Unity Particles System. [6]

Each methods uses a particular shader which is located in the same folder of the script.

Method 1

This method creates a single mesh with all the points. Then points will render a triangle or a sphere using the geometry shader. This approach seems more performant than Methods 2-3, but does not have the possibility to draw quads, cubes or other shapes. The material is inherited from the PCL GameObject, so it is possible to switch from points to triangles and viceversa directly from the inspector. This will cause the material to change from PointMeshShader to TriangleMeshShader and viceversa.

Preview

Points Triangles
Drawing points Drawing triangles
Larger Points Larger Triangles
Changing size Changing size

Methods 2 and 3

These approaches exploit GPUs parallelism to render each point as a single mesh (quad, cube, sphere, ...). This is done using the Graphics.RenderMeshInstanced function. MaterialPropertyBlock is also used to set different parameters value in the shaders. The material must be provided as argument in the inspector tab of the script. For Method 2 the material is called PC2GPUMaterial, for Method 3 the material is called BufferedMaterial.

Preview

You can modify the scale of the meshes used to represent each points from the Inspector.

Quads Cubes Spheres
Drawing quads Drawing cubes Drawing spheres
Larger Quads Larger Cubes Larger Spheres
Changing size of quads Changing size of cubes Changing size of spheres

Method 4

This methods exploits the Unity Particle System to render the pointcloud. It is the most efficient way to render a point cloud in Unity given its good integration and parallelization. The only drawback is that you might need to switch from URP to HDRP. For this purpose the implementation is in the branch HDRP since HDRP is incompatible with the materials of the other methods. Different shapes can be used to represent the points. It is sufficient to acces the ShadersGraph asset and connect the Initialize Particle node to a new node like Output Particle Point or Output Particle Triangle or the shape you want to use.

Preview

Small Particles Medium Particles Big Particles
Drawing particles Drawing particles Drawing particles

How to use it:

ROS Users:

  1. Install the ROS-TCP-Connector package from here and ROS-TCP-Endopoint ros-package from here. You can follow this guide.
  2. Set up the ROS-TCP-Connector and ROS-TCP-Endpoint to communicate with your ROS2 environment.
  3. Clone this repository and open the project in Unity.
  4. open one of the Method scene.
  5. Set your topic name and other options in the script inspector of the PCL object.
  6. Activate Use Normals if your PointCloud2 message contains normals.
    (You can check if it does by running:
    ros2 topic echo /your_topic_name
    in your terminal and look for the field normals_. If it is present, then you can activate the option.) This will modify offsets of data in the pointcloud decoder.
  7. Run the scene and you should see the pointcloud being visualized in Unity.

Non-ROS Users:

In this case you have to write your own code to fill the MeshInfo struct. You can look how I decoded the PointCloud2 message in the PointCloud2Decoder script for a reference. Then you can pass your Decoder as a parameter to the main script associated to the method you want to use.

Common Parameters

The three methods share some common parameters:

  • Vertices Max: The maximum number of vertices that will be extracted from the pointcloud for visualization.
  • Topic Name: The name of the topic you want to visualize.
  • Use Normals: If the pointcloud contains normals, this will modify the offsets of the data in the pointcloud decoder algorithm.
  • Point size: The size of the verteces/meshes (works in real time) Normals are also used in Method 1 to rotate triangles towards the camera.

Tips for improving performances

  • Preprocess the pointcloud filtering some points. An simple example with segmentation + voxel grid is provided here.

References

For this project I took inspiration from the following repositories/websites:
[1]: Pcx
[2]: GPU Instancing tutorial
[3]: PointCloud Processing tutorial
[4]: PointCloud Streaming
[5]: Vertex Point Cloud
[6]: Unity Particle System tutorial