/sensorable

sensorable is a prototype tool to aid users of Ubisense sensor systems with determining good locations to place sensors in a factory.

Primary LanguageJavaScriptMIT LicenseMIT

sensorable

sensorable is a prototype tool to aid users of Ubisense sensor systems with determining good locations to place sensors in a factory.

sensorable takes a pointcloud file of the factory and visualizes it in the browser. The user can then inspect the factory and place sensor in its virtual representation. sensorable then simulates the range of the sensors and colorizes the environment according to its reachability by sensors.

Installation

  1. Clone this repository.

  2. Install all of the dependencies listed below.

Dependencies

  1. In a terminal, navigate to this repository.
  2. Run the following commands:
cd extrusion
mkdir build && cd build
cmake ..
make

Once you have completed these steps, you can proceed to Usage.

Usage

Make sure you have completed the steps in Installation.

  1. Name and copy the following three files into this repository:
  • model.pcd This is the file with the pointcloud of your factory. If your pointcloud is in a different format, e.g. .laz, you can convert it to .pcd with CloudCompare. Make sure that the pcd file uses the pcd ascii format. To convert a binary pcd file to an ascii pcd file, you can use thepcl_convert_pcd_ascii_binary tool from the Point Cloud Library.
  • trajectory.txt This is the trajectory file generated by GeoSLAM.
  • path.MP4 This is the video for the trajectory generated by GeoSLAM.
    • In a terminal, navigate to this repository.
    • Run python run.py.

  2. Run python3.8 -m http.server to start a webserver.

  3. Open the application in the browser on 0.0.0.0:8000/index.html

The last command opens the application in your browser. It should look similar to this:

  • The factory is visualized in the left view. You can rotate it and movie it around with your left and right mouse buttons.

  • You can add sensors in two ways: Either, use the "Add Sensor" button to add a sensor that you can move around freely. Or, enable the "Place a sensor" feature and click on a wall to add a sensor on that wall.

  • You can delete a sensor by pressing its "x" button in the sensor list.

  • You can visualize the sensor coverage for a slice of the factory by moving around the inspection plane with the slider on the right of the view. The blacker the plane, the better the coverage.

Running tests

sensorable consists of several components, each with its own tests.

kd-tree

To test the kd-tree and its integration with Pointcloud files, navigate to kd_tree and run ./bin/test_kd_tree. Make sure to copy your model.pcd into that directory first for the integration tests to work.

FloorPlan

To test the floor plan, import test/FloorPlanTest.js and run the floorTest function. Make sure to copy the convex hull file to test/model_fastPLY_chull.json and include a floor plan Image as test/floor_plan.jpeg.

Extrusion tool

To run an intergration test of the wall extrusion tool, go to its directory at extrusion/, then go to the integration_tests/ directory and run the shell script run.sh which will automatically recompile the tool binary and run it on a sample model file with 10 planes and compare the results with the correct extractions.

Documentation

SensorControls

Constructor

SensorControls(camera, domElement, orbit, gui)

  • camera: THREE.Camera
  • domElement: DOMElement
  • orbit: THREE.OrbitControl
  • gui: GUI from dat.gui

Methods

.createSensor(position, hAxis, vAxis, height, width, free) : Object3D

  • position : THREE.Vector3 - the position of the new sensor in world coordinates.
  • hAxis : THREE.Vector3 - horizontal translation axis for the sensor, must be perpendicular to vAxis, azimuth will be calculated off from this axis.
  • vAxis : THREE.Vector3 - vertical translation axis for the sensor, must be perpendicular to hAxis.
  • height : Number - how much the sensor can be translated in the vertical direction.
  • width : Number - how much the sensor can be translated in the horizontal direction.
  • free : Boolean - whether the sensor can be freely movable.

The method returns the reference to the created geometry.

.deleteSensor(sensor)

  • sensor : THREE.Object3d - sensor reference

.getSensors() : THREE.Object3D array

return all sensors in the scene.

.getPositionData(sensor, point)

  • sensor : THREE.Object3D
  • point : THREE.Vector3

.selectSensor(s)

  • s : THREE.Object3D - sensor to select.

.updateGizmoSize()

Makes sure the gizmos and the selected sensor are proportional. Should be called in animate().

SensorTest

This module creates a sphere around the sensor and colors it based on azimuth reading.

Contructor

SensorTest(_sensor, _mode, _radius, _segments, _sensorControls)

  • _sensor : THREE.Object3D - sensor you want to test.
  • _mode : must be "azimuth".
  • _radius : Number - the radius of the test sphere.
  • _sengment : Number - the sphere's width and height segments.
  • _sensorControls : SensorControls

Methods

.setMode(mode)

  • mode : must be "azimuth"

.updateColors()

Update the coloring of the sphere when sensor orientation has changed.

InspectPlane

Add this module to the scene if you want to visualise what the sensors can "see".

Constructor

InspectPlane(_plane_size, _plane_fragments, _altitude, _camera, _renderer, _orbit)

  • _plane_size : Number - initial plane size in world unit.
  • _plane_fragments : Whole Number - the initial plane will have 2*_plane_fragments^2 triangles this ratio will be preserved as the plane is scaled.
  • _altitude : Number - initial z coordinate of the plane
  • _camere : THREE.Camera
  • _renderer : THREE.Renderer
  • _orbit : THREE.OrbitControls

Methods

.updateColors(getColor)

Call this whenever the plane or sensor configuration has changed.

  • getColor : function (point : THREE.Vector3) => THREE.Color - figures out the color of each vertex on the plane.

.setModeRotate()

Set TransformControls on the plane to rotation.

.setModeTranslate()

Set TransformControls on the plane to translation.

.setModeScale()

Set TransformControls on the plane to scale.

.removeGizmos()

Detaches TransformControls from the plane.

.addGizmos()

Attaches TransforControls to the plane.

Colorer

Module that interpolates vertex colors for the inspection plane based on sensor data.

Constructor

Colorer(_sensorControls, _plane)

  • _sensorControls : SensorControls
  • _plane : InspectPlane

Properties

.getColor : function (THREE.Vector3) => THREE.Color - intended to be passed to InspectPlane updateColor method.

PlaneModel

Constructor

PlaneModel(_modelJSON, _sensorControls)

  • _modelJSON : path - file from which to read extrapolated wall data.
  • _sensorControls : SensorControls

Methods

.createSensor(mouse, camera) : Object3D

Intersects a ray with a model and places a sensor if mouse is pointing at a wall. Intended to be used on mouse click. Returns the newly created sensor.

  • mouse : THREE.Vector2D - mouse position on the screen
  • camera : THREE.Camera

.intersect(mouse, camera) : THREE.Object3D

Selects a plane that the mouse is currently pointing at.

  • mouse : THREE.Vector2D - mouse position on the screen
  • camera : THREE.Camera

.deleteSelected()

Deletes plane selected via the previous method.

TrajectoryTracking

This feature works only in nodejs http server.

Constructor

TrajectoryTracking(_trajectoryFile, gui, onChange)

  • _trajectoryFile : path - file from which to read trajectory data
  • gui : dat.GUI - a place to attach trajectory slider
  • onChange : function(seconds) - callback when a particular place on the trajectory is selected.

Methods

.updateBallSize(_camera)

Makes the current position marker scale with the zoom. Intended to be called in animate() method.

  • _camera : THREE.Camera

kd-tree

Methods

  • kd_element_t* make_tree(point_t *t, int len, int i): creates the kd-tree from a pointcloud
  • void initialize(): loads model.pcd into memory and creates a kd-tree from it
  • void get_first_intersection(kd_element_t *root, const point_t ray_orig, const point_t ray_dir, point_t **first_intersection): determines the first intersection of the ray ray_orig, ray_dir with the kd-tree at root and stores it in first_intersection.
  • FLOAT_UNIT get_distance_to_first_intersection(FLOAT_UNIT ray_orig_x, FLOAT_UNIT ray_orig_y, FLOAT_UNIT ray_orig_z, FLOAT_UNIT ray_dir_x, FLOAT_UNIT ray_dir_y, FLOAT_UNIT ray_dir_z);: wrapper around get_first_intersection that is suitable for a WebAssembly interface.

extrusion

Configuration parameters

Embedded default values into code, but can be read from a config.json file in the execution directory as well.

  • rectangleDistThreshold: The minimum squared distance between two rectangles in a plane. Used when the plane is split into rectangles.
  • angleModelThreshold: The standard deviation of the angle discrepancy of the plane from the desired model fit (horizontal/vertical).
  • modelDistThreshold: The standard deviation of the distance of the inliers, points within a plane, to the matched plane.
  • rectangleSizeFraction: The minimum number of points to be included into a rectangle from those in the plane in order it to be displayed.
  • pointsModelThreshold: The minimum points to be within a plane from the overall filtered point cloud size in order the plane to be considered valid. Determines the stopping criteria.
  • filterRadius: The radius within which the filtering is done
  • modelIterations: The number of iterations for which the plane fitting algorithm runs when fitting a single plane
  • filterNeighbours: The number of neighbours a point should have within the given radius

Classes

  • Model: A superclass for all models that are about to be exported from the extraction tool
    • protected coefficients: a storage of the model coefficients if it is a PCL model
    • public Json::Value toJSON(): a method that transforms the model to a Json::Value which is written in the output file
  • Rectangle: a child class of Model used for the rectangle representation
    • private vector<pcl::PointXYZ> corners: containing the 4 corners of the rectangle
    • public Json::Value toJSON(): writing the 4 corners and adding a flag rectangle

Methods

  • distXY(pcl::PointXYZ a, pcl::PointXYZ b): finds the distance of the projections of points A and B into the X-Y plane
  • orientation(pcl::PointXYZ a, pcl::PointXYZ b, pcl::PointXYZ c, Eigen::Vector3f normal): finds the oriented area of the triangle given by the points A, B, and C assuming that the direction of the normal is the positive one
  • extract_plane(pcl::PointCloud<pcl::PointXYZ>::Ptr &cloud, pcl::SACSegmentation<pcl::PointXYZ> seg, std::string model_id, int type, int cloud_size): performs the plane fitting, rectangularization of plane, and extracting it from the cloud through the following steps:
    1. Feeds the point cloud to the RANSAC segmentation instance
    2. Runs the plane fitting
    3. Projects all of the inliers onto the matched plane
    4. If the plane is of type 0 it processes it as a floor plane and extracts its convex hull
    5. Otherwise, it is a vertical plane, so
      1. We find its normal
      2. Order the points by orientation, so they can be parsed by a sweep line algorithm
      3. Use a sweep line algorithm to detect horizontal rectangles and then partition every horizontal rectangle into vertical ones
      4. Push back to the clouds the points that were not located into any rectangles
    6. Remove all of the matched points from the cloud, so that new plane detection can take place
  • showHelp(char *filename): Prints a help message to the user with the basic usage of the tool
  • parseCommandLine(int argc, char *argv[], pcl::PointCloud<pcl::PointXYZ>::Ptr cloud): Parses the command-line arguments for the tool, opening and reading the point cloud files and importing the configuration options
  • int main(int argc, char *argv[]): The main function performing the following tasks
    1. Initializing the point clouds
    2. Calling parseCommandLine()
    3. Doing the filtering
    4. Setting up the RANSAC plane fitting algorithm
    5. Extracting the floor
    6. Extracting the walls
    7. Writing the extracted data to the JSON model files

FloorPlane

Methods

  • floorPlane(string image_file, THREE.Renderer renderer, number[] convex ): Uses the convex hull as dictated by x,y coordinates in the convex array where every two values indicate a point and takes in a image file name, returning a Three.Plane object with the image_file overlayed on both sides of the plane and placed at a best-fit rectangle over the convex hull.