/3d-cuboid-cashmission

Primary LanguageJavaScriptOtherNOASSERTION

3D Bounding Box Annotation Tool (3D BAT)

3D Bounding Box Annotation Tool

Installation

  1. Clone repository: git clone https://github.com/walzimmer/bat-3d.git
  2. Install npm
  3. Install PHP Storm or WebStorm (IDE with integrated web server): https://www.jetbrains.com/phpstorm/download/download-thanks.html
  4. [OPTIONAL] Install WhatPulse to measure the number of clicks and key strokes while labeling: https://whatpulse.org/
  5. Open folder bat-3d in PHP Storm.
  6. Move into directory: cd bat-3d.
  7. Download sample scenes extracted from the NuScenes dataset from here and extract the content into the bat-3d/input/ folder.
  8. Install required packages: npm install
  9. Open index.html with chromium-browser (Linux) or Chrome (Windows) within the IDE. Right click on index.html -> Open in Browser -> Chrome/Chromium

Overview

Overview

Paper

Paper Reference: https://arxiv.org/abs/1905.00525

Animation

Animation

Video

Video Link: https://www.youtube.com/watch?v=gSGG4Lw8BSU

Annotate your own data (point cloud and image or point cloud only)

To annotate your own data, follow this steps:

  1. Create a new folder under the input folder (e.g. bat-3d/input/waymo)
  2. For each annotation sequence create a separate folder e.g.

input/waymo/20210103_waymo

input/waymo/20210104_waymo

  1. Under each sequence create the following folders:

input/waymo/20210103_waymo/annotations (this folder will contain the downloaded annotations)

input/waymo/20210103_waymo/pointclouds (place your point cloud scans (in .pcd ascii format) here)

input/waymo/20210103_waymo/pointclouds_without_ground (optional: Remove the ground using the scripts/nuscenes_devkit/python-sdk/scripts/export_pointcloud_without_ground_nuscenes.py script to use the checkbox "Filter ground". Change the threshold of -1.7 to the height of the LiDAR sensor.)

input/waymo/20210103_waymo/images (optional: For each camera image, create a folder: e.g. CAM_BACK, CAM_BACK_LEFT, CAM_BACK_RIGHT, CAM_FRONT, CAM_FRONT_LEFT, CAM_FRONT_RIGHT)

Make sure, that the LiDAR scan file names follow this naming format 000000.pcd, 000001.pcd, 000002.pcd and so on. Same for image file names: 000000.png, 000001.png, 000002.png) and the annotation file names: 000000.json, 000001.json, 000002.json).

  1. When annotating only point cloud data, make sure to set this variable to true in js/base_label_tool.js:

pointCloudOnlyAnnotation: true

  1. Open the annotation tool in you web browser and change the dataset from NuScenes to your own dataset (e.g. waymo) in the drop down field.

3D Bounding Box Labelling Instructions

  1. Watch raw video (10 sec) to get familiar with the sequence and to see where interpolation makes sense
  2. Watch tutorial videos to get familiar with (translation/scaling/rotating objects, interpolation and how to use helper views)
  3. Start WhatPulse. Login with koyunujiju@braun4email.com and password: labeluser
  4. Draw bounding box in the Bird's-Eye-View (BEV)
  5. Move/Scale it in BEV using 3D arrows (drag and drop) or sliders
  6. Choose one of the 5 classes (Car, Pedestrian, Cyclist, Motorbike, Truck)
  7. Interpolate if necessary
    1. Select Object to interpolate by clicking on a Bounding Box
    2. Activate 'Interpolation Mode' in the menu (checkbox) -> start position will be saved
    3. Move to desired frame by skipping x frames
    4. Translate object to new position
    5. Click on the 'Interpolate' button in the menu
  8. Repeat steps 4-7 for all objects in the sequence
  9. Download labels to your computer (JSON file)
  10. Stop the time after labeling is done.
  11. Make screenshots of keyboard and mouse heat map, record number of clicks and keystrokes

Keyboard Shortcuts

Key Description
V Toggle view (3D view/Bird's-Eye-View)
W Move forward in 3D view (TODO)
A Move left in 3D view (TODO)
S Move backward in 3D view (TODO)
D Move right in 3D view (TODO)
Q Move down in 3D view (TODO)
E Move up in 3D view (TODO)
C Enlarge camera image
N Next frame
P Previous frame
I Interpolate
CTRLMOUSELEFT Snap to grid in 0.5m steps (floor alignment mode)
T Enable/Disable Translation mode
W Move selected object forward (TODO)
A Move selected object to left (TODO)
S Move selected object backward (TODO)
D Move selected object to right (TODO)
Q Move selected object down (TODO)
E Move selected object up (TODO)
R Enable/Disable Rotation mode
LEFT Rotate selected object counter-clock-wise (TODO)
RIGHT Rotate selected object clock-wise (TODO)
Y Enable/Disable Scaling mode
UP Increase length along longitudinal axis (y-axis) (TODO)
DOWN Decrease length along longitudinal axis (y-axis) (TODO)
LEFT Decrease width along lateral axis (x-axis) (TODO)
RIGHT Increase width along lateral axis (x-axis) (TODO)
SHIFTRIGHT Increase height along vertical axis (z-axis) (TODO)
CTRLRIGHT Decrease height along vertical axis (z-axis) (TODO)
PLUS Increase arrow size
MINUS Decrease arrow size
X Show/Hide X-axis
Y Show/Hide Y-axis
Z Show/Hide Z-axis (only in 3D mode)
SPACE Switch between different operation modes (translate, rotate, scale) (TODO)
TAB Select next object (TODO)
DEL OR BACKSPACE Delete selected object (TODO)
M Marking mode (TODO)
LEFT Move orientation of object counter-clock-wise (TODO)
RIGHT Move orientation of object clock-wise (TODO)
ONE Select class CAR (TODO)
TWO Select class Truck (TODO)
THREE Select class Motorcycle (TODO)
FOUR Select class Bicycle (TODO)
FIVE Select class Pedestrian (TODO)
MOUSELEFT On a 2D/3D object: Show bounding box
On a camera image: Enlarge/Shrink camera image (TODO)
On ego vehicle: Show field-of-view (TODO)
LEFTRIGHT: Switch FOV to next channel
R Reset all selected bounding boxes (TODO)
F11 Full Screen Mode
P Play video (TODO)
K Keyboard navigation (only in 3D view) (TODO)
L Toggle Lighting (TODO)
LONE Label random color (TODO)
LTWO Label class color (TODO)
T Show/Hide trajectory (TODO)
SHIFTS Save current screen into file (requires request to server) (TODO)
SHIFTD Download annotation file (TODO)
CTRLZ Undo operation (TODO)
CTRLY Redo operation (TODO)
CTRLW Close tab (exit)
G Show/Hide grid (TODO)
H Toggle aggregated pointcloud (TODO)
J Hide all labels except selected object (Press again to show all labels) (TODO)
QUESTIONMARK Show keyboard shortcuts
ESC Unselect box (TODO)
Quit fullscreen cam image (TODO)
ALTMOUSELEFT Copy bounding box (by dragging) (TODO)

Hints:

  • Select Copy label to next frame checkbox if you want to keep the label (position, size, class) for next frame
  • Use helper views to align object along z-axis (no need to switch into 3D view)
  • Label one object from start to end (using interpolation) and then continue with next object
  • Do not apply more than one box to a single object.
  • Check every cuboid in every frame, to make sure all points are inside the cuboid and look reasonable in the image view.
  • The program has been quite stable in my use cases, but there is no guarantee that it won't crash. So please back up (download) your annotated scenes (~every 10 min). Saving to local storage (browser) is done automatically.
  • Download the annotation file into the following folder: bat-3d/input/<DATASET>/<SEQUENCE>/annotations
  • Please open new issue tickets on Github for questions and bug reports or write me an email (wzimmer@eng.ucsd.edu). Thanks!

Special Rules

  • Minimum LIDAR Points :
    • Label any target object containing at least 10 LIDAR points, as long as you can be reasonably sure you know the location and shape of the object. Use your best judgment on correct cuboid position, sizing, and heading.
  • Cuboid Sizing :
    • Cuboids must be very tight. Draw the cuboid as close as possible to the edge of the object without excluding any LIDAR points. There should be almost no visible space between the cuboid border and the closest point on the object.
  • Extremities :
    • If an object has extremities (eg. arms and legs of pedestrians), then the bounding box should include the extremities.
    • Exception: Do not include vehicle side view mirrors. Also, do not include other vehicle extremities (crane arms etc.) that are above 1.5 meters high.
  • Carried Object :
    • If a pedestrian is carrying an object (bag, umbrella, tools etc.), such objects will be included in the bounding box for the pedestrian. If two or more pedestrians are carrying the same object, the bounding box of only one of them will include the object.
  • Use Images when Necessary:
    • For objects with few LIDAR points, use the images to make sure boxes are correctly sized. If you see that a cuboid is too short in the image view, adjust it to cover the entire object based on the image view.

Labels

For every bounding box, include one of the following labels:

  1. Car: Vehicle designed primarily for personal use, e.g. sedans, hatch-backs, wagons, vans, mini-vans, SUVs, jeeps and pickup trucks (a pickup truck is a light duty truck with an enclosed cab and an open or closed cargo area; a pickup truck can be intended primarily for hauling cargo or for personal use).

  2. Truck: Vehicles primarily designed to haul cargo including lorrys, trucks.

  3. Motorcycle: Gasoline or electric powered 2-wheeled vehicle designed to move rapidly (at the speed of standard cars) on the road surface. This category includes all motorcycles, vespas and scooters. It also includes light 3-wheel vehicles, often with a light plastic roof and open on the sides, that tend to be common in Asia (rickshaws). If there is a rider and/or passenger, include them in the box.

  4. Bicycle: Human or electric powered 2-wheeled vehicle designed to travel at lower speeds either on road surface, sidewalks or bicycle paths. If there is a rider and/or passenger, include them in the box.

  5. Pedestrian: An adult/child pedestrian moving around the cityscape. Mannequins should also be annotated as Pedestrian.

Detailed Instructions and Examples

Bounding Box color convention in example images:

  • Green: Objects like this should be annotated

Car

  • Vehicle designed primarily for personal use, e.g. sedans, hatch-backs, wagons, vans, mini-vans, SUVs and jeeps.

    • If it is primarily designed to haul cargo it is a truck.

Truck

  • Vehicles primarily designed to haul cargo including lorrys, trucks, pickup truck (a pickup truck is a light duty truck with an enclosed cab and an open or closed cargo area; a pickup truck can be intended primarily for hauling cargo or for personal use).

Motorcycle

  • Gasoline or electric powered 2-wheeled vehicle designed to move rapidly (at the speed of standard cars) on the road surface. This category includes all motorcycles, vespas and scooters. It also includes light 3-wheel vehicles, often with a light plastic roof and open on the sides, that tend to be common in Asia (rickshaws).

    • If there is a rider, include the rider in the box.
    • If there is a passenger, include the passenger in the box.
    • If there is a pedestrian standing next to the motorcycle, do NOT include in the annotation.

Bicycle

  • Human or electric powered 2-wheeled vehicle designed to travel at lower speeds either on road surface, sidewalks or bicycle paths.

    • If there is a rider, include the rider in the box
    • If there is a passenger, include the passenger in the box
    • If there is a pedestrian standing next to the bicycle, do NOT include in the annotation

Pedestrian

  • An adult/child pedestrian moving around the cityscape.

    • Mannequins should also be treated as pedestrian.

License

Copyright © 2019 The Regents of the University of California

All Rights Reserved. Permission to copy, modify, and distribute this tool for educational, research and non-profit purposes, without fee, and without a written agreement is hereby granted, provided that the above copyright notice, this paragraph and the following three paragraphs appear in all copies. Permission to make commercial use of this software may be obtained by contacting:

Office of Innovation and Commercialization
9500 Gilman Drive, Mail Code 0910
University of California
La Jolla, CA 92093-0910
(858) 534-5815
innovation@ucsd.edu

This tool is copyrighted by The Regents of the University of California. The code is supplied “as is”, without any accompanying services from The Regents. The Regents does not warrant that the operation of the tool will be uninterrupted or error-free. The end-user understands that the tool was developed for research purposes and is advised not to rely exclusively on the tool for any reason.

IN NO EVENT SHALL THE UNIVERSITY OF CALIFORNIA BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS, ARISING OUT OF THE USE OF THIS TOOL, EVEN IF THE UNIVERSITY OF CALIFORNIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. THE UNIVERSITY OF CALIFORNIA SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE TOOL PROVIDED HEREUNDER IS ON AN “AS IS” BASIS, AND THE UNIVERSITY OF CALIFORNIA HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.