metavision_ros_driver
A combined ROS/ROS2 driver for event based cameras using Prophesee's Metavision SDK. This driver is not written or supported by Prophesee.
If you are looking for more speed and features than the official Prophesee ROS driver you have found the right repository. This driver can cope with the large amount of data produced by Prophesee's Gen3 and Gen4 sensors because it does little more than getting the RAW (currently EVT3 format) events from the camera and publishing them in ROS event_array_msg format.
Accessing the events now requires decoding them using the following ROS/ROS2 packages:
- event_array_codecs has C++ routines to decode event_array_msgs.
- event_array_py module for fast event decoding in python.
- event_array_viewer a node / nodelet that renders and publishes ROS image messages.
- event_array_tools a set of tools to echo, monitor performance and convert event_array_msgs to legacy formats and into "RAW" format.
Supported platforms
Tested on the following platforms:
- ROS Noetic
- ROS2 Galactic / Humble
- Ubuntu 20.04, 22.04 LTS
- Metavision SDK (OpenEB) 2.2.2 - 4.0.1
Tested on the following hardware:
How to build
Prerequisites:
- install Metavision SDK or OpenEB
- install
wstool
(ubuntu package python-wstool or python3-wstool).
Make sure you have your ROS1 or ROS2 environment sourced such that ROS_VERSION is set. For example for ROS1 noetic:
source /opt/ros/noetic/setup.bash
Create a workspace (metavision_ros_driver_ws
), clone this repo, and use wstool
to pull in the remaining dependencies:
mkdir -p ~/metavision_ros_driver_ws/src
cd ~/metavision_ros_driver_ws
git clone git@github.com:berndpfrommer/metavision_ros_driver src/metavision_ros_driver
wstool init src src/metavision_ros_driver/metavision_ros_driver.rosinstall
# or to update an existing space
# wstool merge -t src src/metavision_ros_driver/metavision_ros_driver.rosinstall
# wstool update -t src
Optional (ROS1): to use the efficient recording nodelet clone the nodelet_rosbag repository into our src tree:
git clone -b open_bag_later git@github.com:berndpfrommer/nodelet_rosbag.git src/nodelet_rosbag
Now configure and build:
ROS1:
catkin config -DCMAKE_BUILD_TYPE=RelWithDebInfo # (optionally add -DCMAKE_EXPORT_COMPILE_COMMANDS=1)
catkin build
. devel/setup.bash
ROS2:
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
. install/setup.bash
Driver Features
This driver differs from the Prophesee ROS driver in the following ways:
- publishes event_array_msg that store more densely and are faster to access than the older message formats.
- less CPU consumption by avoiding unnecessary memory copies.
- implemented as nodelet such that it can be run in the same address space as e.g. a rosbag record nodelet without worrying about message loss in transmission.
- prints out message rate statistics so you know when the sensor saturates bandwidth.
- supports these additional features:
- dynamic reconfiguration for bias parameters
- ROI specification
- camera synchronization (stereo)
- external trigger events
- event rate control
Parameters:
bias_file
: path to file with camera biases. See example in thebiases
directory.from_file
: path to Metavision raw file. Instead of opening camera, driver plays back data from this file.serial
: specifies serial number of camera to open (useful for stereo). To learn serial number format first start driver without specifying serial number and look at the log files.event_message_time_threshold
: (in seconds) minimum time span of events to be aggregated in one ROS event message before message is sent. Defaults to 1ms.event_message_size_threshold
: (in bytes) minimum size of events (in bytes) to be aggregated in one ROS event message before message is sent. Defaults to 1MB.statistics_print_interval
: time in seconds between statistics printouts.send_queue_size
: outgoing ROS message send queue size (defaults to 1000 messages).use_multithreading
: decouples the SDK callback from the processing to ensure the SDK does not drop messages (defaults to false). The SDK already queues up messages but there is no documentation on the queue size and no way to determine if messages are dropped. Use multithreading to minimize the risk of dropping messages. However, be aware that this incurs an extra memory copy and threading overhead, raising the maximum CPU load by about 50% of a CPU.frame_id
: the frame id to use in the ROS message headerroi
: sets hardware region of interest (ROI). You can set multiple ROI rectangles with this parameter by concatenation:[top_left_x_1, top_left_y_1, width_1, height_1, top_left_x_2, top_left_y_2, width_2...]
. The length of theroi
parameter vector must therefore be a multiple of 4. Beware that when using multiple ROIs, per Metavision SDK documentation: "Any line or column enabled by a single ROI is also enabled for all the other".erc_mode
: event rate control mode (Gen4 sensor):na
,disabled
,enabled
. Default:na
.erc_rate
: event rate control rate (Gen4 sensor) events/sec. Default: 100000000.sync_mode
: Used to synchronize the time stamps across multiple cameras (tested for only 2). The cameras must be connected via a sync cable, and two separate ROS driver nodes are started, see example launch files. Theprimary
node'sready
topic must be remapped so it receives thesecondary
node'sready
messages. Allowed values:standalone
(default): freerunning camera, no sync.primary
: camera that drives the sync clock. Will not start publishing data until it receives aready
message from the secondary.secondary
: camera receiving the sync clock. Will sendready
messages until it receives a sync signal from the primary.
trigger_in_mode
: Controls the mode of the trigger input hardware. Allowed values:disabled
(default): Does not enable this functionality within the hardwareexternal
: Enables the external hardware pin to route to the trigger input hardware. This will be the pin on the camera's connector.loopback
: Connects the trigger out pin to the trigger input hardware.
trigger_out_mode
: Controls the mode of the trigger output hardware. NOTE: 4-th gen sensors no longer support trigger out! Allowed values:disabled
(default): Does not enable this functionality within the hardwareenabled
: Enables the external hardware pin to route to the trigger in hardware.
trigger_out_period
: Controls the period in microseconds of the trigger out pulse.trigger_out_duty_cycle
: Controls the duty cycle of the trigger out pulse. This is the period ratio.
Services:
save_biases
: write out current bias settings to bias file. For this to work thebias_file
parameter must be set to a non-empty value.
Dynamic reconfiguration parameters (see MetaVision documentation here):
bias_diff
(read only)bias_diff_off
bias_diff_on
bias_fo
bias_hpf
bias_pr
bias_refr
How to use (ROS1):
roslaunch metavision_ros_driver driver_node.launch # (run as node)
roslaunch metavision_ros_driver driver_nodelet.launch # (run as nodelet)
The driver should print out message rate statistics like this:
[ INFO] [1663845360.494651335]: /event_camera: bw in: 106.08359 MB/s, msgs/s in: 51930, out: 0
Prints out the incoming (from the SDK) bandwidth and incoming and
published message rate. In multithreaded mode there will also be shown
the maximum queue size observed during
statistics_print_interval
.
To use the combined driver/recording facility:
roslaunch metavision_ros_driver recording_driver.launch bag:=`pwd`/test.bag
Then start/stop recording like this:
rosrun metavision_ros_driver start_recording.py
rosrun metavision_ros_driver stop_recording.py
To visualize the events, run a viewer
node from the
event_array_viewer:
roslaunch event_array_viewer viewer.launch
How to use (ROS2):
For efficient recording of the events you need to run the driver and the recorder in the same address space as ROS2 composable nodes. For this you will need to install the composable recorder into your workspace as well (see below).
ros2 launch metavision_ros_driver driver_node.launch.py # (run as node)
ros2 launch metavision_ros_driver driver_composition.launch.py # (run as composable node)
The printout should be similar to the one for ROS1.
To visualize the events, run a viewer
node from the
event_array_viewer:
ros2 launch event_array_viewer viewer.launch.py
To use the combined driver/recorder and start the recording:
ros2 launch recording_driver.launch.py
ros2 run metavision_ros_driver start_recording_ros2.py
To stop the recording you have to kill (Ctrl-C) the recording driver.
CPU load
Here are some approximate performance numbers on a 16 thread (8-core) AMD Ryzen 7480h with max clock speed of 2.9GHz. All numbers were obtained by producing maximum event rates about (48Mevs) with a SilkyEVCam:
ROS1
All CPU loads below are with sensor saturating at close to 50Mevs.
settings | single threaded | multi threaded | note |
---|---|---|---|
driver no subscriber | 22% | 59% | no pub, extra copy for multithreaded |
driver with subscriber | 37% | 61% | does interprocess communication |
driver + rosbag record node | 70% | 96% | combined driver + record cpu load |
driver + rosbag record nodelet | 52% | 74% | single process no ipc but disk/io |
ROS2
All CPU loads below are with sensor saturating at close to 50Mevs. Middleware used was cyclonedds.
settings | single threaded | multi threaded | note |
---|---|---|---|
driver no subscriber | 22% | 59% | no pub, extra copy for multithreaded |
driver with subscriber | 35% | 44% | does interprocess communication |
driver + rosbag record node | 80% | 90% | combined driver + record cpu load |
driver + rosbag record composable | 58% | 80% | single process no ipc but disk/io |
About ROS time stamps
The SDK provides hardware event time stamps directly from the camera. Because for efficiency reasons the packets are not decoded the time stamps are not available to the driver. Thus the ROS driver simply puts the host wall clock arrival time into the header's stamp field.
About Trigger Pins
External triggers on prophesee cameras allows for a signal to be injected into the event stream. This is useful for synchronizing external devices. The event stream contained in the packages will now contain trigger events that can be recovered with the decoder.
Prophesee provides documentation on the trigger functions at a high level here.
Trigger out
functionality is exposed through trigger_out_mode
,
trigger_out_period
, and trigger_out_duty_cycle
. These
variables follow the same meaning as laid out in the internal
documentation.
trigger_out_mode
can beenabled
ordisabled
trigger_out_period
can be from 2us to 1h (units are us)trigger_out_duty_cycle
is the pulse width ratio (trigger_out_period * trigger_out_duty_cycle
must be at least 1us)
Trigger in
functionality is abstracted away from pins to just loopback
or
external
as the pin mappings are constant for a given camera
configuration.
trigger_in_mode
allows the user to specify for each cameraloopback
orexternal
and lookup which pins are associated with that camera.
WARNING Running synchronization and triggers at the same time is possible, but requires understanding of your camera's underlying hardware (as most share trigger out and sync out pins).
Hardware configuration
The hardware configuration file is config/trigger_pins.yaml
. The
mappings for hal_plugin_gen*
come from
Prophesee
documentation.
The mapping for evc3a_plugin_gen31
has been validated on the
SilkyEvCam (March 2022). The mapping goes from the HAL Software Info
to pin numbers.
If you camera is not yet supported, the software info is printed out on driver startup. Look for a line that contains:
Plugin Software Name:
This will be the key to place under prophesee_pin_config
which can
then be populated based on your camera's documentation.
WARNING If this file is not loaded (or your camera is not yet supported), the default pin loaded will be 0. This may work in some cases, but not all.
SilkyEvCam
Documentation on the SilkyEvCam pinout can be found here on page 6. This system uses 3.3V logic for both trigger in as well as trigger out.
While the loopback configuration is internal to the chip, the trigger out line will still pulse externally. This is useful if using an event camera to trigger an external system as you will maintain the timing relative to the internal clock (after association between the trigger event and the external system).
Other cameras
External triggering works on SilkyEVCam HD, i.e. you can trigger the SilkyEVCam HD externally the same way as the SilkyEVCam VGA.
License
This software is issued under the Apache License Version 2.0.