/fast_rosbag_pandas

High-speed import of rosbags into Python

Primary LanguageC++MIT LicenseMIT

fast_rosbag_pandas

A high-speed library for parsing rosbags into Pandas DataFrames. C++ under the hood makes it much faster than approaches using the Python rosbag API.

Example:

>>> from fast_rosbag_pandas import rosbag_to_dataframe
>>> print(rosbag_to_dataframe("example.bag", "stamped_point_topic"))

  header/frame_id  header/seq        header/stamp  point/x  point/y  point/z
0          frame1           0 2020-01-01 00:01:40      1.0      2.0      3.0
1          frame2           1 2020-01-01 00:01:40      4.0      5.0      6.0

Conversion rules are as follows:

  • One column is produced per field of the ROS message
  • Column names are the path to the field, e.g. pose/point/x
  • Column dtypes:
    • ROS string becomes numpy object
    • ROS Time/Duration become numpy datetime64[ns]/timedelta64[ns]
    • All other primitive ROS types are directly mapped to numpy primitives
  • Fixed-size arrays are unpacked into separate columns, with names like foo/bar.2/baz
  • Dynamic-size arrays are skipped

Benchmarks

Benchmarking code is at fast_rosbag_pandas_benchmark.

Graphs of performance over time for this package are on the airspeed velocity site.

Benchmarks comparing fast_rosbag_pandas (latest) to rosbag_pandas (0.5.0.0, pure python implementation) are below. Note: fast_rosbag_pandas doesn't currently construct an index, while rosbag_pandas does.

Bag of StampedPoint messages

rosbag_pandas (s) fast_rosbag_pandas (s) speedup factor
points_1k.bag 0.3186 0.0017 191.6
points_1m.bag 19.7710 0.8971 22.0
points_10m.bag 195.9612 8.4081 23.3

Limitations

This library assumes it is built and run on little-endian architectures only.

Currently, fast_rosbag_pandas does not support:

  • Building an index
  • Dynamic-length arrays

TODOs

  • Expose blob/array handling options
  • Add optional support for dynamic arrays
  • Add optional support for blobs
  • Allow including/excluding fields
  • Support extracting multiple topics at once