pbview
Read protobuf messages without deserialization
If you only need to read a few fields of large Google Protocol Buffer messages you can use this library to do this much faster than with the original google/protobuf parser.
The pbview compiler (pbviewc) generates classes that have the same (read-only) interface as the C++ classes generated by the protobuf compiler (protoc).
Usage
Generate classes the same way, that you generate the regular protobuf c++ messages:
$ pbviewc --cpp_out=out_dir --proto_path=in_dir mymessage.proto
Access binary message fields directly inplace:
#include <mymessage.pbview.h>
MyMessage msg;
msg.set_id(42);
auto binStr = msg.SerializeAsString();
auto view = pbview::View<MyMessage>::fromBytesString(binStr);
REQUIRE(view.id() == 42);
Requirements
- C++17 compiler
- google/protobuf
- range-v3
Features
- The parser seekes fast to the requested fields. Large strings and even sub-messages are skipped in one step
- Working with serialized messages has significant lower memory consumptions than holding deserialized messages in memory
- No memory allocations (std::string_view directly pointing into the serialized message, instead of std::string)
- Variant types that contain either a binary view or a google::protobuf::Message
Drawbacks
Please note, that every field access requires a (partial) parsing of the containing message.
This means, that there is always a break-even point at which the deserialization of the message is faster. (Hybrid approaches are possible: It is easy to deserialize only single sub-messages of larger structs with this library.)
Benchmark your exact use-cases and than make a well-founded decision!
TODO
- Compatibility with proto3 syntax
- Reflection+Descriptor interface
- libfuzzer + asan tests
- Evaluate caching strategies
- value or offset of already accessed fields
- id of first field in each cache-line (for binary search)
- offset of each field (like done by Cap’n Proto or FlatBuffers)
- Support ZeroCopyInputStreams instead of only flat memory data (for compressed data)
- Support uncanonically serialized messages
- fields not ordered by field number (untested)
- repeated fields marked as packed but serialized without packing
License
Boost Software License 1.0