protobuf-php/protobuf

Add support for message delimiting

Opened this issue · 2 comments

Google's Java implementation of protobuf provides the writeDelimitedTo/parseDelimitedFrom methods, basically just prefixing individual messages with their length and thus allowing multiple messages to be combined in one output stream.

I've been trying to cobble this together myself, but I think it would be nice to have as part of the library as well. This would make API creation using your library much easier, since you might wanna support requests querying multiple resources/messages of one kind.

If you're not likely to implement this, could you maybe point me in the right direction for doing this myself? It's no problem for small messages (where the length fits into 8 bytes)

pack("c*", strlen($message->toStream()));

but for larger messages, packing with v* or V* produces errors on the client's side, because their length needs to be converted to a varint.

Hey you don`t need do 'strlen' on Stream. You can use Stream::getSize() and prefix message with that value.
If your message is very tiny you need only 1 or 2 bytes for encode size.

Write:

$stream = $message->toStream();
$packedSize = pack('n', $stream->getSize());
$data = $packedSize.$stream->getContents();
fwrite($file, $data);

Read:

$packedSize = fread($file, 2);
$size = unpack('n' $packedSize)[1];
$messageData = fread($file, $size);
$message = new YourMessage(Stream::fromString($messageData));

And its good idea to add that functionality to lib.

You can probably use YourMessage#writeTo() and YourMessage#serializedSize() to write multiple messages to the same Stream.

Implementing writeDelimitedTo/parseDelimitedFrom would require changes to protobuf-php/protobuf-plugin to generate the new methods or if possible implementing it in AbstractMessage

But given a good PR I would be happy to reconsider it.