Inconsistent message formats?

[jimfleming]

Ok, so I have two programs, one in node, another in rust. Rust can use capnp over zmq to communicate with another rust process. Works great. Node can also communicate with another node process. Works great (so far).

Now the node process is supposed to communicate with a rust process. Node can successfully send a request to rust and rust can successfully decode this message and send back a response. But the response I get back from rust looks... wrong? And it throws this exception when I try to parse it:

{ [Error: expected array.size() >= offset + segmentSize; Message ends prematurely in first segment.] cppFile: 'src/capnp/serialize.c++', line: 56 }

Which comes from here:
https://github.com/sandstorm-io/capnproto/blob/master/c%2B%2B/src/capnp/serialize.c%2B%2B#L56

For testing I encoded the expected response in node and the bytes don't match the same response I receive from the rust process:

Response encoded in rust (encoding/decoding works in rust):

<SlowBuffer 00 00 00 00 01 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00>

Same response encoded in node (encoding/decoding works in node):

<SlowBuffer 00 00 00 00 05 00 00 00 00 00 00 00 01 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00>

The lengths are different and it looks like the response from rust is missing a header despite having the same encoded content. Both response schemas are identical, including capnp ids (but not the same files due to different repos).

• capnp: v0.5.1
• capnp-rust: v0.1.26
• capnpc-rust: v0.1.26
• node-capnp: master / v0.1.11
• Both the request and response schemas make use of a single union field if that makes a difference.

Any ideas for reconciling the two? Is it a version mismatch somewhere? What is the extra header coming from node-capnp? How does rust work without it?

Thanks in advance for any help.

EDIT: If it would be helpful and its not something immediately obvious that I'm missing I'm happy to rig up a GitHub project to replicate the problem.

[dwrensha]

Looks like the message that was encoded in Rust does not include framing information. See https://capnproto.org/encoding.html#serialization-over-a-stream .

The capnp-zmq-rust stuff uses MessageBuilder::get_segments_for_output() directly because ZeroMQ handles framing. If you're just sending the message over a byte stream, you should use ::capnp::serialize::write_message().

[jimfleming]

Interesting. That sounds like the problem but in my case both ends (rust and node) are using zmq to pass messages. Can I tell node-capnp not to expect the framing information since, as you say, zeromq should handle it for me or is it just easier to include the framing information redundantly on the rust side? I don't really have any bandwidth constraints so I'm okay with that.

[dwrensha]

I don't think there's currently a way to tell node-capnp to parse an unframed message directly, though it would probably be straightforward to add such a feature. To work around this limitation, maybe you could construct the segment table yourself before passing the bytes to node-capnp, though it seems like this would necessarily involve copying the whole message.

You might also want to investigate @popham's javascript implementation: https://github.com/capnp-js/serialization

[jimfleming]

Huzzah! That makes a ton of sense and now everything is working. I actually was using capnp-js because it was supposed to be faster but was told (correctly) that it didn't follow the spec which would cause problems with other languages so I abandoned that approach. Turns out the part it wasn't following was the frames header which I don't want anyway.