Packing messages into packets

[cBournhonesque]

Hi!
I was wondering what your opinion was on how to pack messages into packets.

• Is it better to have one big message that I manually split between sub-messages of size MTU? (to avoid having the packet be fragmented, which makes it much more unreliable if there is any packet loss or jitter)
• Is it better to have lots of small messages? Then there might be additional overhead of having many MessageIds and other header information added.
• If I want to guarantee that 2 pieces of data are received together (for example I have data about a parent object and a child object. Because of these hierarchy I cannot have the data about the child arrive before the data about the parent arrives first), I need to put these 2 pieces of data into a single message?

[gafferongames]

Yojimbo lets you attach blocks of memory on messages.

a) In the reliable channel, these blocks will be sliced up and sent reliably in packets smaller than MTU automatically, and the reliable channel will block any messages to be received after the message with block attach so they arrive after the block send completes, so messages will always be received reliably and in-order.

b) Alternatively, if you need to send a large block of data and it's not too large, eg. 4-8 packets @ MTU worth, you could send it as a big fat unreliable message, and the system will fragment into MTU packets and re-assemble automatically for you, but if any of the fragments are dropped, the whole message is lost.

Basically, a) was designed for you to send down a large block of data on connect, or on load level or whatever on the reliable channel. b) was designed so that when you are sending delta encoded snapshots, it would allow you to go over MTU and it would do fragmentation and reassembly for you automatically, since some routers on the internet drop any packet fragments done at the IP level.

cheers

[gafferongames]

And to answer your question -- I need more information about what you are trying to do. The correct answer depends very much on whether you are using the reliable or unreliable channels to send messages, and what you are trying to achieve.

[cBournhonesque]

My question was not directly related to Yojimbo, sorry about that. (maybe I could send you an email directly if that is a more appropriate channel)?
I've been developing a networking library for an ECS game engine as a hobby project: https://github.com/cBournhonesque/lightyear
I've read all your articles many times as they provide a great starting point!

One thing i've been struggling is figuring out how messages should fit into packets.
I have two types of messages: Actions (Entity spawn/Entity despawn/Component insertion/Component removal) and Updates (Component update). Actions are sent reliably, and Updates are sent unreliably.

Some invariants I'm trying to guarantee:

• An entity's state is always consistent (i.e. we cannot have some Component updates for entity 1 in packet A, and some components updates for the same entity in packet 2; they must all be in the same packet)
• An update for an entity 1 is only applied after all actions for that entity have been applied
• An update for an entity 1 is only applied if we haven't already received an update for that entity for a more recent tick
• I want to be able to guarantee in some cases that information about 2 entities are networked together, so that I can guarantee on the receiver side that the replicated entities are both equal the sender state at a given tick T. (i.e. I want to avoid that entity 1 corresponds the sender tick T and entity 2 corresponds to the sender tick T') This is useful for:
  • parent-child hierarchies, where I cannot spawn the child if the parent isn't spawned. So they both need to be included in the same packet
  • client-side prediction. I want all the entities that are rollbacked to be on the same tick, so they must be sent in the same packet

Currently I achieve this by having lots of individual messages (Actions for entity E1 or Updates for entities [E1, E2]). I'm wondering if I can optimize how messages are packed into packets.

1. Should I just send all Actions in one big message? Even if it's split up into multiple packets, it is sent reliably so it shouldn't be too much of an issue? In general I'm very wary of sending big fragmented packets.
2. Is it recommended to have big individual messages instead of small individual messages to avoid wasting bandwidth on per-packet metadata?
3. How do you solve the problem of having to guarantee that 2 pieces of state arrive together, for example for client-side prediction?