Add compression option
Max-Meldrum opened this issue · 3 comments
Add compression feature to the networking layer using zstd.
We can also add a compression level variable to the NetworkConfig to allow users to configure compression ratio / speed tradeoffs.
I do like having it in config instead of feature flag tbh. We already have so many feature flags. If it's not a huge performance drain when deactivated via config I think that'll be the nicer way for a feature like this.
Update: it was trickier than expected.
Ideal scenario? Compression and decompression happen on the component thread in order to (1) reduce the load on the network buffers; (2) avoid having the network thread doing the compression/decompression.
This is basically what happens if one does the compress/decompress by hand in Serializer/Deserializer. While the compression could be done in the serialise_msg function, the blocker is more on how to do the decompression at the component thread. That is, making sure the data is decompressed before receive_network is called.
// ActorRaw
self.receive_network(nm) May have another go at it down the line..
Just an idea: In stream-oriented protocols we may benefit from a quick compression at the network level if we put it before the buffer flushing. Because we may actually have collected multiple small messages worth of data there that can be compressed while individual small messages cannot be compressed well. I don't know exactly how that logic there works right now (@adamhass ?) but this is basically what's happening in Kompics iirc.