RecordBatchEncoder::compute_size would be really nice
thedodd opened this issue · 2 comments
Thoughts on adding such a method? Similar interface to ::encode, but would calculate the encoding size based on the given params.
Totally! These kinds of additional utilities are great. In general, it seems like you're using things a bit differently then others, and so some integration tests / examples of your patterns might be helpful here for others if you end up adding this method.
One of the primary use cases would be for ensuring buffers have enough space to be able to encode a full record batch.
However, this might be tricky as compression will obviously influence this. Perhaps the best we could do is provide the size of the batch without compression, and make it very clear in the docs that if compression is used, the size will almost always be smaller.