Protocol buffer size limit
Closed this issue · 1 comments
ondovb commented
Messages cannot be greater than 2GB (before compression) due to the implementation of Protocol Buffers. This limit is hit with large alignments because Gingr files rely on zlib compression after serialization. There are several potential solutions:
- Reduce the serialized size by storing only variant rows rather than entire columns (also enables row-based filters)
- Write multiple protobuf messages to each Gingr file
- Use another serialization format, such as HDF5
ondovb commented
Fixed (using Cap'n Proto) in v1.2.