Failure to read compressed empty table from java implementation
DrChainsaw opened this issue · 3 comments
The problem is that the java implementation sets the length field in RecordBatch to 8 bytes in this case. The consequence is that this check does nothing so the uncompression continues, finds len=0 when reading the first 8 bytes from the pointer and passes a zero length array to transcode which then seems to hang indefinitely.
Doing the same type of check and return as for buffer.length on len resolves the issue for me.
It might very well be the java implementation which does something wrong here as I couldn't find any reference in the format specification on how to describe lengths when using compression. I have opened an issue there as well. File generated by the java code in that issue: randon_access_to_file.zip.
Both pyarrow and the Julia implementation sets it to 0 when writing an empty table to disk.
It does make some sense to set buffer.length to 8 and let the len field carry the information even though it is 8 bytes which could have been saved. Both the java implementation and pyarrow can read the attached file.
No, just happens to touch the same function.