lz4/lz4-java

Stream ended prematurely when trying to readAllBytes

yash29896 opened this issue · 0 comments

So I'm trying to switch our compressor to lz4-java from snappy and I was trying something like this for a POC

When I try a readAllBytes() on the LZ4FrameInputStream it throw the below exception an breaks.

ERROR

Exception in thread "main" java.io.IOException: Stream ended prematurely
	at net.jpountz.lz4.LZ4FrameInputStream.readInt(LZ4FrameInputStream.java:245)
	at net.jpountz.lz4.LZ4FrameInputStream.readBlock(LZ4FrameInputStream.java:259)
	at net.jpountz.lz4.LZ4FrameInputStream.read(LZ4FrameInputStream.java:353)
	at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
	at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
	at com.yash.lz4.App.main(App.java:22)

Calling Function

    public static void main( String[] args ) throws IOException
    {
    	ByteHandler handler = new LZ4Compressor();
        byte[] b = {4, 34, 77, 24, 96, 112, 115, 6, 0, 0, -128, 3, 0, 1, 0, 0, 0};  // Test data from one of our unit tests
        InputStream inputStream = handler.deserialize(new ByteArrayInputStream(b));
        byte[] i = inputStream.readAllBytes();
    }

I've done some debugging and it seems like its not marking the frame as finished.

private void readBlock() throws IOException {
    // Reads Ok up to this point
    // Reads the last 6 bytes 3, 0, 1, 0, 0, 0  
    int blockSize = readInt(in);  
    final boolean compressed = (blockSize & LZ4FrameOutputStream.LZ4_FRAME_INCOMPRESSIBLE_MASK) == 0;
    blockSize &= ~LZ4FrameOutputStream.LZ4_FRAME_INCOMPRESSIBLE_MASK;  // blocksize = 6

    // Check for EndMark
    if (blockSize == 0) {           // Since this is 6 its doesnt enter the loop
......
      frameInfo.finish();
      return;
    }

Hence the next time it tries to read the stream which has already been completed it leaks into readBlock(); and breaks.

 public int read() throws IOException {
    while (!firstFrameHeaderRead || buffer.remaining() == 0) {
      if (!firstFrameHeaderRead || frameInfo.isFinished()) {
        if (firstFrameHeaderRead && readSingleFrame) {
          return -1;
        }
	if (!nextFrameInfo()) {
	  return -1;
	}
      }
      readBlock();
    }

This is our Compressor implementation

public final class LZ4Compressor implements ByteHandler {

	  
	  public OutputStream serialize(final OutputStream toSerialize) {
	    try {
	      return new LZ4FrameOutputStream(toSerialize);
	    } catch (IOException e) {
	      throw new UncheckedIOException(e);
	    }
	  }

	  public InputStream deserialize(final InputStream toDeserialize) {
	    try {
	      return new LZ4FrameInputStream(toDeserialize);
	    } catch (IOException e) {
	      throw new UncheckedIOException(e);
	    }
	  }

	  public ByteHandler getInstance() {
	    return new LZ4Compressor();
	  }
	}