zlib -3 runtime error

[tomalrussell]

Hi - thank you for making this tool available and open, it's super useful.

I'm able to run it most of the time, but I have a particular FileGDB which throws this error for each layer:

terminate called after throwing an instance of 'std::runtime_error'
  what():  zlib error: -3
Aborted (core dumped)

I'm afraid I'm not sure how to provide a reproducible example, the data I'm working with is proprietary.

This SO answer talks about the same error code. Could it be that the FileGDB is using another compression method (e.g. GZIP instead of ZLIB)? Would ArcRasterRescue handle this case by default?

Another quirk (possible red herring) is that the system I'm working on only has cmake at 3.5, so I downgraded cmake_minimum_required to VERSION 3.5 and replaced the cxx_std_11 argument in calls to target_compile_features with cxx_constexpr (which needs c++ 11 anyway, I understand) to get it to build. This may have been a horrible hack, I'm not very familiar with cmake.

[r-barnes]

That cmake modification sounds like a fine one to me if it otherwise compiles without warnings. Would you like to open a PR for it?

[r-barnes]

This SO answer implies that ZLIB can autodetect and, indeed, that's what's being used in the code:

err = inflateInit2(&strm, (15 + 32)); //15 window bits, and the +32 tells zlib to to detect if using gzip or zlib

If there's any way you could release a data file or access Arc to see what's going on that would be helpful.

I've also modded master to try to get a better handle on where the error is occurring. If you could pull the latest, recompile, and upload the new error message that would be helpful.

Also, note this comment in the code:

        //These two magic bytes indicate zlib compression... unless they don't,
        //since it is possible, if unlikely, for uncompressed data to begin a
        //block with these values. The `band_types` field has bits which
        //indicate compression, but appear to do so non-uniquely (lz77, lzw,
        //maybe others map to the same compression bits). Therefore, since the
        //compression indicators have not yet been entirely figured out, this
        //if- clause checks for the magic bytes and checks the length of the
        //field to determine probabilistically if compression is being used. The
        //check should be fairly robust, though, since we expect some degree of
        //compression for any non-pathological data.

It's possible you've encountered this rare situation and it's time to think carefully about what to do about it. To test for this, could you please change this line:

if(rb.compression_type=="lz77"){

to this

if(false && rb.compression_type=="lz77"){

and

} else if(rb.compression_type=="uncompressed") {

to

} else if(true || rb.compression_type=="uncompressed") {

and see if you get a reasonable output that way?

[tomalrussell]

Thanks for the pointers.

I've rebuilt from master and get the later zlib inflate error: -3.

I've got access to ArcMap - here's the raster info, which says it uses LZ77.

If I clip out a tiny area and export that back to the GDB, and then try ArcRasterRescue on the clipped area, it runs fine.

If I assume uncompressed data with the changes above, ArcRasterRescue runs but outputs a noisy tiff which is presumably just the compressed data interpreted directly as though it were uncompressed.

If I define EXPLOREUNPACK, I get a lot of happy decompressing reported, followed by the same error at the end:

...
Decompressing with zlib
Decompressed: 4 4 4 4 4 4 4 4 4 4
Unpacked: 4 4 4 4 4 4 4 4 4 4
Length = 18
Compressed: 48 48 48 48 48 48 48 50 48 48
Decompressing with zlib
terminate called after throwing an instance of 'std::runtime_error'
  what():  zlib inflate error: -3
Aborted (core dumped)

I'm not blocked by this at all, but would be happy to put some more debugging elements into the code if it's of interest.

[r-barnes]

@tomalrussell : I appreciate your willingness to help here, but I'm not sure if we'll be able to get to the bottom of this without some kind of data we can both look at.

One though, though: if you mod the code so it std::cout's a warning but doesn't throw an exception you can set it up to warn and keep going. I wonder what percentage of blocks are affected and whether there's anything distinctive about them (e.g., where they're located in the raster)?