Zlib.compress issues with zlib >= 1.2.9

[mfp]

Zlib 1.2.9 adds a stream state integrity check (accidental clobbering detection) by storing a pointer to itself in the stream state:

https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da3b647f07d718623f/deflate.c#L361

Cryptokit uses an abstract block to hold the stream state. When the block is moved (from the minor to the major heap, or as the result of a compaction), the pointer it contains will no longer match the stream state address. The above check will fail and deflate will return Z_STREAM_ERROR which becomes an exception.

I first noticed this bug with a larger program that used to work fine on older macOS versions and broke in more recent ones (with zlib 1.2.11). I reproduced it with zlib 1.2.9 and 1.2.11 (macOS again) and finally managed to discover the origin and trigger it reliably with this minimal testcase:

module C = Cryptokit

let () =
  let c = C.Zlib.compress () in
    (* move the associated custom block to the major heap *)
    Gc.full_major ();
    Printf.printf "%d\n%!" @@ String.length @@ C.transform_string c "bug"

I also verified that the problem disappears if I modify the executable to use zlib 1.2.8, thus providing further assurance that this is the source of the issue.

It would be possible (if hackish) to just overwrite that pointer to ensure it matches the address (would require a dynamic zlib check to do it only on >=1.2.9 because the self-pointer might not have existed in early versions) before calling deflate, but this is fragile since the internal state is opaque and could change at any time (the pointer is the first member so it seems unlikely, though).

So the safest way is to malloc the stream and only keep a pointer in the custom block, along with the associated custom block machinery (release in finalizer, etc).

I'm fairly sure I bumped into an error of similar nature ("self-pointer" check and failure when block is moved) in another binding, but cannot remember which one (curl maybe?). Maybe a word of caution against direct use of the block memory for such opaque handles would make sense in the OCaml manual.

[mfp]

This is likely the bug behind #7, which happened randomly -- depending on GC activity, I presume -- with zlib 1.2.11 but couldn't be reproduced with older versions (from Ubuntu 16.04). It does not affect utop only, so I'm leaving it as a standalone issue.

[mfp]

It turns out that camlzip used to have the same bug. I have adapted the fix in #17.

[UnixJunkie]

This can be closed I guess.