bug: invalid CRC computed
hairyhenderson opened this issue · 1 comments
In previous versions of the GCS storage client (cloud.google.com/go/storage, up to v1.29.0), there was a parsing bug that caused the client to skip CRC validation.
However, in v1.30.0, a fix for that bug was released, and since then I've received this error while reading:
storage: bad CRC on read: got 1077016486, want 2266108419
My hunch is the CRC calculation in fsouza/fake-gcs-server is incorrect (compared to how GCS does it); when the feature was added (#307), this comment in the description stands out:
I compared the results against actually uploading to GCS (and using
gsutil hash -m) and they differ for some reason. Whilst this is somewhat annoying it isn't a problem for me as all I need is something consistent.
I think this wasn't a problem then, since the header wasn't used, but now that cloud.google.com/go/storage has fixed that bug this is now a problem!
/cc @antonydenyer @fsouza
I've done a bit more digging and it looks like the issue is due to compression - I'm gzip-compressing the object on upload, and when downloading the object Decompressive transcoding is occurring.
When trying to determine if the server uncompressed the content, the client expects there to be a X-Goog-Stored-Content-Encoding header set to gzip, without a Content-Encoding header set to gzip:
https://github.com/googleapis/google-cloud-go/blob/main/storage/reader.go#L135
It appears that fake-gcs-server is decompressing the content, but it is not setting the X-Goog-Stored-Content-Encoding header. I think that's the root of this bug. I'll see if I can find a fix.