docs: detail on integrity checks
udf2457 opened this issue · 4 comments
Looking through the codebase, it looks like lightningstream does not use the Content-MD5 header in S3-compatible APIs in order to ensure end-to-end integrity of blobs uploaded.
It would be nice to have a couple of paragraphs in the docs as to how lightningstream approaches this.
Lightning Stream uses https://github.com/PowerDNS/simpleblob for object storage, which uses https://github.com/minio/minio-go for S3 storage.
Sure, I looked briefly through that code yesterday. However you need to explicitly set the MD5 option in the minio-go struct when calling it and - as far as I can tell after a quick glance - simpleblob is neither calculating the MD5 or calling minio-go with the MD5 option.
Sending Content-MD5 is highly recommended, because otherwise the receiving S3 service has no way of ensuring integrity. With Content-MD5, the S3 service caluclates MD5 and compares it against the MD5 provided in Content-MD5, and so you therefore have a high level of confidence that what you uploaded is what was stored.
Thanks, I added PowerDNS/simpleblob#49 for this and added it to the next patch release milestone.