asmaloney/libE57Format

Add ability to simply, fully verify file checksum

nh2 opened this issue · 2 comments

nh2 commented

For accepting E57 data in automated processing, it would be awesome if libE57Format could offer a simple way to verify (the checsums of) an entire file with a simple function that can be called ahead of time, vs noticing E57_ERROR_BAD_CHECKSUM during (potentially more time consuming) stream processing.

If we wanted to add a .checkAll() function, how should it best be designed to generically traverse all data while still being as fast as possible?

Are there some slower approaches that would do the trick until then?

Finally, after a file has been verified this way, what would be the best way to disable checksumming during subsequent stream processing?

nh2 commented

As a related aside:

I am getting a bit confused by CHECKSUM_POLICY_ALL docs saying:

Only verify 50% of the checksums.

and CHECKSUM_POLICY_HALF saying

Only verify 25% of the checksums.

Are those copy-paste errors?

Yes - the checksums are very expensive which is why I added the checksum policy.

Offhand I don't think there's an easy way to skip processing of the data - i.e. just read the bytes and do the checksum. The format is very.... intricate. The checksum is calculated on each 1020 bytes of the file, so that alone makes reading these thing a mess. Speed-wise it might be fine for a 5MB files, but with 1.5GB it becomes unmanageable.

I haven't looked at this code in a very long time, but if I can see a way to do what you're asking I'll let you know.

Are those copy-paste errors?

Nope - just my mishandling of the doxygen comments. I've fixed it - thanks!