Help offered: File Integrity Plugin update from blake2 to blake3
Closed this issue · 5 comments
Hi, does anybody like to have this feature?
https://github.com/BLAKE3-team/BLAKE3/releases/tag/1.0.0
Performance improvements are huge.
Do you like to get a pull request from me for this update? Or is already somebody working on it?
If somebody did evaluate and postponed this task, I would like to know the reason and reevaluate it.
Based on your feedback, I will share my work or keep it for myself.
Hi,
I haven't tested the new version yet. Main point, it has to be compatible with the older version and be able to work with the existing hashes.
You may make a PR and further testing can be done.
Thanks
Yea, I cannot replace blake2. We have to add a blake3 option because we don't want to waste the hash effort of the existing user base.
Blake3 is multithreaded, mmap has a bit of overhead, for that reason, blake3 team did add a 16kb file size check into the tool to speed up the process for smaller files.
Performance is around factor 2 at single threading without AVX. And factor 6 is expected with AVX
If you have cores to spare, then it really begins to fly.
I would add a number of threads option as well to fine tune the overhead during parallel usage.
Maybe we can leave num-threads at 1 because if you have AVX you are very unlikely to have enough data bandwidth to keep more than 1 core feed.
@bergware after blake3 got merged, which is basically a power consumption tweak, I will have another pull request to double or triple the IO Performance to actually reduce the execution time of this thing.
How would you like the 2nd pull request? Same like the first one with README update or just the bash script change?
@bergware merge was step one.
Do you need help with the change log for your binary commit?
Should I prepare a pull request for https://github.com/unraid/lang-en_US/tree/master/Dynamix%20File%20Integrity ?